Hi,
I’m trying to make a driver and user-land application for streaming data to or from a PCI Express card (but never in both directions at the same time). I’ll focus on data flow in read case:
Camera -> DMA1 -> board RAM -> DMA2 -> PCI Express bus -> PC RAM -> PC hard disk
There are multiple cameras and the maximum amount of data they would generate is 240MB, three times per second (720MB/s in total).
So, before investing time I wanted to ask for suggestions as to how to design my software on PC side for this kind of streaming.
DMA on PCIe card supports scatter/gather mode and, although it supports 64bit addresses, that part is not so great because I can’t really use 64bit address directly. I have to divide it, and write upper 32bits in some register and then provide lower 32bits on the bus. So DMA in S/G can work properly as long as upper 32bits are constant. Otherwise, I’d have to stop the DMA and set that register again which defeats the purpose of S/G I guess.
Anyway, I had this idea:
- allocate two 240MB buffers in application
- use IO completion ports API, ReadFile/WriteFile and switch between the two buffers
- since the hard disk I have is capable of reads and writes up to 120MB/s, I’d just save first 25MB or so from the stream buffer on each WriteFile until I get a better disk
Initially I thought I’d issue many small read requests (32KB or so) to the driver but then it would always create S/G lists so perhaps it’s better to have application pass the two pointers before starting the stream, driver then locks the memory and creates two S/G lists and sends them to the device. That way S/G lists are transferred only once and memory is unlocked after the stream stops. Then I found SetFileIoOverlappedRange function which does the memory-locking so if I got it right, I don’t have to do MmGetSystemAddressForMdlSafe inside my driver?
It’s kind of messy concept in my head so I hope someone can clear this up for me.
Thanks.