> Hello,
My FPGA will support scatter gather.
****
Do you know what scatter-gather is? From the description below, it sounds
like you need a contiguous buffer, which suggests that it can’t support
scatter-gather
****
The goal is that the data will be written from the FPGA directly to the
allocated RAM.
****
This is how Direct I/O mode works
****
I do not want to copy the data once again using ReadFile.
****
So set DO_DIRECT_IO in the device object
****
I need to allocate more than 32MB because I want as much data as possible.
****
No, *someone* needs to allocate more than 32MB, but why do you think it
has to be you? The app, for example, could allocate it.
I did not know there was a restriction on MDLs to 32MB, but this would
mean that you simply can’t *use* more than 32MB in a single transfer. So
a larger buffer would require multiple transfers, as I described. I don’t
know if the old USB example from earlier Windows still appears in the WDK,
but it shows how to do this in great detail.
****
Currently the FPGA works like this: It gets a start address and size of
the
buffer allocated by AllocateCommonBuffer.
*****
It sounds like it handles one address/size pair. For scatter-gather, you
would have a list of address/size pairs, and you would program your device
with the start address and length of that list. It would then
successively fetch address/size pairs for data transfer, and when it
exhausted the list, it would interrupt.
*****
Upon ‘start’ command written to the FPGA (which is memory mapped) the data
is written to the physical buffer.
******
It doesn’t matter how you access the control and status registers of the
FPGA. What is important is that they exist, and there is a known “start”
command.
******
When FPGA reaches to the end of the buffer it starts writing from start.
*****
That’s not scatter-gather; that’s circular buffering, which is quite
different
*****
Currently FPGA does not support scatter gather but as I said this will be
changed soon.
*****
As soon as you add scatter/gather, you let the application allocate the
storage. You deal with the storage you are given by the user. So there
is no particular reason for the driver to have to allocate a buffer.
*****
Thanks,
Zvika.
----- Original Message -----
From:
> To: “Windows System Software Devs Interest List”
> Sent: Saturday, November 05, 2011 05:25
> Subject: Re: [ntdev] Allocating a big non-continuous physical buffer in
> user
> space
>
>
>> See below…
>>> Hello,
>>>
>>> I’m developing a KMDF device driver for a PCIe card.
>>>
>>> This card receives data at a rate of ~2.5Gb/Sec x 4 channels.
>>>
>>> The FPGA writes the data to a physical RAM preallocated using
>>> AllocateCommonBuffer.
>>>
>>> After trying this on Windows XP-SP3 and Windows Sever 2008-64, it
>>> seems AllocateCommonBuffer fails to allocated buffer greater than
>>> 32MB.
>>>
>>> Many experts in this forum warned me that using AllocatedCommonBuffer
>>> is a bad idea.
>>>
>>> But I wanted to keep the way the old driver (and FPGA) worked. Now I
>>> see those experts were right.
>>>
>>> So I want to allocate a big user space buffer (e.g 128MB) that is not
>>> continuous but constructed by physical pages.
>>
>> Note that every page of memory is constructed by physical pages.
>>
>>>
>>> How can make sure my buffer is in physical RAM only ?
>>
>> Well, since you can’t put it any other place, there’s no way to avoid
>> putting your buffer in physical memory. If you mean “not paged out”,
>> which is a different question, then you can either use Direct mode I/O,
>> which will cause the I/O Manager to do an MmProbeAndLockPages, or use
>> Neither mode I/O, and do all the (really complex) work yourself. There
>> are times when Neither mode is appropriate, such as when the user has
>> large buffers that cannot be successfully locked down, so you have to
>> build partial MDLs and split the transfer into multiple transfers. But
>> an
>> app cannot create “a buffer in physical memory” because an app has no
>> concept of what “physical memory” means.
>>
>>>
>>> Then I have to pass the driver the information of the pages
>>> constructing this buffer using IOCTL.
>>
>> Why? Why isn’t ReadFile, WriteFile, or an DeviceIoControl that does a
>> single transfer sufficient? You are trying to invent a gratuitously
>> complex solution to a problem that was solved decades ago.
>>
>>>
>>> I have to give the FPGA this information in advance.
>>
>> No, you have to give it this information when you start the transfer.
>> “In
>> advance” has no meaning in this context.
>>
>>>
>>> I do not want the FPGA to wait for the user level to pass the
>>> information via a read queue.
>>
>> First, you have to realize that the user level does not pass information
>> via a queue. The queue is what YOU do. More importantly, you are
>> thinking in terms of synchronous I/O, and you may find that using Asynch
>> I/O and “pumping up” the queue with a number of buffers may solve your
>> problem.
>>
>> Are you supporting scatter/gather?
>>
>> When I did this, I was getting data overruns until I put 40 ReadFile
>> requests in via async I/O. Note there is a problem (already discussed
>> in
>> this forum) that IRPs are not necessarily put into an I/O Completion
>> Port
>> (IOCP) in the order they are actually completed, but in my case, the
>> buffers had sequence numbers and they could have been reassembled if
>> necessary.
>>
>> You need to restate the problem in terms of what you are trying to
>> accomplish and why a 32MB Common Buffer is inadequate. Would it work if
>> you had two or more 32MB common buffers, would that change the picture?
>> If the FPGA supported scatter/gather, this would help a lot.
>>
>> Note the reason that it fails might be because the kernel address space
>> is
>> fragmented and there are not more than 32MB of contiguous virtual
>> addresses to map it to.
>>
>>>
>>> Thanks,
>>> Zvika.
>>>
>>> —
>>> NTDEV is sponsored by OSR
>>>
>>> For our schedule of WDF, WDM, debugging and other seminars visit:
>>> http://www.osr.com/seminars
>>>
>>> To unsubscribe, visit the List Server section of OSR Online at
>>> http://www.osronline.com/page.cfm?name=ListServer
>>>
>>
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>