Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Before Posting... Please check out the Community Guidelines in the
Announcements and Administration Category, below.

How to avoid Bouced Buffer

I've posted the problem before in thread:
http://www.osronline.com/cf.cfm?PageURL=showlists.cfm?list=NTDEV

It looks the problem is caused by bounced buffer as described in thread:
http://osronline.com/showThread.CFM?link=210428

There're two kinds of memory: SGL and memory returned by StorPortGetSystemAddress (SA). For the write SRB, data in SGL and SA memory are the same before it send to the storport driver. For the read SRB, our storport driver send the correct data to SA memory. But after StorPortNotification with RequestComplete, system will always copy data from SGL memory to SA memory. We've removed all function call to StorPortGetScatterGatherList. But it still do the memory copy.

Since our device can do page alignment only , we will always need to use StorPortGetSystemAddress to return data.
Either stop system from copying memory from SGL to SA or find a way to get virtual address for SGL memory will work. But HOW?

Comments

  • You can set a flag in the device properties during device initialization to tell the OS you always need memory mapped to system virtual addresses. This is NOT going to make I/O requests magically always be page aligned. Mapping memory to virtual addresses does not change the offset in a page.

    If an unbuffered request from an application is not page aligned, and your hardware can only do transfers to page aligned memory, a copy will have to happen. How you fix this is you get every application developer who uses unbuffered I/O to conform to your hardware limitations, or you make your hardware capable of efficiently handing the requests from all important applications. Your company only has control over one of these options.

    Jan

    -----Original Message-----
    From: xxxxx@lists.osr.com <xxxxx@lists.osr.com> On Behalf Of xxxxx@gmail.com
    Sent: Monday, September 10, 2018 11:44 PM
    To: Windows System Software Devs Interest List <xxxxx@lists.osr.com>
    Subject: [ntdev] How to avoid Bouced Buffer

    I've posted the problem before in thread:
    http://www.osronline.com/cf.cfm?PageURL=showlists.cfm?list=NTDEV

    It looks the problem is caused by bounced buffer as described in thread:
    http://osronline.com/showThread.CFM?link=210428

    There're two kinds of memory: SGL and memory returned by StorPortGetSystemAddress (SA). For the write SRB, data in SGL and SA memory are the same before it send to the storport driver. For the read SRB, our storport driver send the correct data to SA memory. But after StorPortNotification with RequestComplete, system will always copy data from SGL memory to SA memory. We've removed all function call to StorPortGetScatterGatherList. But it still do the memory copy.

    Since our device can do page alignment only , we will always need to use StorPortGetSystemAddress to return data.
    Either stop system from copying memory from SGL to SA or find a way to get virtual address for SGL memory will work. But HOW?


    ---
    NTDEV is sponsored by OSR

    Visit the list online at: <http://www.osronline.com/showlists.cfm?list=ntdev>;

    MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
    Details at <http://www.osr.com/seminars>;

    To unsubscribe, visit the List Server section of OSR Online at <http://www.osronline.com/page.cfm?name=ListServer>;
  • We've already set MapBuffers to STOR_MAP_NON_READ_WRITE_BUFFERS. STOR_MAP_ALL_BUFFERS_INCLUDING_READ_WRITE can't be used because the target OS is server 2008. This option is available after win8.
    We can bear with memory copy for those unaligned situations. But we can't get the right data for read request due to the memory copy after RequestComplete.


    >> You can set a flag in the device properties during device initialization to tell
    >> the OS you always need memory mapped to system virtual addresses. This is NOT
    >> going to make I/O requests magically always be page aligned. Mapping memory to
    >> virtual addresses does not change the offset in a page.
    >>
    >> If an unbuffered request from an application is not page aligned, and your
    >> hardware can only do transfers to page aligned memory, a copy will have to
    >> happen. How you fix this is you get every application developer who uses
    >> unbuffered I/O to conform to your hardware limitations, or you make your
    >> hardware capable of efficiently handing the requests from all important
    >> applications. Your company only has control over one of these options.
    >>
    >> Jan
  • >Since our device can do page alignment only

    Yeah.... THAT's the problem, then, right?

    Cuz nothing prevents a user from sending I/Os with arbitrary buffer base alignment.... There's nothing ANY operating system can do to "fix" this. Either you copy the data to MAKE it page aligned, you fail the request because it doesn't meet your requirements, or you change your hardware to work properly with unaligned buffers.

    I don't see any other alternative...

    Or, perhaps, I'm missing some aspect of what you're asking... which is entirely possible

    Peter
    OSR
    @OSRDrivers

    Peter Viscarola
    OSR
    @OSRDrivers

  • > But we can't get the right data for read request due to the memory copy after RequestComplete.

    That sounds like you tell the OS your hardware has some limitation, like only 4K aligned transfers, and the STORPORT wrapper is giving you requests that meet your stated requirements by using a bounce buffer. For a hardware storport, request initiation is essentially processed inside a DMA adapter callback. If you want to be sure the DMA adapter won't use bounce buffers you need to assure you tell the OS your hardware can process requests in their original form. A storport driver that says it's a virtual device, will always get the requests mapped, and all the DMA adapter processing doesn't happen.

    Jan
  • I've tried a lot of settings.
    The lba length is set to 512.
    The align mask is set to 0 to allow byte alignment.
    Mapbuffer is set to STOR_MAP_NON_READ_WRITE_BUFFERS. (STOR_MAP_ALL_BUFFERS_INCLUDING_READ_WRITE available after win8).

    But none works for me. I don't know what else related settings can be set.

    So there's no way to get virtual address for the physical address returned in SGL?

    >> That sounds like you tell the OS your hardware has some limitation, like only 4K
    >> aligned transfers, and the STORPORT wrapper is giving you requests that meet
    >> your stated requirements by using a bounce buffer. For a hardware storport,
    >> request initiation is essentially processed inside a DMA adapter callback. If
    >> you want to be sure the DMA adapter won't use bounce buffers you need to assure
    >> you tell the OS your hardware can process requests in their original form. A
    >> storport driver that says it's a virtual device, will always get the requests
    >> mapped, and all the DMA adapter processing doesn't happen.
  • M_M-2M_M-2 Posts: 833
    Well, I suppose you could always try to explain to all of the C# & java developers why all of their stream writes / reads raise the hypothetical UnallignedIOOperationException exception ????



    That will work really well I think when they are focused on REST interfaces using IIS and WCA – or maybe not.





    Sent from Mail for Windows 10



    ________________________________
    From: xxxxx@lists.osr.com on behalf of xxxxx@osr.com
    Sent: Tuesday, September 11, 2018 12:19:44 PM
    To: Windows System Software Devs Interest List
    Subject: RE:[ntdev] How to avoid Bouced Buffer

    >Since our device can do page alignment only

    Yeah.... THAT's the problem, then, right?

    Cuz nothing prevents a user from sending I/Os with arbitrary buffer base alignment.... There's nothing ANY operating system can do to "fix" this. Either you copy the data to MAKE it page aligned, you fail the request because it doesn't meet your requirements, or you change your hardware to work properly with unaligned buffers.

    I don't see any other alternative...

    Or, perhaps, I'm missing some aspect of what you're asking... which is entirely possible

    Peter
    OSR
    @OSRDrivers



    ---
    NTDEV is sponsored by OSR

    Visit the list online at:

    MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
    Details at

    To unsubscribe, visit the List Server section of OSR Online at
Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!