How to avoid Bouced Buffer

I’ve posted the problem before in thread:
http://www.osronline.com/cf.cfm?PageURL=showlists.cfm?list=NTDEV

It looks the problem is caused by bounced buffer as described in thread:
http://osronline.com/showThread.CFM?link=210428

There’re two kinds of memory: SGL and memory returned by StorPortGetSystemAddress (SA). For the write SRB, data in SGL and SA memory are the same before it send to the storport driver. For the read SRB, our storport driver send the correct data to SA memory. But after StorPortNotification with RequestComplete, system will always copy data from SGL memory to SA memory. We’ve removed all function call to StorPortGetScatterGatherList. But it still do the memory copy.

Since our device can do page alignment only , we will always need to use StorPortGetSystemAddress to return data.
Either stop system from copying memory from SGL to SA or find a way to get virtual address for SGL memory will work. But HOW?

You can set a flag in the device properties during device initialization to tell the OS you always need memory mapped to system virtual addresses. This is NOT going to make I/O requests magically always be page aligned. Mapping memory to virtual addresses does not change the offset in a page.

If an unbuffered request from an application is not page aligned, and your hardware can only do transfers to page aligned memory, a copy will have to happen. How you fix this is you get every application developer who uses unbuffered I/O to conform to your hardware limitations, or you make your hardware capable of efficiently handing the requests from all important applications. Your company only has control over one of these options.

Jan

-----Original Message-----
From: xxxxx@lists.osr.com On Behalf Of xxxxx@gmail.com
Sent: Monday, September 10, 2018 11:44 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] How to avoid Bouced Buffer

I’ve posted the problem before in thread:
http://www.osronline.com/cf.cfm?PageURL=showlists.cfm?list=NTDEV

It looks the problem is caused by bounced buffer as described in thread:
http://osronline.com/showThread.CFM?link=210428

There’re two kinds of memory: SGL and memory returned by StorPortGetSystemAddress (SA). For the write SRB, data in SGL and SA memory are the same before it send to the storport driver. For the read SRB, our storport driver send the correct data to SA memory. But after StorPortNotification with RequestComplete, system will always copy data from SGL memory to SA memory. We’ve removed all function call to StorPortGetScatterGatherList. But it still do the memory copy.

Since our device can do page alignment only , we will always need to use StorPortGetSystemAddress to return data.
Either stop system from copying memory from SGL to SA or find a way to get virtual address for SGL memory will work. But HOW?


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:>

We’ve already set MapBuffers to STOR_MAP_NON_READ_WRITE_BUFFERS. STOR_MAP_ALL_BUFFERS_INCLUDING_READ_WRITE can’t be used because the target OS is server 2008. This option is available after win8.
We can bear with memory copy for those unaligned situations. But we can’t get the right data for read request due to the memory copy after RequestComplete.

> You can set a flag in the device properties during device initialization to tell
> the OS you always need memory mapped to system virtual addresses. This is NOT
> going to make I/O requests magically always be page aligned. Mapping memory to
> virtual addresses does not change the offset in a page.
>
> If an unbuffered request from an application is not page aligned, and your
> hardware can only do transfers to page aligned memory, a copy will have to
> happen. How you fix this is you get every application developer who uses
> unbuffered I/O to conform to your hardware limitations, or you make your
> hardware capable of efficiently handing the requests from all important
> applications. Your company only has control over one of these options.
>
> Jan

>Since our device can do page alignment only

Yeah… THAT’s the problem, then, right?

Cuz nothing prevents a user from sending I/Os with arbitrary buffer base alignment… There’s nothing ANY operating system can do to “fix” this. Either you copy the data to MAKE it page aligned, you fail the request because it doesn’t meet your requirements, or you change your hardware to work properly with unaligned buffers.

I don’t see any other alternative…

Or, perhaps, I’m missing some aspect of what you’re asking… which is entirely possible

Peter
OSR
@OSRDrivers

> But we can’t get the right data for read request due to the memory copy after RequestComplete.

That sounds like you tell the OS your hardware has some limitation, like only 4K aligned transfers, and the STORPORT wrapper is giving you requests that meet your stated requirements by using a bounce buffer. For a hardware storport, request initiation is essentially processed inside a DMA adapter callback. If you want to be sure the DMA adapter won’t use bounce buffers you need to assure you tell the OS your hardware can process requests in their original form. A storport driver that says it’s a virtual device, will always get the requests mapped, and all the DMA adapter processing doesn’t happen.

Jan

I’ve tried a lot of settings.
The lba length is set to 512.
The align mask is set to 0 to allow byte alignment.
Mapbuffer is set to STOR_MAP_NON_READ_WRITE_BUFFERS. (STOR_MAP_ALL_BUFFERS_INCLUDING_READ_WRITE available after win8).

But none works for me. I don’t know what else related settings can be set.

So there’s no way to get virtual address for the physical address returned in SGL?

> That sounds like you tell the OS your hardware has some limitation, like only 4K
> aligned transfers, and the STORPORT wrapper is giving you requests that meet
> your stated requirements by using a bounce buffer. For a hardware storport,
> request initiation is essentially processed inside a DMA adapter callback. If
> you want to be sure the DMA adapter won’t use bounce buffers you need to assure
> you tell the OS your hardware can process requests in their original form. A
> storport driver that says it’s a virtual device, will always get the requests
> mapped, and all the DMA adapter processing doesn’t happen.

Well, I suppose you could always try to explain to all of the C# & java developers why all of their stream writes / reads raise the hypothetical UnallignedIOOperationException exception ???

That will work really well I think when they are focused on REST interfaces using IIS and WCA – or maybe not.

Sent from Mailhttps: for Windows 10

________________________________
From: xxxxx@lists.osr.com on behalf of xxxxx@osr.com
Sent: Tuesday, September 11, 2018 12:19:44 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] How to avoid Bouced Buffer

>Since our device can do page alignment only

Yeah… THAT’s the problem, then, right?

Cuz nothing prevents a user from sending I/Os with arbitrary buffer base alignment… There’s nothing ANY operating system can do to “fix” this. Either you copy the data to MAKE it page aligned, you fail the request because it doesn’t meet your requirements, or you change your hardware to work properly with unaligned buffers.

I don’t see any other alternative…

Or, perhaps, I’m missing some aspect of what you’re asking… which is entirely possible

Peter
OSR
@OSRDrivers


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></https:>