DMA buffer allocation in Storport miniport

Hi,
I was looking at ways to allocate a DMA buffer (of size 10 MB) for
communicating between my device and driver (storport miniport) outside the
context of HwStorFindAdapter. As discussed on the group a couple of days
back, this seems to rule out the usage of StorPortGetUncachedExtension API.
This DMA buffer is not for issuing read/write SRBs but for other data
transfer.

As alternatives, I was looking at doing the following:

a. Allocate a non-paged pool buffer of 10 MB, using
StorPortAllocatePool.
b. Allocate a MDL for the buffer allocated using StorPortAllocateMdl
c. With the MDL and buffer, create scatter-gather list using
StorPortBuildScatterGatherList.
d. Use the SG list for DMA transfers.

Is this approach correct? Are there any pitfalls/issues in this
approach?

I have attempted to do the above and see the following failures:

  1. When I specify the size for StorPortBuildScatterGatherList as 10MB,
    with verifier turned on, I see a stop code of 0xE6. The description
    indicates that the DMA was attempted with buffer which was not locked and is
    from paged pool. I am not sure who attempted the DMA transfer (as I am yet
    setting it up) and also the memory for the buffer was allocated from
    non-paged pool.

  2. I then tried to carve out a smaller DMA buffer from the original 10
    MB buffer by specifying 256K as the size for StorPortBuildScatterGatherList
    (just for testing). The call fails with STOR_STATUS_UNSUCCESSFUL. Since
    there are no guidelines on the size that can be specified when using this
    API, I am not sure if it’s the size that is the issue here or I am doing
    some thing wrong with my code.

On a side note, I am able to successfully allocate a DMA buffer for 10 MB if
I do this from within HwStorFindAdapter using StorPortGetUncachedExtension
API on the same server (Win2k8 R2 with 4GB memory and device supports 64bit
DMA).

Any advice from the experts would definitely help.

Regards,
Girish.

> -----Original Message-----

From: xxxxx@lists.osr.com [mailto:bounce-428754-
xxxxx@lists.osr.com] On Behalf Of Girish Aithal
Sent: Friday, October 22, 2010 5:49 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] DMA buffer allocation in Storport miniport

Hi,
I was looking at ways to allocate a DMA buffer (of size 10 MB) for
communicating between my device and driver (storport miniport) outside the
context of HwStorFindAdapter. As discussed on the group a couple of days
back, this seems to rule out the usage of StorPortGetUncachedExtension
API.
This DMA buffer is not for issuing read/write SRBs but for other data
transfer.

[snip]

On a side note, I am able to successfully allocate a DMA buffer for 10 MB
if
I do this from within HwStorFindAdapter using StorPortGetUncachedExtension
API on the same server (Win2k8 R2 with 4GB memory and device supports
64bit
DMA).

Why was this unacceptable? It works, the alternative is a lot of code for
little gain, so why not just do this? I haven’t gone back to the archives
to see why you didn’t do this. Was it something about not being able to
release the buffer?

Phil

Philip D. Barila (303) 776-1264

>On a side note, I am able to successfully allocate a DMA buffer for 10 MB if I do this from within >HwStorFindAdapter using StorPortGetUncachedExtension API on the same server (Win2k8 R2 with >4GB memory and device supports 64bit DMA).

As an alternative approach to StorPortGetUncachedExtension you could use StorPortAllocateContiguousMemorySpecifyCacheNode. This function could be called outside of HwStorFindAdapter. To get a physical address you need to call just StorPortGetPhysicalAddress because physical memory is contiguous.

Igor Sharovar

>

Why was this unacceptable? It works, the alternative is a lot of code for
little gain, so why not just do this? I haven’t gone back to the archives
to see why you didn’t do this. Was it something about not being able to
release the buffer?

Thanks Phil. I could use StorPortGetUncachedExtension API during
FindAdapter. The only catch I see is that if in case the API fails to
allocate the buffer of the size I requested (possibly due to a lack of
contiguous memory), I still have to revert to allocating smaller non-paged
memory chunks at run time and then mapping them to SG elements.

Regards,
Girish.

wrote in message news:xxxxx@ntdev…
> As an alternative approach to StorPortGetUncachedExtension you could use
> StorPortAllocateContiguousMemorySpecifyCacheNode. This function could be
> called outside of HwStorFindAdapter. To get a physical address you need
> to call just StorPortGetPhysicalAddress because physical memory is
> contiguous.
>

Thanks Igor. I will try the API you suggested, but it may not fit the bill
completely as I need to support Windows 2003, 2008 and 2008 R2 platforms.

wrote in message news:xxxxx@ntdev…
> As an alternative approach to StorPortGetUncachedExtension you could use
> StorPortAllocateContiguousMemorySpecifyCacheNode. This function could be
> called outside of HwStorFindAdapter. To get a physical address you need
> to call just StorPortGetPhysicalAddress because physical memory is
> contiguous.
>
> Igor Sharovar
>
Igor,
I was able to successfully allocate the required memory using
StorPortAllocateContiguousMemorySpecifyCacheNode. However, I need to do the
same for pre Windows 7 OS flavors. As such, was wondering if the method of
allocating memory and creating SG list is my only option. I am doing the
following steps (as mentioned in my original post), but encountering
failures.

1. Allocate non-paged memory using StorPortAllocatePool for 10 MB -
Succesful
2. Allocate an MDL for this memory using StorPortAllocateMDL (size 10 MB) -
Successful
3. Build MDL for non-paged memory using StorPortBuildMdlForNonPagedPool -
Successful
4. Allocate memory for SG list buffer using StorPortAllocatePool -
Successful
5. Build SG list using StorPortBuidlScatterGatherList for the entire 10 MB -
Failed (with STATUS_UNSUCCESSFUL)

I then attempted to re-do step 5 with a smaller chunk of memory (256 KB).
The API succeeded and the Execute routine got invoked. However, when I
traverse through the list of STOR_SCATTER_GATHER_ELEMENT that I receive, I
see that some of the SG elements have the physical address set to 0 with
length field having a non-zero value. This is making me wonder if the SG
list I received is valid or not.

I am pretty sure that I missing/mis-understanding something in steps 1 to 5
above, as this requirement of allocating SG list should be regular in the
miniport world and there should be ways to do it successfully. Could the
experts please point me in the right direction?

If I have to allocate a common buffer between driver and device (with
support for pre Win 7 OS), looks like StorPortGetUncachedExtension is my
only bet. Is there a theoretical max that I can allocate using this API or
does it entirely depend on the system resources?

Regards,
Girish.

> I am pretty sure that I missing/mis-understanding something in steps 1 to 5

above, as this requirement of allocating SG list should be regular in the
miniport world and there should be ways to do it successfully. Could the
experts please point me in the right direction?

If I have to allocate a common buffer between driver and device (with
support for pre Win 7 OS), looks like StorPortGetUncachedExtension is my
only bet.

Correct, and this seems to satisfy the “miniport world”.

What is the need in huge, multi-MB DMA schedule? how many concurrent requests do you have at a moment?


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

>

What is the need in huge, multi-MB DMA schedule? how many concurrent
requests do you have at a moment?

Hi Maxim,
This is not for regular read/write requests. Instead, this is for
private data that the device has to pass onto the driver at regular
intervals.

Regards,
Girish.

>5. Build SG list using StorPortBuidlScatterGatherList for the entire 10 MB - Failed (with >STATUS_UNSUCCESSFUL)
What did you get exactly after calling StorPortBuidlScatterGatherList? STATUS_UNSUCCESSFUL is not in the list of return value.
You should also try to build a Scatter Gather list yourself by using MDL which you get after StorPortBuildMdlForNonPagedPool.

Igor Sharovar

> What did you get exactly after calling StorPortBuidlScatterGatherList?

STATUS_UNSUCCESSFUL is not in the list of return value.
I got the return status of 0xc1000001 (STOR_STATUS_UNSUCCESSFUL). Even I was
surprised at this status as this is not in the list of possible return
values.

Here is the code snippet:

PVOID buf_ptr = NULL;
PVOID mdl_ptr = NULL;
PVOID sg_list_ptr = NULL;
ULONG size, sg_list_length;

size = 10 * 1024 * 1024; // 10 MB

status = StorPortAllocatePool(hw_ext, size, ‘GAT’, &buf_ptr);
if (status != STOR_STATUS_SUCCESS) return FALSE;

/**
* Now, allocate an MDL with the memory allocated.
*/
status = StorPortAllocateMdl(hw_ext, buf_ptr, size, &mdl_ptr);
if (status != STOR_STATUS_SUCCESS) return FALSE;

/**
* Build the MDL allocated.
*/
status = StorPortBuildMdlForNonPagedPool(hw_ext, mdl_ptr);
if (status != STOR_STATUS_SUCCESS) return FALSE;

/**
* Allocate memory for holding the SG list
*/
sg_list_length = sizeof(STOR_SCATTER_GATHER_LIST) + (100 * sizeof
(STOR_SCATTER_GATHER_ELEMENT));
status = StorPortAllocatePool(hw_ext, sg_list_length, ‘GAT’, &sg_list_ptr);
if (status != STOR_STATUS_SUCCESS) return FALSE;

/**
* Build an SG list with the MDL created
*/
status = StorPortBuildScatterGatherList(hw_ext, mdl_ptr, buf_ptr, size,
post_execute, (PVOID)hw_ext, FALSE, sg_list_ptr, sg_list_length);
if (status == STOR_STATUS_SUCCESS) {
} else {
DbgPrint(“Unable to allocate build sg list. Status = %x\n”, status);
return FALSE;
}

You should also try to build a Scatter Gather list yourself by using MDL
which you get after StorPortBuildMdlForNonPagedPool.
Can you please advice me on how do I go about doing this?

So this is just a wrapper around BuildScatterGatherList. I note that
CalculateScatterGatherList is missing from the storport extensions,
which is a way of my asking "how do you know that ‘100’ is the
appropriate number of elements?

Given that there is no StorPortCalculateScatterGatherList extension,
perhaps you want to try StorPortGetScatterGatherList ?

You could also step through the code to see where it is failing with
STATUS_UNSUCCESSFUL.

I assume your error handling is a bit more correct than the code you provided.

Mark Roddy

On Fri, Oct 29, 2010 at 5:06 AM, Girish Aithal wrote:
>> What did you get exactly after calling StorPortBuidlScatterGatherList?
>> STATUS_UNSUCCESSFUL is not in the list of return value.
>
> I got the return status of 0xc1000001 (STOR_STATUS_UNSUCCESSFUL). Even I was
> surprised at this status as this is not in the list of possible return
> values.
>
> Here is the code snippet:
>
> ? ? ? ? ? ? ? PVOID buf_ptr = NULL;
> ? ? ? ? ? ? ? PVOID mdl_ptr = NULL;
> ? ? ? ? ? ? ? PVOID sg_list_ptr = NULL;
> ? ? ? ? ? ? ? ULONG size, sg_list_length;
>
> ? ? ? ?size = 10 * 1024 * 1024; // 10 MB
>
> ? ? ? ?status = StorPortAllocatePool(hw_ext, size, ‘GAT’, &buf_ptr);
> ? ? ? ?if (status != STOR_STATUS_SUCCESS) return FALSE;
>
> ? ? ? ?/
> ? ? ? ? * Now, allocate an MDL with the memory allocated.
> ? ? ? ? */
> ? ? ? ?status = StorPortAllocateMdl(hw_ext, buf_ptr, size, &mdl_ptr);
> ? ? ? ?if (status != STOR_STATUS_SUCCESS) return FALSE;
>
> ? ? ? ?/

> ? ? ? ? * Build the MDL allocated.
> ? ? ? ? /
> ? ? ? ?status = StorPortBuildMdlForNonPagedPool(hw_ext, mdl_ptr);
> ? ? ? ?if (status != STOR_STATUS_SUCCESS) return FALSE;
>
> ? ? ? ?/**
> ? ? ? ? ?
Allocate memory for holding the SG list
> ? ? ? ? ?*/
> ? ? ? ?sg_list_length = ?sizeof(STOR_SCATTER_GATHER_LIST) + (100 * sizeof
> (STOR_SCATTER_GATHER_ELEMENT));
> ? ? ? ?status = StorPortAllocatePool(hw_ext, sg_list_length, ‘GAT’,
> &sg_list_ptr);
> ? ? ? ?if (status != STOR_STATUS_SUCCESS) return FALSE;
>
> ? ? ? ?/**
> ? ? ? ? * Build an SG list with the MDL created
> ? ? ? ? */
> ? ? ? ?status = StorPortBuildScatterGatherList(hw_ext, mdl_ptr, buf_ptr,
> size, post_execute, (PVOID)hw_ext, FALSE, sg_list_ptr, sg_list_length);
> ? ? ? ?if (status == STOR_STATUS_SUCCESS) {
> ? ? ? ?} else {
> ? ? ? ? ? ? ? ?DbgPrint(“Unable to allocate build sg list. Status = %x\n”,
> status);
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? return FALSE;
> ? ? ? ?}
>
>> You should also try to build a Scatter Gather list yourself by using MDL
>> which you get after StorPortBuildMdlForNonPagedPool.
>
> Can you please advice me on how do I go about doing this?
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

>> You should also try to build a Scatter Gather list yourself by using MDL

> which you get after StorPortBuildMdlForNonPagedPool.
Can you please advice me on how do I go about doing this?
Look at MDL struct. MappedSystemVa is a virtual address which you need to translate to physical by calling StorPortGetPhysicalAddress. The “ByteCount” a size of memory of particular MDL entry. The field “Next” points to the next MDL entry. Using these parameters you could build SGL.
I would also suggest to you to verify some parameters in PORT_CONFIGURATION_INFORMATION.
For example, NumberOfPhysicalBreaks could limit a size of allocated Scatter Gather List.

Igor Sharovar

Girish Aithal wrote:

> What did you get exactly after calling StorPortBuidlScatterGatherList?
> STATUS_UNSUCCESSFUL is not in the list of return value.
I got the return status of 0xc1000001 (STOR_STATUS_UNSUCCESSFUL). Even I was
surprised at this status as this is not in the list of possible return
values.

Here is the code snippet:

size = 10 * 1024 * 1024; // 10 MB

/**
* Allocate memory for holding the SG list
*/
sg_list_length = sizeof(STOR_SCATTER_GATHER_LIST) + (100 * sizeof
(STOR_SCATTER_GATHER_ELEMENT));

100 pages is not nearly enough to cover a 10 megabyte region. In the
worst case, you might need as many as 2,560 entries.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

?? Wouldn’t you just use the standard macro ADDRESS_AND_SIZE_TO_SPAN_PAGES to determine the number of pages you need?? Why would there have to be a Miniport specific function to do this?

Sorry if I’m missing something obvious,

Peter
OSR

Aren’t the number of SGE needed determined by run length not by pages?
So the worst case is you need ADDRESS_AND_SIZE_TO_SPAN_PAGES, but you
might need considerably fewer SGEs.

Mark Roddy

On Fri, Oct 29, 2010 at 1:20 PM, wrote:
> ?? Wouldn’t you just use the standard macro ADDRESS_AND_SIZE_TO_SPAN_PAGES to determine the number of pages you need?? ?Why would there have to be a Miniport specific function to do this?
>
> Sorry if I’m missing something obvious,
>
> Peter
> OSR
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>

Ah, yes… of course you’re correct. Our old friend ADDRESS_AND_SIZE_TO_SPAN_PAGES (I’ve always wondered about that name… who’s idea WAS that?) will give you WORST-case numbers… Good point.

Peter
OSR

In my experience, the number of occurrences of physically adjacent pages
is minimal.
There are exceptions, of course.

xxxxx@osr.com
Sent by: xxxxx@lists.osr.com
10/29/2010 02:56 PM
Please respond to
“Windows System Software Devs Interest List”

To
“Windows System Software Devs Interest List”
cc

Subject
RE:[ntdev] DMA buffer allocation in Storport miniport



Ah, yes… of course you’re correct. Our old friend
ADDRESS_AND_SIZE_TO_SPAN_PAGES (I’ve always wondered about that name…
who’s idea WAS that?) will give you WORST-case numbers… Good point.

Peter
OSR


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

> So this is just a wrapper around BuildScatterGatherList. I note that

CalculateScatterGatherList is missing from the storport extensions,
which is a way of my asking "how do you know that ‘100’ is the
appropriate number of elements?

100 pages is not nearly enough to cover a 10 megabyte region. In the
worst case, you might need as many as 2,560 entries.

Thanks Mark and Tim, you are right. 100 was a number that I started off
with, expecting that the API would return a status of
STOR_STATUS_BUFFER_TOO_SMALL. I will surely try increasing the
number of elements using the ADDRESS_AND_SIZE_TO_SPAN_PAGES.

Given that there is no StorPortCalculateScatterGatherList extension,
perhaps you want to try StorPortGetScatterGatherList ?

I thought StorPortGetScatterGatherList needed the miniport to provide
the SRB (and there is no mention that SRB could be NULL). I need this
memory to be available without any specific SRB context.

You could also step through the code to see where it is failing with
STATUS_UNSUCCESSFUL.

Sure Mark, will try this and share my findings.

On Sat, Oct 30, 2010 at 10:09 AM, Girish Aithal wrote:
>>
> I thought StorPortGetScatterGatherList needed the miniport to provide the SRB

- good point. So you are stuck with maxing out the SGL using
ADDRESS_AND_SIZE_TO_SPAN_PAGES.

Wrappers seriously suck when they invariably fail to anticipate all of
the needs of their wrappees and also fail to provide useful supported
mechanisms to get out of the box the wrapper puts you in. KMDF at
least gets this right by allowing the programmer to use WDM as needed
and providing the appropriate conversion interfaces.

In this case, and yours seems to be a legitimate if unusual design
problem, you need access to the adapter object so you can correctly
construct your huge SGL.