Large SCSI transfers from user mode

NTDEV
1

Hi all,

I’m trying to determine if it is possible to issue a large data transfer
with a single SCSI command to a device. The transfer needs to be 16MB in a
single command, which exceeds the Scatter/Gather list capacities of most
HBA’s. My assumption is that this can be done by implemeting a driver that
uses MDL or large contiguous physical memory blocks. Are there ways to
accomplish the same from a user-mode program?

Thanks,
Rob

2

Rob Turk wrote:

Hi all,

I’m trying to determine if it is possible to issue a large data transfer
with a single SCSI command to a device. The transfer needs to be 16MB in a
single command, which exceeds the Scatter/Gather list capacities of most
HBA’s. My assumption is that this can be done by implemeting a driver that
uses MDL or large contiguous physical memory blocks. Are there ways to
accomplish the same from a user-mode program?

Look at the SPTI sample in the WDK, especially the explanatory spti.htm
in that sample. It explicitly discusses this scenario.

Location: src\stoarage\tools\spti\spti.htm

–
Jakob B?hm, M.Sc.Eng. * xxxxx@danware.dk * direct tel:+45-45-90-25-33
Danware Data A/S * Bregnerodvej 127 * DK-3460 Birkerod * DENMARK
http://www.netop.com * tel:+45-45-90-25-25 * fax tel:+45-45-90-25-26
Information in this mail is hasty, not binding and may not be right

3

No, I don’t think what Jakob says is true. Yes, the SPTI sample in the DDK discusses the maximum transfer length to some degree,
but it does not directly address Rob’s question. What is not mentioned in the SPTI documentation is that the MaximumSGList entry
for a particular HBA is limited to an 8-bit value. Therefore, the maximum transfer size with SPTI is limited to (255 - 1) * 4KB =
1016 KB, which is just shy of 1 MB. You can’t do 16 MB transfers without some special kernel-mode programming.

Rob, I have done what you describe, so yes, it is indeed possible. That is, one can implement a driver that takes a list of memory
segments instead of one virtually contiguous chunk of memory. But if you are able to figure out how to do it with just user-mode
programming (no kernel-mode development) that would be of interest. Let me know if you have a need for a driver that does the 16 MB
transfers and I will be happy to tell you more.

Thanks,
Don

“Jakob Bohm” wrote in message news:xxxxx@ntdev…
Rob Turk wrote:
> Hi all,
>
> I’m trying to determine if it is possible to issue a large data transfer with a single SCSI command to a device. The transfer
> needs to be 16MB in a single command, which exceeds the Scatter/Gather list capacities of most HBA’s. My assumption is that this
> can be done by implemeting a driver that uses MDL or large contiguous physical memory blocks. Are there ways to accomplish the
> same from a user-mode program?
>

Look at the SPTI sample in the WDK, especially the explanatory spti.htm
in that sample. It explicitly discusses this scenario.

Location: src\stoarage\tools\spti\spti.htm

–
Jakob Bøhm, M.Sc.Eng. * xxxxx@danware.dk * direct tel:+45-45-90-25-33
Danware Data A/S * Bregnerodvej 127 * DK-3460 Birkerod * DENMARK
http://www.netop.com * tel:+45-45-90-25-25 * fax tel:+45-45-90-25-26
Information in this mail is hasty, not binding and may not be right

4

Hi Don,

That’s what I though as well. I tried to contact you specifically before
posting on NTDev, but unfortunately your e-mail system refuses my mails… I
am indeed looking for a way to go beyond the MaximumSGList limit. If you can
lift a tip of the curtain then I’d appreciate it.

Thanks,
Rob

“Don Matthews” wrote in message news:xxxxx@ntdev…
>
> No, I don’t think what Jakob says is true. Yes, the SPTI sample in the
> DDK discusses the maximum transfer length to some degree, but it does not
> directly address Rob’s question. What is not mentioned in the SPTI
> documentation is that the MaximumSGList entry for a particular HBA is
> limited to an 8-bit value. Therefore, the maximum transfer size with SPTI
> is limited to (255 - 1) * 4KB = 1016 KB, which is just shy of 1 MB. You
> can’t do 16 MB transfers without some special kernel-mode programming.
>
> Rob, I have done what you describe, so yes, it is indeed possible. That
> is, one can implement a driver that takes a list of memory segments
> instead of one virtually contiguous chunk of memory. But if you are able
> to figure out how to do it with just user-mode programming (no kernel-mode
> development) that would be of interest. Let me know if you have a need
> for a driver that does the 16 MB transfers and I will be happy to tell you
> more.
>
> Thanks,
> Don
>
>
>
> “Jakob Bohm” wrote in message news:xxxxx@ntdev…
> Rob Turk wrote:
>> Hi all,
>>
>> I’m trying to determine if it is possible to issue a large data transfer
>> with a single SCSI command to a device. The transfer needs to be 16MB in
>> a single command, which exceeds the Scatter/Gather list capacities of
>> most HBA’s. My assumption is that this can be done by implemeting a
>> driver that uses MDL or large contiguous physical memory blocks. Are
>> there ways to accomplish the same from a user-mode program?
>>
>
> Look at the SPTI sample in the WDK, especially the explanatory spti.htm
> in that sample. It explicitly discusses this scenario.
>
> Location: src\stoarage\tools\spti\spti.htm
>
>
>
>
> –
> Jakob Bøhm, M.Sc.Eng. * xxxxx@danware.dk * direct tel:+45-45-90-25-33
> Danware Data A/S * Bregnerodvej 127 * DK-3460 Birkerod * DENMARK
> http://www.netop.com * tel:+45-45-90-25-25 * fax tel:+45-45-90-25-26
> Information in this mail is hasty, not binding and may not be right
>
>
>

5

Don Matthews wrote:

No, I don’t think what Jakob says is true.

I did not say what you think I said, I merely pointed to a WDK sample
which included a discussion of the issue. In my hurry to get the OP an
answer I did not read spti.htm closely enough to notice that it did not
explain how to allocate physically contiguous memory from user mode.

One way (on recent versions of Windows) is to call VirtualAlloc with
MEM_LARGE_PAGES. This allocates memory using big 2MB or 4MB pages,
thus providing much larger memory per entry in the scatter/gather lists.
Note that this flag requires the SE_LOCK_MEMORY_PRIVILEGE to prevent
malicious non-administrator code from using it to overload the system.

–
Jakob B?hm, M.Sc.Eng. * xxxxx@danware.dk * direct tel:+45-45-90-25-33
Danware Data A/S * Bregnerodvej 127 * DK-3460 Birkerod * DENMARK
http://www.netop.com * tel:+45-45-90-25-25 * fax tel:+45-45-90-25-26
Information in this mail is hasty, not binding and may not be right

6

Actually, Jakob, I think you hit the nail right on the head, you just
didn’t notice it, as I also didn’t notice it, until I looked at it for the
400th time, and just had an epiphany.

Most of us, I’m sure, use IOCTL_SCSI_PASS_THROUGH_DIRECT, because we don’t
want to pay for the data buffer copy. Because the UM buffer needs to be
described by the SGL, we find ourselves limited to whatever the miniport
can handle, which is usually a lot less than we need.

But look at IOCTL_SCSI_PASS_THROUGH, instead. It’s a METHOD_BUFFERED
IOCTL, with a contiguous buffer. So the IO manager is going to allocate a
non-paged buffer big enough to hold the structure plus the user data, copy
the structure and user data (on a DATA_OUT, anyway) into the non-paged
buffer, then send the IRP down with the VA pointing at that non-paged
buffer. Voila, no SGL problem, since it’s going to be contiguous memory,
and even those HBAs that have quirky limitations on the size of a single
SGL element should be able to describe the entire buffer in the SGL.

Anyone see any obvious holes in this?

Phil

Philip D. Barila
Seagate Technology LLC
(720) 684-1842

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jakob Bohm

Sent: Wednesday, May 02, 2007 8:20 AM
To: “Windows System Software Devs Interest List”
Subject: Re:[ntdev] Large SCSI transfers from user mode

Don Matthews wrote:
> No, I don’t think what Jakob says is true.

I did not say what you think I said, I merely pointed to a WDK sample
which included a discussion of the issue. In my hurry to get the OP an
answer I did not read spti.htm closely enough to notice that it did not
explain how to allocate physically contiguous memory from user mode.

One way (on recent versions of Windows) is to call VirtualAlloc with
MEM_LARGE_PAGES. This allocates memory using big 2MB or 4MB pages,
thus providing much larger memory per entry in the scatter/gather lists.
Note that this flag requires the SE_LOCK_MEMORY_PRIVILEGE to prevent
malicious non-administrator code from using it to overload the system.

7

Philip-

>
But look at IOCTL_SCSI_PASS_THROUGH, instead. It’s a METHOD_BUFFERED IOCTL, with a contiguous buffer. So the IO manager is going to allocate a non-paged buffer big enough to hold the structure plus the user data, copy the structure and user data (on a DATA_OUT, anyway) into the non-paged buffer, then send the IRP down with the VA pointing at that non-paged buffer. Voila, no SGL problem, since it’s going to be contiguous memory, and even those HBAs that have quirky limitations on the size of a single SGL element should be able to describe the entire buffer in the SGL.

Anyone see any obvious holes in this?
<<

I just checked the IO Manager code (Vista SP1, but I’m sure this has pretty much always been the case), and it uses ExAllocatePool type API for the buffers. That won’t give you contiguous physical memory, just contiguous virtual memory.

8

OK, how fragmented does NP pool actually get? Anyone ever looked at the
segment sizes of a NP pool buffer? I haven’t. I’ve spent a lot of time
looking at UM buffer segments, and segments of 8K (2 pages) are very
common, 12K (3 pages) are not rare, and anything more than 16K (4 pages)
happen almost never.

I don’t suppose it matters that much, if it can be badly fragmented, at
some point, when you need it most, it probably will be.

Phil

Philip D. Barila
Seagate Technology LLC
(720) 684-1842

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Bob Kjelgaard

Sent: Wednesday, May 02, 2007 9:50 AM
To: “Windows System Software Devs Interest List”
Subject: RE: [ntdev] Large SCSI transfers from user mode

Philip-

>>
But look at IOCTL_SCSI_PASS_THROUGH, instead. It’s a METHOD_BUFFERED
IOCTL, with a contiguous buffer. So the IO manager is going to allocate a
non-paged buffer big enough to hold the structure plus the user data, copy
the structure and user data (on a DATA_OUT, anyway) into the non-paged
buffer, then send the IRP down with the VA pointing at that non-paged
buffer. Voila, no SGL problem, since it’s going to be contiguous memory,
and even those HBAs that have quirky limitations on the size of a single
SGL element should be able to describe the entire buffer in the SGL.

Anyone see any obvious holes in this?
< <

I just checked the IO Manager code (Vista SP1, but I’m sure this has
pretty much always been the case), and it uses ExAllocatePool type API for
the buffers. That won’t give you contiguous physical memory, just
contiguous virtual memory.

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

9

Much the same … I’ve allocated many multi-megabyte buffers from NP and have nearly alwyas seen a few 8K PFNs in the MDL; e.g. I get less than the number of SG elements I computed.

–
The personal opinion of
Gary G. Little

wrote in message news:xxxxx@ntdev…

OK, how fragmented does NP pool actually get? Anyone ever looked at the segment sizes of a NP pool buffer? I haven’t. I’ve spent a lot of time looking at UM buffer segments, and segments of 8K (2 pages) are very common, 12K (3 pages) are not rare, and anything more than 16K (4 pages) happen almost never.

I don’t suppose it matters that much, if it can be badly fragmented, at some point, when you need it most, it probably will be.

Phil

Philip D. Barila
Seagate Technology LLC
(720) 684-1842

------------------------------------------------------------------------------
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Bob Kjelgaard
Sent: Wednesday, May 02, 2007 9:50 AM
To: “Windows System Software Devs Interest List”
Subject: RE: [ntdev] Large SCSI transfers from user mode

Philip-

>>
But look at IOCTL_SCSI_PASS_THROUGH, instead. It’s a METHOD_BUFFERED IOCTL, with a contiguous buffer. So the IO manager is going to allocate a non-paged buffer big enough to hold the structure plus the user data, copy the structure and user data (on a DATA_OUT, anyway) into the non-paged buffer, then send the IRP down with the VA pointing at that non-paged buffer. Voila, no SGL problem, since it’s going to be contiguous memory, and even those HBAs that have quirky limitations on the size of a single SGL element should be able to describe the entire buffer in the SGL.

Anyone see any obvious holes in this?
< <

I just checked the IO Manager code (Vista SP1, but I’m sure this has pretty much always been the case), and it uses ExAllocatePool type API for the buffers. That won’t give you contiguous physical memory, just contiguous virtual memory.

—
Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

10

You’re right, Jakob. Poor choice of words on my part.
I thought you were saying that the SPTI sample shows how
to send large requests (>1 MB), which I don’t believe it does,
but upon re-reading, you were not claiming that it did. My bad.

“Jakob Bohm” wrote in message news:xxxxx@ntdev…
Don Matthews wrote:
> No, I don’t think what Jakob says is true.

I did not say what you think I said, I merely pointed to a WDK sample
which included a discussion of the issue. In my hurry to get the OP an
answer I did not read spti.htm closely enough to notice that it did not
explain how to allocate physically contiguous memory from user mode.

One way (on recent versions of Windows) is to call VirtualAlloc with
MEM_LARGE_PAGES. This allocates memory using big 2MB or 4MB pages,
thus providing much larger memory per entry in the scatter/gather lists.
Note that this flag requires the SE_LOCK_MEMORY_PRIVILEGE to prevent
malicious non-administrator code from using it to overload the system.

–
Jakob Bøhm, M.Sc.Eng. * xxxxx@danware.dk * direct tel:+45-45-90-25-33
Danware Data A/S * Bregnerodvej 127 * DK-3460 Birkerod * DENMARK
http://www.netop.com * tel:+45-45-90-25-25 * fax tel:+45-45-90-25-26
Information in this mail is hasty, not binding and may not be right