Hi All,
My USB storage device is capable of max transfer size of 512KB. I used the SPTI app to probe for the MaximumTransferLength in STORAGE_ADAPTER_DESCRIPTOR structure. It
was 64kb.
I had written a disk lower filter to override the default 64KB to 512KB. The SPTI now shows MaximumTransferLength as 512 KB.
When i transfer a block size greater than 64KB i see it is always split in multiple of 64KB. My question is when my device is capable of transfering more than 64KB blocks why do the data transfer size always limits to 64KB? Do i need to change any other parameters of STORAGE_ADAPTER_DESCRIPTOR structure.
Regards,
madhu
MaximumPhysicalPages.
xxxxx@hotmail.com
Sent by: xxxxx@lists.osr.com
01/22/2010 07:56 AM
Please respond to
“Windows System Software Devs Interest List”
To
“Windows System Software Devs Interest List”
cc
Subject
[ntdev] MaximumTransferLength in Storage Adapter
Hi All,
My USB storage device is capable of max transfer size of 512KB. I used the
SPTI app to probe for the MaximumTransferLength in
STORAGE_ADAPTER_DESCRIPTOR structure. It
was 64kb.
I had written a disk lower filter to override the default 64KB to 512KB.
The SPTI now shows MaximumTransferLength as 512 KB.
When i transfer a block size greater than 64KB i see it is always split in
multiple of 64KB. My question is when my device is capable of transfering
more than 64KB blocks why do the data transfer size always limits to 64KB?
Do i need to change any other parameters of STORAGE_ADAPTER_DESCRIPTOR
structure.
Regards,
madhu
—
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
Thanks for the quick reply.
I had already tried changing the default MaximumPhysicalPages. But the result was same.
Try to change following fields in PORT_CONFIGURATION_INFORMATION in driver HwScsiFindAdapter routine: DeviceExtensionSize, SpecificLuExtensionSize and SrbExtensionSize.
http://msdn.microsoft.com/en-us/library/ms810317.aspx
Igor Sharovar
>Try to change following fields in PORT_CONFIGURATION_INFORMATION in driver
How can i access PORT_CONFIGURATION_INFORMATION in the filter driver. Is there any way i can get access to these fields.
madhu
xxxxx@hotmail.com wrote:
My USB storage device is capable of max transfer size of 512KB. I used the SPTI app to probe for the MaximumTransferLength in STORAGE_ADAPTER_DESCRIPTOR structure. It
was 64kb.
I had written a disk lower filter to override the default 64KB to 512KB. The SPTI now shows MaximumTransferLength as 512 KB.
When i transfer a block size greater than 64KB i see it is always split in multiple of 64KB. My question is when my device is capable of transfering more than 64KB blocks why do the data transfer size always limits to 64KB? Do i need to change any other parameters of STORAGE_ADAPTER_DESCRIPTOR structure.
The basic reason is because there’s no benefit in going any larger. A
64KB transfer is going to take a couple of milliseconds, so the
reduction in overhead by going to 512KB is not measurable.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
Sorry, I didn’t read your message carefully. I gave you information which is related to a mini-port SCSI(Storport) driver not to a disk lower filter of USB storage device.
Igor Sharovar
>The basic reason is because there’s no benefit in going any larger. A 64KB transfer is going to >take a couple of milliseconds, so the reduction in overhead by going to 512KB is not measurable.
But there should be a way to edit the properties of transfer size even though the reduction in overhead is not measurable.
We have a competetor who had observed significant change by going through these changes. Although his device support max transfer length of upto 1MB.
madhu
>you can’t. How would that work? The hardware driver has said “the largest block I can handle is >64 k”, and you want to hand it a 512k block. What is the hardware driver supposed to do with that >data?
In fact the hardware driver support 512K block. But the max block size i see in filter driver is 64K.
Have you set MaximumSGList in the registry key for your driver? The range on
that value is 16 to 255. Using that value you will be able to build an SG
list with a maximum of 255 elements, or about 1meg.
Seach for it in the index of the WDK documentaion, it will tell you the
registry path to use, and provide a few other values.
Gary G. Little
H (952) 223-1349
C (952) 454-4629
xxxxx@comcast.net
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@hotmail.com
Sent: Friday, January 22, 2010 12:23 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] MaximumTransferLength in Storage Adapter
you can’t. How would that work? The hardware driver has said “the largest
block I can handle is >64 k”, and you want to hand it a 512k block. What is
the hardware driver supposed to do with that >data?
In fact the hardware driver support 512K block. But the max block size i see
in filter driver is 64K.
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
__________ Information from ESET Smart Security, version of virus signature
database 4797 (20100122) __________
The message was checked by ESET Smart Security.
http://www.eset.com
__________ Information from ESET Smart Security, version of virus signature
database 4797 (20100122) __________
The message was checked by ESET Smart Security.
http://www.eset.com
You should do tests using large pages, which will allow much larger
transfers without exceeding the maximum number of fragments. Each large page
is 2 Mbytes as I remember (PAE mode). An app can allocate large pages with
VirtualAlloc and the right flags, and you have to give the app correct
privleges.
Jan
I had already tried changing the default MaximumPhysicalPages. But the
result was same.
> Have you set MaximumSGList in the registry key for your driver? The range on that value is 16 to > 255. Using that value you will be able to build an SG list with a maximum of 255 elements, or >about 1meg.
Do you mean i need to set this for the filter driver. I searched complete registry but did not find such entry.
You should do tests using large pages, which will allow much larger transfers without exceeding > the maximum number of fragments. Each large page is 2 Mbytes as I remember (PAE mode). An > app can allocate large pages with VirtualAlloc and the right flags, and you have to give the app >correct privleges.
I had a test app which trasfer a block size of 512kb. I was also able to see the transfer buffer of 512kb in filter driver. But there was a write error after trying for few seconds. I’m sure if i this is device FW problem. May be i need some configuration in USB stack level.
As I already mentioned we have competetor device which supports 1mb transfer. I had notice he had used a two filters, disk upper and usbstor lower filter to achieve this performance. He had used single driver acting as upper and lower filter.
I’m not sure the need of two filters to achieve this.
Regards,
madhu
By default USBSTOR.SYS sets the IOCTL_STORAGE_QUERY_PROPERTY StorageAdapterProperty STORAGE_DEVICE_DESCRIPTOR and the IOCTL_SCSI_GET_CAPABILITIES IO_SCSI_CAPABILITIES MaximumTransferLength fields to 64KB and the MaximumPhysicalPages fields to 17.
The 64KB limit was the original limit back in the Windows 2000 timeframe. At some point the default limit was increased only to find that it broke a large number of existing devices. Without any reliable means to discover the device specific transfer size limits the default limit was reverted back to 64KB.
Even if a filter driver modified the MaximumTransferLength and MaximumPhysicalPages fields returned by USBSTOR.SYS, I believe it should fail requests with STATUS_INVALID_PARAMETER if they exceed the internal MaximumTransferLength value used by USBSTOR.SYS. I don’t know how you could change that behavior with a filter driver.
There is a device instance specific USBSTOR.SYS registry flag that will change the MaximumTransferLength limit to 128KB and the MaximumPhysicalPages limit to 33. There is no existing mechanism to set these limits to arbitrary values.
-Glen
USBSTOR.SYS 128KB MaximumTransferLength registy setting:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\usbstor\VVVVPPPP]
“DeviceHackFlags”=dword:00000008
Where VVVV = Device Descriptor idVendor in HEX
Where PPPP = Device Descriptor idProduct in HEX
* Applies to Windows XP Vista and above.
* Only increases the transfer size from default 64K to 128K.
* Support is limited to BOT devices only. There are no guarantees that we will continue supporting this setting on UASP devices.
* Thorough testing of the target device is required in order to ensure that it can tolerate the increased transfer size.
* In the future, this registry setting shall be superseded by any other mechanism we introduce for the purpose of establishing the maximum transfer size. The new mechanism will take precedence.
Sorry, that was an editing error. When I wrote this:
“* Applies to Windows XP Vista and above.”
I really meant to write this:
“* Applies to Windows Vista and above.”
The USBSTOR.SYS MaximumTransferLength is hard coded at 64KB in Windows 2000 and Windows XP. There is no built in mechansim to allow that to be changed.
The USBSTOR.SYS 128KB MaximumTransferLength registry setting only applies to Windows Vista and Windows 7.
-Glen
Interesting a 16-bit Dos limit raised its head in USB which of course is a much more recent standard. One would have thought there might be registry hacks only for the broken devices rather than punishing the working ones. Also surprising the tweek to relax the limit merely doubles the size rather than makes for unlimited like other buses. I hope the new solution goes further.
The max transfer on storage in general needs fixing as well which is the MDL ceiling, which actually reduced the maximum transfer limit in 64-bit Windows compared to 32-bit. Tape drives for instance can capitalize on large transfers which already constrained in 32-bits have become all but impractical in 64-bits.
The original 64KB MaximumTransferLength limit in USBSTOR.SYS had nothing to do with it being a 16-bit limit nor a DOS limit. Some limit needed to be chosen and 64KB was picked as a nice round limit that was somewhat arbitrary and somewhat reasonable at the time when the most common USB storage device might have been the full speed USB Zip drive or USB floppy drive. If the host controller and device were able to move 16 64-byte bulk packets (or 1KB) per frame that would take 64 frames to complete a 64KB transfer. If the overhead per request were 2 frames and the transfer size were increased to 512KB that would be around 3% faster. (If I had time I could check these figures with a bus analyzer and real devices from the Windows 2000 timeframe).
512KB transfer at 512KB MaximumTransferLength:
2 frames overhead + 512 data frames = 514 frames total
512KB transfer at 64KB MaximumTransferLength:
16 frames overhead + 512 data frames = 528 frames total
If the 100MB USB Zip drive could transfer approximately 1MB per second and the transfer rate was improved approximately 3% then that?s only a difference of approximately 3 seconds to read or write the entire disk. Sure a 3% difference could add up, but at the time of 100MB USB Zip drives and transfer rates of 1MB/second people were happy it just worked and had no idea in a few years multi-GB USB flash keys and TB USB hard drives would be everywhere.
When USB 2.0 arrived and larger and faster USB storage drives arrived the MaximumTransferLength limit in USBSTOR.SYS was revisited, but it was found that a large number of existing devices failed with transfer sizes greater than 64KB. The approach of breaking large numbers of existing devices and generating endless customer complaints so that some devices could possibly operate marginally faster is a rather foolish idea in the real world. The approach of collecting a list of all “bad” exception devices doesn’t work as it could never be complete enough by the time the list needed to be finalized for product RTM.
The real problem is that the USB Mass Storage Class specification provides no standard mechanism to reliably discover the maximum transfer size supported by a USB storage device and no one thought about that issue at the time the specification was written.
All that said, I agree that it would have been better to allow the MaximumTransferLength limit be specified as a registry variable instead of a fixed increase to 128KB. I am somewhat curious what the real world performance difference is between a limit of 64KB and a limit of 512KB for typical high speed USB storage devices which correctly handle transfers of that size. Increasing the MaximumTransferLength limit above 512KB might not be any better than a limit of 512KB.
-Glen
Thank you for the detailed explanation
* In the future, this registry setting shall be superseded by any other
mechanism we introduce for the purpose of establishing the maximum transfer
size. The new mechanism will take precedence.
As you said a new mechanism is being proposed. I suggest not to restrict the maximum transfer size to 128KB. Even though the performance achieved is minor it should be easily set by the user.
Regards,
madhu
> Interesting a 16-bit Dos limit
This is also the ATA limit I think.
–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com