USB Iso low latency - modify buffer after sending write URB?

NTDEV
1

Hello all,

I am developing a KMDF loopback driver using high speed iso transfer (the data stream originates on hardware, gets read in my driver and is written back to hardware). I am experimenting with ways to achieve minimum latency.

My question: is there a way to modify the contents of PURB->UrbIsochronousTransfer.TransferBuffer in between the WdfRequestSend and the EvtRequestCompletionRoutine for a write operation? I’ve tried this, but modifying the write URB’s data buffer after submitting the request hasn’t seemed to have any effect.

I’ve been trying to get the following sequence:

  • Queue a bunch of read URB’s and a bunch of write URB’s, specifying some latency between each read/write pair.

  • On read URB completion routine, fill in corresponding, already-submitted write URB’s TransferBuffer contents, (a little more complicated than this, I use an overflow buffer since my driver must adapt to a hardware dictated transmission rate). Note that I am not trying to change anything in the write URB struct itself, e.g. the IsoPacket[N].Offsets or Lengths; rather, I just want to modify the contents of its TransferBuffer.

  1. Is the above even possible?

  2. Is this a bad idea for other reasons, e.g. is URB write request submission so fast that this avenue of optimization doesn’t even buy me anything?

(I am already using WdfRequestReuse, which certainly helps streamline things.)

Regards,
Andrew

2

On Sun, Mar 22, 2009 at 11:32:45PM -0400, xxxxx@yahoo.com wrote:

Hello all,

I am developing a KMDF loopback driver using high speed iso transfer
(the data stream originates on hardware, gets read in my driver and
is written back to hardware). I am experimenting with ways to achieve
minimum latency.

At the risk of jumping to conclusions, I suspect you are wasting your time.
Remember that USB is a scheduled bus. When you submit a request, it
doesn't happen immediately. Instead, it goes into the queue of requests
that will be scheduled for the NEXT frame. There's no point in breaking
a sweat to hyperoptimize the turnaround from an input request to an output
request, because you will be at the mercy of frame scheduling anyway.

My question: is there a way to modify the contents of
PURB->UrbIsochronousTransfer.TransferBuffer in between the WdfRequestSend
and the EvtRequestCompletionRoutine for a write operation?

No. Once you call WdfRequestSend, the request and the buffers no longer
belong to you. The buffers will eventually be entered into the USB
host controller's DMA queue, and you cannot predict when that will
actually happen.

I've been trying to get the following sequence:

  • Queue a bunch of read URB's and a bunch of write URB's, specifying some
    latency between each read/write pair.

  • On read URB completion routine, fill in corresponding,
    already-submitted write URB's TransferBuffer contents, (a little more
    complicated than this, I use an overflow buffer since my driver must
    adapt to a hardware dictated transmission rate). Note that I am not
    trying to change anything in the write URB struct itself, e.g. the
    IsoPacket[N].Offsets or Lengths; rather, I just want to modify the
    contents of its TransferBuffer.

  1. Is the above even possible?

No. You need to follow the contract, and the scheme you describe violates
the contract.

  1. Is this a bad idea for other reasons, e.g. is URB write request
    submission so fast that this avenue of optimization doesn't even buy
    me anything?

Well, it's the USB scheduling that means it won't buy you anything. By the
time you get a read request back, you will already have missed the
scheduling for the next microframe, so there will always be at least a
1 microframe gap between your read requests and your write resposnes.

Tim Roberts, xxxxx@probo.com
Providenza & Boeklheide, Inc.

3

> The buffers will eventually be entered into the USB host controller’s

DMA queue, and you cannot predict when that will actually happen.

I bet that’s why changing the buffer seemed to have no effect.

No. You need to follow the contract, and the scheme you describe
violates the contract.

Thanks very much, Tim, that’s what I needed to know.

4

On second thought… I just found the following in the Mac documentation:


GetLowLatencyOptionsAndPhysicalMask

Low Latency transfers require that the client have access to the memory after the Isochronous I/O request has already been scheduled. This might be used, for example to fill in outgoing data “just in time.” Some controllers, however, may have requirements which need to be followed in order to make sure that the memory buffer isn’t moved after the call is made. This call will return an IOOptionBits and mach_vm_address_t which can be used in a call to IOBufferMemoryDescriptor::inTaskWithPhysicalMask which will help meet these requirements.

------------------------------------

… which reminds me of this in Microsoft’s documentation under USBD_PIPE_INFORMATION:

Flag Name Meaning
…
USBD_PF_SHORT_PACKET_OPT Indicates that the pipe should be optimized for short packets.
USBD_PF_ENABLE_RT_THREAD_ACCESS Indicates that the pipe should be optimized for streaming transfers.
USBD_PF_MAP_ADD_TRANSFERS Indicates that the pipe should operate in fast isochronous mode.

I’m particularly interested in USBD_PF_ENABLE_RT_THREAD_ACCESS. But after googling and searching Microsoft’s documentation and samples, I have been unable to find out more of a detailed explanation of what each of the above USBD_PDF_ options mean. Does anybody know if they have anything to do with low latency iso transfer, possibly equivalent to the Mac functionality above?

Are they flags that affect how the Microsoft-written portions of the USB stack behave, or are they more flags for sort of user-defined, custom driver behavior?

Also, are these pipe configuration flags available when using the KMDF multi-interface USB configuration routine?

5

xxxxx@yahoo.com wrote:

On second thought… I just found the following in the Mac documentation:


> GetLowLatencyOptionsAndPhysicalMask
>
> Low Latency transfers require that the client have access to the memory after the Isochronous I/O request has already been scheduled. This might be used, for example to fill in outgoing data “just in time.” Some controllers, however, may have requirements which need to be followed in order to make sure that the memory buffer isn’t moved after the call is made. This call will return an IOOptionBits and mach_vm_address_t which can be used in a call to IOBufferMemoryDescriptor::inTaskWithPhysicalMask which will help meet these requirements.
>
> ------------------------------------
>

Interesting. Windows doesn’t provide this, nor does Linux. Indeed, it
is hard for me to imagine how this could be actually be implemented in a
non-real-time operating system. How would you know that the host
controller hardware had not started reading the buffer? You would need
some kind of interlock to force the hardware to stall while you filled
in the data, and I don’t think there is any such thing.

> … which reminds me of this in Microsoft’s documentation under USBD_PIPE_INFORMATION:
>
> Flag Name Meaning
> …
> USBD_PF_SHORT_PACKET_OPT Indicates that the pipe should be optimized for short packets.
> USBD_PF_ENABLE_RT_THREAD_ACCESS Indicates that the pipe should be optimized for streaming transfers.
> USBD_PF_MAP_ADD_TRANSFERS Indicates that the pipe should operate in fast isochronous mode.
>
> I’m particularly interested in USBD_PF_ENABLE_RT_THREAD_ACCESS. But after googling and searching Microsoft’s documentation and samples, I have been unable to find out more of a detailed explanation of what each of the above USBD_PDF_ options mean. Does anybody know if they have anything to do with low latency iso transfer, possibly equivalent to the Mac functionality above?
>

In 13 years of Windows USB work, I’ve never encountered these flags. We
will have to hope someone with source code access is listening.

> Also, are these pipe configuration flags available when using the KMDF multi-interface USB configuration routine?

A very interesting question. As far as I know,
WdfUsbTargetDeviceSelectConfig doesn’t use the USBD_PIPE_INFORMATION
structure, and without that structure, I don’t know how you’d pass in
those flags. That may be a sign that they’ve never really been used
from the beginning.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

6

> Interesting. Windows doesn’t provide this, nor does Linux.

Indeed, it is hard for me to imagine how this could be
actually be implemented in a non-real-time operating system.

I am guessing that under that setting, the Mac doesn’t copy the buffers until “the last minute”. I am also guessing that developing drivers for this operating mode is an inexact science where the results vary a lot by specific software environment and hardware capabilities. But that’s sort of the nature of the problem with attempting low latency on non-real time operating systems, right?

A very interesting question. As far as I know,
WdfUsbTargetDeviceSelectConfig doesn’t use the
USBD_PIPE_INFORMATION structure, and without
that structure, I don’t know how you’d pass in
those flags. That may be a sign that they’ve never
really been used from the beginning.

There is an URB request function URB_FUNCTION_SELECT_INTERFACE that uses the same structure. What I think is the Wdf equivalent, WdfUsbInterfaceSelectSetting, doesn’t seem to provide any option for PipeFlags.

I would be very interested to hear the Microsoft scoop on this. I’ve seen about 3 or 4 questions about these USBD_PF_ flags out there that seem to have remained unanswered, just from googling them.

Regards,
Andrew

7

None of these flags are supported on vista or win7. It looks like they have minimal support on server 2003 and previous releases, but they were never documented so I would not rely on the support nor what you perceive to be their behavior.

You can specify your own flags for select config or select interface by building your own URB and using WDF_USB_DEVICE_SELECT_CONFIG_PARAMS_INIT_URB/WDF_USB_INTERFACE_SELECT_SETTING_PARAMS_INIT_URB to init the config params structure

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@yahoo.com
Sent: Monday, March 23, 2009 12:09 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] USB Iso low latency - modify buffer after sending write URB?

Interesting. Windows doesn’t provide this, nor does Linux.
Indeed, it is hard for me to imagine how this could be
actually be implemented in a non-real-time operating system.

I am guessing that under that setting, the Mac doesn’t copy the buffers until “the last minute”. I am also guessing that developing drivers for this operating mode is an inexact science where the results vary a lot by specific software environment and hardware capabilities. But that’s sort of the nature of the problem with attempting low latency on non-real time operating systems, right?

A very interesting question. As far as I know,
WdfUsbTargetDeviceSelectConfig doesn’t use the
USBD_PIPE_INFORMATION structure, and without
that structure, I don’t know how you’d pass in
those flags. That may be a sign that they’ve never
really been used from the beginning.

There is an URB request function URB_FUNCTION_SELECT_INTERFACE that uses the same structure. What I think is the Wdf equivalent, WdfUsbInterfaceSelectSetting, doesn’t seem to provide any option for PipeFlags.

I would be very interested to hear the Microsoft scoop on this. I’ve seen about 3 or 4 questions about these USBD_PF_ flags out there that seem to have remained unanswered, just from googling them.

Regards,
Andrew

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

8

xxxxx@yahoo.com wrote:

> Interesting. Windows doesn’t provide this, nor does Linux.
> Indeed, it is hard for me to imagine how this could be
> actually be implemented in a non-real-time operating system.
>

I am guessing that under that setting, the Mac doesn’t copy the buffers until “the last minute”.

There shouldn’t be any copies at all. All of the host controller
drivers are heavily scatter/gather DMA driven; the hardware reads the
user’s pages directly.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

9

> None of these flags are supported on vista or win7.

It looks like they have minimal support on server
2003 and previous releases, but they were never
documented so I would not rely on the support
nor what you perceive to be their behavior.

Thank you, Doron.