USB mysteries

We have an USB 1.1 device with two bulk, one interrupt and one control endpoint. The device has several registers accesible through control endpoint using vendor request URB (URB_FUNCTION_VENDOR_DEVICE). In one mode device generates stream of data which host application should read using bulk endpoint as fast as possible. The endpoint frame size is 64 bytes and frames are numbered so it can be detected is some are lost. Data rate is rather high, about 900 kB/s and device has very limited circular buffer where unread data are replaced. To avoid data loss the reading thread (user mode) sends to the kernel several overlapped reads and when a request is completed, copies data and resends it back so there is always several pending reads. It works as expected with an exception described later. As for drivers, at XP and above my custom driver based on BulkUsb sample is used and at obsolete OSes system supplied UsbScan driver is used. I also tested UsbScan at XP with equal results.

The exception occurs if I use 64 bytes (one bulk frame) buffers for reading. Sometimes the read request is completed successfully but with zero data. The probability is about 3- 10 : 10^6 and decreases when data rate is lowered (for data rate ~ 650 kB/s it disappears). I verified it at driver level; completion function for _URB_BULK_OR_INTERRUPT_TRANSFER is called with STATUS_SUCCESS but with zero in UrbBulkOrInterruptTransfer.TransferBufferLength. Appropriate frame number is missing in the stream then so it seems data weren’t read quickly enough or were read but weren’t copied to buffer. The problem completely disappears if read request are 128 bytes or more long. It can’t be reproduced even at highest possible data rate (several MB/s which is more than USB can transfer).

I’d understand if there is an error but why success? Now I see I should probably examine _URB_HEADER.Status but if there is some error, UsbScan ignores it, too.

Next thing I don’t understand is requests serialization. To avoid data loss at the beginning, I wanted to post several pending reads to kernel and then write device register to tell it to start generate data. Surprisingly, it doesn’t work. Control channel request is blocked until pending bulk read is completed. To be quite clear, driver sends several _URB_BULK_OR_INTERRUPT_TRANSFER URBs and then one _URB_CONTROL_VENDOR_OR_CLASS_REQUEST with URB_FUNCTION_VENDOR_DEVICE command. I’d expect control and bulk endpoints are serviced independently but instead, everything is blocked. When I reset device, I noticed the first bulk URB is completed with error, then control endpoint URB and then the rest of bulk URBs. It seems as the first bulk URB is sent to hw and blocks subsequent requests. I’m probably missing something here, can anybody explain it?

(tested at XP SP2)

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

URB status is 0 for zero data reads and problem occurs even if USBD_SHORT_TRANSFER_OK is removed from transfer flags.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]


From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Michal Vodicka[SMTP:xxxxx@upek.com]
Reply To: Windows System Software Devs Interest List
Sent: Thursday, February 10, 2005 5:47 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] USB mysteries

We have an USB 1.1 device with two bulk, one interrupt and one control endpoint. The device has several registers accesible through control endpoint using vendor request URB (URB_FUNCTION_VENDOR_DEVICE). In one mode device generates stream of data which host application should read using bulk endpoint as fast as possible. The endpoint frame size is 64 bytes and frames are numbered so it can be detected is some are lost. Data rate is rather high, about 900 kB/s and device has very limited circular buffer where unread data are replaced. To avoid data loss the reading thread (user mode) sends to the kernel several overlapped reads and when a request is completed, copies data and resends it back so there is always several pending reads. It works as expected with an exception described later. As for drivers, at XP and above my custom driver based on BulkUsb sample is used and at obsolete OSes system supplied UsbScan driver is used. I also tested UsbScan at XP with equal results.

The exception occurs if I use 64 bytes (one bulk frame) buffers for reading. Sometimes the read request is completed successfully but with zero data. The probability is about 3- 10 : 10^6 and decreases when data rate is lowered (for data rate ~ 650 kB/s it disappears). I verified it at driver level; completion function for _URB_BULK_OR_INTERRUPT_TRANSFER is called with STATUS_SUCCESS but with zero in UrbBulkOrInterruptTransfer.TransferBufferLength. Appropriate frame number is missing in the stream then so it seems data weren’t read quickly enough or were read but weren’t copied to buffer. The problem completely disappears if read request are 128 bytes or more long. It can’t be reproduced even at highest possible data rate (several MB/s which is more than USB can transfer).

I’d understand if there is an error but why success? Now I see I should probably examine _URB_HEADER.Status but if there is some error, UsbScan ignores it, too.

Next thing I don’t understand is requests serialization. To avoid data loss at the beginning, I wanted to post several pending reads to kernel and then write device register to tell it to start generate data. Surprisingly, it doesn’t work. Control channel request is blocked until pending bulk read is completed. To be quite clear, driver sends several _URB_BULK_OR_INTERRUPT_TRANSFER URBs and then one _URB_CONTROL_VENDOR_OR_CLASS_REQUEST with URB_FUNCTION_VENDOR_DEVICE command. I’d expect control and bulk endpoints are serviced independently but instead, everything is blocked. When I reset device, I noticed the first bulk URB is completed with error, then control endpoint URB and then the rest of bulk URBs. It seems as the first bulk URB is sent to hw and blocks subsequent requests. I’m probably missing something here, can anybody explain it?

(tested at XP SP2)

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]


Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Michal Vodicka wrote:

The exception occurs if I use 64 bytes (one bulk frame) buffers for reading. Sometimes the read request is completed successfully but with zero data. The probability is about 3- 10 : 106 and decreases when data rate is lowered (for data rate ~ 650 kB/s it disappears). … The problem completely disappears if read request are 128 bytes or more long. It can’t be reproduced even at highest possible data rate (several MB/s which is more than USB can transfer).

So, you are trying to transfer 64 bytes per URB? The overhead for that
is enormous. Have you checked the CPU load when this is running? That
URB/IRP has to make a round-trip in to your driver and back to USBD
every 60 microseconds. You need to pack multiple frames in each URB.

Next thing I don’t understand is requests serialization. To avoid data loss at the beginning, I wanted to post several pending reads to kernel and then write device register to tell it to start generate data. Surprisingly, it doesn’t work. Control channel request is blocked until pending bulk read is completed. To be quite clear, driver sends several _URB_BULK_OR_INTERRUPT_TRANSFER URBs and then one _URB_CONTROL_VENDOR_OR_CLASS_REQUEST with URB_FUNCTION_VENDOR_DEVICE command. I’d expect control and bulk endpoints are serviced independently but instead, everything is blocked. When I reset device, I noticed the first bulk URB is completed with error, then control endpoint URB and then the rest of bulk URBs. It seems as the first bulk URB is sent to hw and blocks subsequent requests. I’m probably missing something here, can anybody explain it?

A transaction must either complete or abort before another transaction
occurs for that device. When your device returns a NAK, the host will
retry that transaction forever, until it succeeds or aborts, and will
not move on to another. In your case, you should probably have your
device return a good packet that says “no data right now”. As long as
you have a bunch of read URBs queued up, you won’t lose any data.

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Tim Roberts[SMTP:xxxxx@probo.com]
Reply To: Windows System Software Devs Interest List
Sent: Thursday, February 10, 2005 7:16 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] USB mysteries

So, you are trying to transfer 64 bytes per URB? The overhead for that is enormous. Have you checked the CPU load when this is running? That URB/IRP has to make a round-trip in to your driver and back to USBD every 60 microseconds. You need to pack multiple frames in each URB.

Yes, sure. I plan to use 64 frames per URB i.e. 4096 bytes. I tried 1 frame per request just for test purposes and was surprised with this problem. It seems there is something wrong and I’d like to know what. If the problem doesn’t occurs with multiple-frames request just now doesn’t mean it won’t occur in the future.

A transaction must either complete or abort before another transaction occurs for that device. When your device returns a NAK, the host will retry that transaction forever, until it succeeds or aborts, and will not move on to another.

OK but I thought endpoints are serviced separately. I.e. transaction at bulk endpoint shouldn’t block control endpoint transaction. Apparently, I’m wrong but I’d like to know if it is because of USB specification or MS USB drivers limitation.

In your case, you should probably have your device return a good packet that says “no data right now”. As long as you have a bunch of read URBs queued up, you won’t lose any data.

Unfortunately, it isn’t possible. The device is “dumb” with no firmware which was moved to host. Not a big problem, thought, we can live with data loss at the beginning. I just was surprised about it and curious.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

Michal Vodicka wrote:

>A transaction must either complete or abort before another transaction occurs for that device. When your device returns a NAK, the host will retry that transaction forever, until it succeeds or aborts, and will not move on to another.
>
>
OK but I thought endpoints are serviced separately. I.e. transaction at bulk endpoint shouldn’t block control endpoint transaction. Apparently, I’m wrong but I’d like to know if it is because of USB specification or MS USB drivers limitation.

Well, I went back to the USB 2.0 specification, and it is not as clear
as I had hoped. In most cases, it does says that NAK means the
“endpoint” is busy and can accept no more data. In other places,
however, it says “device”.

For the first problem, the timing factor makes me believe that your
device’s buffer is emptying when you request 64 byte requests. At 900
kB/s, you are pretty much maxing out the bandwidth of USB 1.1. There is
a high cost of scheduling a transfer on the bus and completing it. This
sometimes can actually cause the host software to need to pause the
schedule all together (thus losing some bandwidth). So, I don’t think
you will be able to keep up with the device by issuing 64 byte requests,
no matter how many you pend. The bigger the request, the less the
overhead will affect you, which is why 128 byte reads works better. I
don’t think this is a USB stack issue, I think your device is out-pacing
your read algorithm.

For the second problem, there is no serialization between control and
bulk endpoints in the USB stack. Obviously there is some serialization
in the host controller, as only one transfer can happen at a time, but
this is packet serialization, not transfer serialization. So, as soon
as a packet gets NAK’ed or ACK’ed, the host controller gives the next
endpoint a shot. There are a lot of protocols that have always had
reads pending on one endpoint while doing other IO on different
endpoints. So, my guess is that your device is NAKing the control
transfer.

My recommendation would be to get a USB Bus analyzer and looking at your
device’s behavior. I think that will illuminate the problem.

Randy


From: xxxxx@lists.osr.com on behalf of Michal Vodicka
Sent: Wed 2/9/2005 8:47 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] USB mysteries

We have an USB 1.1 device with two bulk, one interrupt and one control endpoint. The device has several registers accesible through control endpoint using vendor request URB (URB_FUNCTION_VENDOR_DEVICE). In one mode device generates stream of data which host application should read using bulk endpoint as fast as possible. The endpoint frame size is 64 bytes and frames are numbered so it can be detected is some are lost. Data rate is rather high, about 900 kB/s and device has very limited circular buffer where unread data are replaced. To avoid data loss the reading thread (user mode) sends to the kernel several overlapped reads and when a request is completed, copies data and resends it back so there is always several pending reads. It works as expected with an exception described later. As for drivers, at XP and above my custom driver based on BulkUsb sample is used and at obsolete OSes system supplied UsbScan driver is used. I also tested UsbScan at XP with equal results.

The exception occurs if I use 64 bytes (one bulk frame) buffers for reading. Sometimes the read request is completed successfully but with zero data. The probability is about 3- 10 : 10^6 and decreases when data rate is lowered (for data rate ~ 650 kB/s it disappears). I verified it at driver level; completion function for _URB_BULK_OR_INTERRUPT_TRANSFER is called with STATUS_SUCCESS but with zero in UrbBulkOrInterruptTransfer.TransferBufferLength. Appropriate frame number is missing in the stream then so it seems data weren’t read quickly enough or were read but weren’t copied to buffer. The problem completely disappears if read request are 128 bytes or more long. It can’t be reproduced even at highest possible data rate (several MB/s which is more than USB can transfer).

I’d understand if there is an error but why success? Now I see I should probably examine _URB_HEADER.Status but if there is some error, UsbScan ignores it, too.

Next thing I don’t understand is requests serialization. To avoid data loss at the beginning, I wanted to post several pending reads to kernel and then write device register to tell it to start generate data. Surprisingly, it doesn’t work. Control channel request is blocked until pending bulk read is completed. To be quite clear, driver sends several _URB_BULK_OR_INTERRUPT_TRANSFER URBs and then one _URB_CONTROL_VENDOR_OR_CLASS_REQUEST with URB_FUNCTION_VENDOR_DEVICE command. I’d expect control and bulk endpoints are serviced independently but instead, everything is blocked. When I reset device, I noticed the first bulk URB is completed with error, then control endpoint URB and then the rest of bulk URBs. It seems as the first bulk URB is sent to hw and blocks subsequent requests. I’m probably missing something here, can anybody explain it?

(tested at XP SP2)

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]


Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Randy Aull[SMTP:xxxxx@windows.microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Saturday, February 12, 2005 6:16 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

For the first problem, the timing factor makes me believe that your
device’s buffer is emptying when you request 64 byte requests. At 900
kB/s, you are pretty much maxing out the bandwidth of USB 1.1. There is
a high cost of scheduling a transfer on the bus and completing it. This
sometimes can actually cause the host software to need to pause the
schedule all together (thus losing some bandwidth). So, I don’t think
you will be able to keep up with the device by issuing 64 byte requests,
no matter how many you pend. The bigger the request, the less the
overhead will affect you, which is why 128 byte reads works better. I
don’t think this is a USB stack issue, I think your device is out-pacing
your read algorithm.

Yes, but my problem is different. I understand some data are lost when read this way and I tried it just for test purposes. What I don’t understand is why USB stack completes read IRP with STATUS_SUCCESS, URB status 0 and data length 0. I’d understand if there is an error returned. I’d understand if an USB frame if missed which is usual situation. I have about 6 - 7% data loss at this speed using 64 bytes read requests. Frames are just missing, one read frame has number N (generated by device) and next N + 2. It is expected but why there are success zero data reads? As I said, there are about 3 - 10 such reads per million. I’d like to know what does it mean and if I can expect something like this when using longer requests (I plan to use 64 frames per request i.e. 4096 bytes).

For the second problem, there is no serialization between control and
bulk endpoints in the USB stack. Obviously there is some serialization
in the host controller, as only one transfer can happen at a time, but
this is packet serialization, not transfer serialization. So, as soon
as a packet gets NAK’ed or ACK’ed, the host controller gives the next
endpoint a shot. There are a lot of protocols that have always had
reads pending on one endpoint while doing other IO on different
endpoints. So, my guess is that your device is NAKing the control
transfer.

Well, the device is prototype and everything is possible. I can’t check it just now but will try. It is rather hard to believe as it starts to work immediatelly when the first read request is cancelled but maybe device has a problem. I’ll ask hw guys to check.

Anyway, thanks for confirmation it should work as I presumed.

My recommendation would be to get a USB Bus analyzer and looking at your
device’s behavior. I think that will illuminate the problem.

You’re rigth. I just don’t have USB analyzer here and will try it when possible.

Thanks.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

Reads with zero bytes are necessarily failures. In fact it can be fairly common. All the device has to do it ACK with no data. That would be a successful read with length = 0. The only times that there would be a failure (barring PnP events, etc) would be if the device issued a STALL, or the device sent more that MaxPacket bytes or some other USB violation.

My guess is that your device is actually ACKing the IN without any data, thus completing your reads with zero bytes.


From: xxxxx@lists.osr.com on behalf of Michal Vodicka
Sent: Fri 2/11/2005 9:41 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries


From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Randy Aull[SMTP:xxxxx@windows.microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Saturday, February 12, 2005 6:16 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

For the first problem, the timing factor makes me believe that your
device’s buffer is emptying when you request 64 byte requests. At 900
kB/s, you are pretty much maxing out the bandwidth of USB 1.1. There is
a high cost of scheduling a transfer on the bus and completing it. This
sometimes can actually cause the host software to need to pause the
schedule all together (thus losing some bandwidth). So, I don’t think
you will be able to keep up with the device by issuing 64 byte requests,
no matter how many you pend. The bigger the request, the less the
overhead will affect you, which is why 128 byte reads works better. I
don’t think this is a USB stack issue, I think your device is out-pacing
your read algorithm.

Yes, but my problem is different. I understand some data are lost when read this way and I tried it just for test purposes. What I don’t understand is why USB stack completes read IRP with STATUS_SUCCESS, URB status 0 and data length 0. I’d understand if there is an error returned. I’d understand if an USB frame if missed which is usual situation. I have about 6 - 7% data loss at this speed using 64 bytes read requests. Frames are just missing, one read frame has number N (generated by device) and next N + 2. It is expected but why there are success zero data reads? As I said, there are about 3 - 10 such reads per million. I’d like to know what does it mean and if I can expect something like this when using longer requests (I plan to use 64 frames per request i.e. 4096 bytes).

For the second problem, there is no serialization between control and
bulk endpoints in the USB stack. Obviously there is some serialization
in the host controller, as only one transfer can happen at a time, but
this is packet serialization, not transfer serialization. So, as soon
as a packet gets NAK’ed or ACK’ed, the host controller gives the next
endpoint a shot. There are a lot of protocols that have always had
reads pending on one endpoint while doing other IO on different
endpoints. So, my guess is that your device is NAKing the control
transfer.

Well, the device is prototype and everything is possible. I can’t check it just now but will try. It is rather hard to believe as it starts to work immediatelly when the first read request is cancelled but maybe device has a problem. I’ll ask hw guys to check.

Anyway, thanks for confirmation it should work as I presumed.

My recommendation would be to get a USB Bus analyzer and looking at your
device’s behavior. I think that will illuminate the problem.

You’re rigth. I just don’t have USB analyzer here and will try it when possible.

Thanks.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]


Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Actually, that should read “Reads with zero bytes are NOT necessarily failures”.


From: xxxxx@lists.osr.com on behalf of Randy Aull
Sent: Sat 2/12/2005 3:01 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

Reads with zero bytes are necessarily failures. In fact it can be fairly common. All the device has to do it ACK with no data. That would be a successful read with length = 0. The only times that there would be a failure (barring PnP events, etc) would be if the device issued a STALL, or the device sent more that MaxPacket bytes or some other USB violation.

My guess is that your device is actually ACKing the IN without any data, thus completing your reads with zero bytes.


From: xxxxx@lists.osr.com on behalf of Michal Vodicka
Sent: Fri 2/11/2005 9:41 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries


From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Randy Aull[SMTP:xxxxx@windows.microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Saturday, February 12, 2005 6:16 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

For the first problem, the timing factor makes me believe that your
device’s buffer is emptying when you request 64 byte requests. At 900
kB/s, you are pretty much maxing out the bandwidth of USB 1.1. There is
a high cost of scheduling a transfer on the bus and completing it. This
sometimes can actually cause the host software to need to pause the
schedule all together (thus losing some bandwidth). So, I don’t think
you will be able to keep up with the device by issuing 64 byte requests,
no matter how many you pend. The bigger the request, the less the
overhead will affect you, which is why 128 byte reads works better. I
don’t think this is a USB stack issue, I think your device is out-pacing
your read algorithm.

Yes, but my problem is different. I understand some data are lost when read this way and I tried it just for test purposes. What I don’t understand is why USB stack completes read IRP with STATUS_SUCCESS, URB status 0 and data length 0. I’d understand if there is an error returned. I’d understand if an USB frame if missed which is usual situation. I have about 6 - 7% data loss at this speed using 64 bytes read requests. Frames are just missing, one read frame has number N (generated by device) and next N + 2. It is expected but why there are success zero data reads? As I said, there are about 3 - 10 such reads per million. I’d like to know what does it mean and if I can expect something like this when using longer requests (I plan to use 64 frames per request i.e. 4096 bytes).

For the second problem, there is no serialization between control and
bulk endpoints in the USB stack. Obviously there is some serialization
in the host controller, as only one transfer can happen at a time, but
this is packet serialization, not transfer serialization. So, as soon
as a packet gets NAK’ed or ACK’ed, the host controller gives the next
endpoint a shot. There are a lot of protocols that have always had
reads pending on one endpoint while doing other IO on different
endpoints. So, my guess is that your device is NAKing the control
transfer.

Well, the device is prototype and everything is possible. I can’t check it just now but will try. It is rather hard to believe as it starts to work immediatelly when the first read request is cancelled but maybe device has a problem. I’ll ask hw guys to check.

Anyway, thanks for confirmation it should work as I presumed.

My recommendation would be to get a USB Bus analyzer and looking at your
device’s behavior. I think that will illuminate the problem.

You’re rigth. I just don’t have USB analyzer here and will try it when possible.

Thanks.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]


Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com


Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Randy,

thanks for you anwers. You evidently believe problems are caused by the device which may or may not be right. Anyway, the only way how to prove or disprove it is using USB analyzer.

I already asked our hw guys and they claim control endpoint can’t be ever NAKed (serialization problem). It is also hard to believe device would cause zero data reads as you describe but we will check it.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]


From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Randy Aull[SMTP:xxxxx@windows.microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Sunday, February 13, 2005 2:25 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

Actually, that should read “Reads with zero bytes are NOT necessarily failures”.


From: xxxxx@lists.osr.com on behalf of Randy Aull
Sent: Sat 2/12/2005 3:01 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

Reads with zero bytes are necessarily failures. In fact it can be fairly common. All the device has to do it ACK with no data. That would be a successful read with length = 0. The only times that there would be a failure (barring PnP events, etc) would be if the device issued a STALL, or the device sent more that MaxPacket bytes or some other USB violation.

My guess is that your device is actually ACKing the IN without any data, thus completing your reads with zero bytes.


From: xxxxx@lists.osr.com on behalf of Michal Vodicka
Sent: Fri 2/11/2005 9:41 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

> ----------
> From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Randy Aull[SMTP:xxxxx@windows.microsoft.com]
> Reply To: Windows System Software Devs Interest List
> Sent: Saturday, February 12, 2005 6:16 AM
> To: Windows System Software Devs Interest List
> Subject: RE: [ntdev] USB mysteries
>
> For the first problem, the timing factor makes me believe that your
> device’s buffer is emptying when you request 64 byte requests. At 900
> kB/s, you are pretty much maxing out the bandwidth of USB 1.1. There is
> a high cost of scheduling a transfer on the bus and completing it. This
> sometimes can actually cause the host software to need to pause the
> schedule all together (thus losing some bandwidth). So, I don’t think
> you will be able to keep up with the device by issuing 64 byte requests,
> no matter how many you pend. The bigger the request, the less the
> overhead will affect you, which is why 128 byte reads works better. I
> don’t think this is a USB stack issue, I think your device is out-pacing
> your read algorithm.
>
Yes, but my problem is different. I understand some data are lost when read this way and I tried it just for test purposes. What I don’t understand is why USB stack completes read IRP with STATUS_SUCCESS, URB status 0 and data length 0. I’d understand if there is an error returned. I’d understand if an USB frame if missed which is usual situation. I have about 6 - 7% data loss at this speed using 64 bytes read requests. Frames are just missing, one read frame has number N (generated by device) and next N + 2. It is expected but why there are success zero data reads? As I said, there are about 3 - 10 such reads per million. I’d like to know what does it mean and if I can expect something like this when using longer requests (I plan to use 64 frames per request i.e. 4096 bytes).

> For the second problem, there is no serialization between control and
> bulk endpoints in the USB stack. Obviously there is some serialization>
> in the host controller, as only one transfer can happen at a time, but
> this is packet serialization, not transfer serialization. So, as soon
> as a packet gets NAK’ed or ACK’ed, the host controller gives the next
> endpoint a shot. There are a lot of protocols that have always had
> reads pending on one endpoint while doing other IO on different
> endpoints. So, my guess is that your device is NAKing the control
> transfer.
>
Well, the device is prototype and everything is possible. I can’t check it just now but will try. It is rather hard to believe as it starts to work immediatelly when the first read request is cancelled but maybe device has a problem. I’ll ask hw guys to check.

Anyway, thanks for confirmation it should work as I presumed.

> My recommendation would be to get a USB Bus analyzer and looking at your
> device’s behavior. I think that will illuminate the problem.
>
You’re rigth. I just don’t have USB analyzer here and will try it when possible.

Thanks.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http:]
>
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at http:
>
> You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
>
> —
> Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
> To unsubscribe send a blank email to xxxxx@lists.osr.com
> —
> Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
> To unsubscribe send a blank email to xxxxx@lists.osr.com
></http:></http:>

Michal Vodicka wrote:

Randy,

thanks for you anwers. You evidently believe problems are caused by the device which may or may not be right. Anyway, the only way how to prove or disprove it is using USB analyzer.

I already asked our hw guys and they claim control endpoint can’t be ever NAKed (serialization problem).

I thought you said this was a bulk endpoint.

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Tim Roberts[SMTP:xxxxx@probo.com]
Reply To: Windows System Software Devs Interest List
Sent: Monday, February 14, 2005 8:05 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] USB mysteries

>I already asked our hw guys and they claim control endpoint can’t be ever NAKed (serialization problem).

I thought you said this was a bulk endpoint.

I said it is serialization between bulk and control endpoints and Randy said it could be because control endpoint request is NAKed. To make things clear: bulk endpoint read is passed down, pends because device has no data and then control endpoint read is passed down. It waits until bulk read is completed which si what I see as a problem.

I’m just preparing a test app for hw guys with USB analyser and next request to my chief to eventually buy one for me :wink:

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

> From: xxxxx@lists.osr.com

[mailto:xxxxx@lists.osr.com]On Behalf Of Michal Vodicka

The exception occurs if I use 64 bytes (one bulk frame)
buffers for reading. Sometimes the read request is completed
successfully but with zero data. The probability is about 3-
10 : 10^6 and decreases when data rate is lowered (for data
rate ~ 650 kB/s it disappears). I verified it at driver
level; completion function for
_URB_BULK_OR_INTERRUPT_TRANSFER is called with STATUS_SUCCESS
but with zero in
UrbBulkOrInterruptTransfer.TransferBufferLength. Appropriate
frame number is missing in the stream then so it seems data
weren’t read quickly enough or were read but weren’t copied
to buffer. The problem completely disappears if read request
are 128 bytes or more long. It can’t be reproduced even at
highest possible data rate (several MB/s which is more than
USB can transfer).

I’d understand if there is an error but why success? Now I
see I should probably examine _URB_HEADER.Status but if there
is some error, UsbScan ignores it, too.

Michal,

did you test your HW and the driver with a different USB host
controller? By different I mean a HC from a different HW vendor
and/or a different HC standard (UHCI, OHCI, EHCI)?

From my USB experience, USB HCs frequently have weird HW bugs.
Many of these bugs require workarounds in MS HC drivers or in
HC vendor filter drivers.

It seems as the first
bulk URB is sent to hw and blocks subsequent requests. I’m
probably missing something here, can anybody explain it?

Looks like a bug somewhere. A HW USB bus analyzer
is a very useful tool in such cases.

Dmitriy Budko, VMware

Randy,

one mystery is solved and you were right. Control endpoint data requests are NAKed once bulk read is started. Our hw developers are trying to find how is it possible just now. From USB analyser log it is clear and works and you described. After SOF host controller asks control endpoint as the first, it is NAKed and host controller queries builk endpoint which is NAKed until next SOF. And again and again. Bad device; fortunately, only prototype.

The second problem can’t be reproduced in the department where we have USB analyser. It can be reproduced on 3 computers of 12 here and it seems to be related to timing as all 3 have similar transfer rates. I guess you’re also right and device ACKs request with no data. I can imagine race conditions in internal device buffers. Needs more examination.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]


From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Randy Aull[SMTP:xxxxx@windows.microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Sunday, February 13, 2005 2:25 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

Actually, that should read “Reads with zero bytes are NOT necessarily failures”.


From: xxxxx@lists.osr.com on behalf of Randy Aull
Sent: Sat 2/12/2005 3:01 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

Reads with zero bytes are necessarily failures. In fact it can be fairly common. All the device has to do it ACK with no data. That would be a successful read with length = 0. The only times that there would be a failure (barring PnP events, etc) would be if the device issued a STALL, or the device sent more that MaxPacket bytes or some other USB violation.

My guess is that your device is actually ACKing the IN without any data, thus completing your reads with zero bytes.


From: xxxxx@lists.osr.com on behalf of Michal Vodicka
Sent: Fri 2/11/2005 9:41 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

> ----------
> From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Randy Aull[SMTP:xxxxx@windows.microsoft.com]
> Reply To: Windows System Software Devs Interest List
> Sent: Saturday, February 12, 2005 6:16 AM
> To: Windows System Software Devs Interest List
> Subject: RE: [ntdev] USB mysteries
>
> For the first problem, the timing factor makes me believe that your
> device’s buffer is emptying when you request 64 byte requests. At 900
> kB/s, you are pretty much maxing out the bandwidth of USB 1.1. There is
> a high cost of scheduling a transfer on the bus and completing it. This
> sometimes can actually cause the host software to need to pause the
> schedule all together (thus losing some bandwidth). So, I don’t think
> you will be able to keep up with the device by issuing 64 byte requests,
> no matter how many you pend. The bigger the request, the less the
> overhead will affect you, which is why 128 byte reads works better. I
> don’t think this is a USB stack issue, I think your device is out-pacing
> your read algorithm.
>
Yes, but my problem is different. I understand some data are lost when read this way and I tried it just for test purposes. What I don’t understand is why USB stack completes read IRP with STATUS_SUCCESS, URB status 0 and data length 0. I’d understand if there is an error returned. I’d understand if an USB frame if missed which is usual situation. I have about 6 - 7% data loss at this speed using 64 bytes read requests. Frames are just missing, one read frame has number N (generated by device) and next N + 2. It is expected but why there are success zero data reads? As I said, there are about 3 - 10 such reads per million.> I’d like to know what does it mean and if I can expect something like this when using longer requests (I plan to use 64 frames per request i.e. 4096 bytes).

> For the second problem, there is no serialization between control and
> bulk endpoints in the USB stack. Obviously there is some serialization
> in the host controller, as only one transfer can happen at a time, but
> this is packet serialization, not transfer serialization. So, as soon
> as a packet gets NAK’ed or ACK’ed, the host controller gives the next
> endpoint a shot. There are a lot of protocols that have always had
> reads pending on one endpoint while doing other IO on different
> endpoints. So, my guess is that your device is NAKing the control
> transfer.
>
Well, the device is prototype and everything is possible. I can’t check it just now but will try. It is rather hard to believe as it starts to work immediatelly when the first read request is cancelled but maybe device has a problem. I’ll ask hw guys to check.

Anyway, thanks for confirmation it should work as I presumed.

> My recommendation would be to get a USB Bus analyzer and looking at your
> device’s behavior. I think that will illuminate the problem.
>
You’re rigth. I just don’t have USB analyzer here and will try it when possible.

Thanks.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http:]
>
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at http:
>
> You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
>
> —
> Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
> To unsubscribe send a blank email to xxxxx@lists.osr.com
> —
> Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
> To unsubscribe send a blank email to xxxxx@lists.osr.com
></http:></http:>

Control endpoints absolutely can NAK. There is no problem with that.
It happens all the time. It could be that the setup phase can’t be
NAKed, but the data and status phases absolutely can.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Michal Vodicka
Sent: Monday, February 14, 2005 10:44 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

Randy,

thanks for you anwers. You evidently believe problems are caused by the
device which may or may not be right. Anyway, the only way how to prove
or disprove it is using USB analyzer.

I already asked our hw guys and they claim control endpoint can’t be
ever NAKed (serialization problem). It is also hard to believe device
would cause zero data reads as you describe but we will check it.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]


From:
xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com
] on behalf of Randy Aull[SMTP:xxxxx@windows.microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Sunday, February 13, 2005 2:25 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

Actually, that should read “Reads with zero bytes are NOT necessarily
failures”.


From: xxxxx@lists.osr.com on behalf of Randy Aull
Sent: Sat 2/12/2005 3:01 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

Reads with zero bytes are necessarily failures. In fact it can be
fairly common. All the device has to do it ACK with no data. That
would be a successful read with length = 0. The only times that there
would be a failure (barring PnP events, etc) would be if the device
issued a STALL, or the device sent more that MaxPacket bytes or some
other USB violation.

My guess is that your device is actually ACKing the IN without any
data, thus completing your reads with zero bytes.


From: xxxxx@lists.osr.com on behalf of Michal Vodicka
Sent: Fri 2/11/2005 9:41 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

> ----------
> From:
xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com
] on behalf of Randy Aull[SMTP:xxxxx@windows.microsoft.com]
> Reply To: Windows System Software Devs Interest List
> Sent: Saturday, February 12, 2005 6:16 AM
> To: Windows System Software Devs Interest List
> Subject: RE: [ntdev] USB mysteries
>
> For the first problem, the timing factor makes me believe that your
> device’s buffer is emptying when you request 64 byte requests. At
900
> kB/s, you are pretty much maxing out the bandwidth of USB 1.1.
There is
> a high cost of scheduling a transfer on the bus and completing it.
This
> sometimes can actually cause the host software to need to pause the
> schedule all together (thus losing some bandwidth). So, I don’t
think
> you will be able to keep up with the device by issuing 64 byte
requests,
> no matter how many you pend. The bigger the request, the less the
> overhead will affect you, which is why 128 byte reads works better.
I
> don’t think this is a USB stack issue, I think your device is
out-pacing
> your read algorithm.
>
Yes, but my problem is different. I understand some data are lost when
read this way and I tried it just for test purposes. What I don’t
understand is why USB stack completes read IRP with STATUS_SUCCESS, URB
status 0 and data length 0. I’d understand if there is an error
returned. I’d understand if an USB frame if missed which is usual
situation. I have about 6 - 7% data loss at this speed using 64 bytes
read requests. Frames are just missing, one read frame has number N
(generated by device) and next N + 2. It is expected but why there are
success zero data reads? As I said, there are about 3 - 10 such reads
per million. I’d like to know what does it mean and if I can expect
something like this when using longer requests (I plan to use 64 frames
per request i.e. 4096 bytes).

> For the second problem, there is no serialization between control
and
> bulk endpoints in the USB stack. Obviously there is some
serialization>
> in the host controller, as only one transfer can happen at a time,
but
> this is packet serialization, not transfer serialization. So, as
soon
> as a packet gets NAK’ed or ACK’ed, the host controller gives the
next
> endpoint a shot. There are a lot of protocols that have always had
> reads pending on one endpoint while doing other IO on different
> endpoints. So, my guess is that your device is NAKing the control
> transfer.
>
Well, the device is prototype and everything is possible. I can’t
check it just now but will try. It is rather hard to believe as it
starts to work immediatelly when the first read request is cancelled but
maybe device has a problem. I’ll ask hw guys to check.

Anyway, thanks for confirmation it should work as I presumed.

> My recommendation would be to get a USB Bus analyzer and looking at
your
> device’s behavior. I think that will illuminate the problem.
>
You’re rigth. I just don’t have USB analyzer here and will try it when
possible.

Thanks.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http:]
>
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at
http:
>
> You are currently subscribed to ntdev as: unknown lmsubst tag
argument: ‘’
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
>
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: unknown lmsubst tag
argument: ‘’
> To unsubscribe send a blank email to xxxxx@lists.osr.com
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: unknown lmsubst tag
argument: ‘’
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com</http:></http:>

Ah, I meant they believed it can’t be NAKed in our case as the device should be able to answer all control endpoint requests immediatelly. Apparently, something is wrong there.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]


From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Randy Aull[SMTP:xxxxx@windows.microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Tuesday, February 15, 2005 9:57 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

Control endpoints absolutely can NAK. There is no problem with that.
It happens all the time. It could be that the setup phase can’t be
NAKed, but the data and status phases absolutely can.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Michal Vodicka
Sent: Monday, February 14, 2005 10:44 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

Randy,

thanks for you anwers. You evidently believe problems are caused by the
device which may or may not be right. Anyway, the only way how to prove
or disprove it is using USB analyzer.

I already asked our hw guys and they claim control endpoint can’t be
ever NAKed (serialization problem). It is also hard to believe device
would cause zero data reads as you describe but we will check it.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

> ----------
> From:
xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com
] on behalf of Randy Aull[SMTP:xxxxx@windows.microsoft.com]
> Reply To: Windows System Software Devs Interest List
> Sent: Sunday, February 13, 2005 2:25 AM
> To: Windows System Software Devs Interest List
> Subject: RE: [ntdev] USB mysteries
>
> Actually, that should read “Reads with zero bytes are NOT necessarily
failures”.
>
> _____
>
> From: xxxxx@lists.osr.com on behalf of Randy Aull
> Sent: Sat 2/12/2005 3:01 PM
> To: Windows System Software Devs Interest List
> Subject: RE: [ntdev] USB mysteries
>
>
> Reads with zero bytes are necessarily failures. In fact it can be
fairly common. All the device has to do it ACK with no data. That
would be a successful read with length = 0. The only times that there
would be a failure (barring PnP events, etc) would be if the device
issued a STALL, or the device sent more that MaxPacket bytes or some
other USB violation.
>
> My guess is that your device is actually ACKing the IN without any
data, thus completing your reads with zero bytes.
>
>
> _____
>
> From: xxxxx@lists.osr.com on behalf of Michal Vodicka
> Sent: Fri 2/11/2005 9:41 PM
> To: Windows System Software Devs Interest List
> Subject: RE: [ntdev] USB mysteries
>
>
>
> > ----------
> > From:
xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com
] on behalf of Randy Aull[SMTP:xxxxx@windows.microsoft.com]
> > Reply To: Windows System Software Devs Interest List
> > Sent: Saturday, February 12, 2005 6:16 AM
> > To: Windows System Software Devs Interest List
> > Subject: RE: [ntdev] USB mysteries
> >
> > For the first problem, the timing factor makes me believe that your
> > device’s buffer is emptying when you request 64 byte requests. At
900
> > kB/s, you are pretty much maxing out the bandwidth of USB 1.1.
There is
> > a high cost of scheduling a transfer on the bus and completing it.
This
> > sometimes can actually cause the host software to need to pause the
> > schedule all together (thus losing some bandwidth). So, I don’t
think
> > you will be able to keep up with the device by issuing 64 byte>
requests,
> > no matter how many you pend. The bigger the request, the less the
> > overhead will affect you, which is why 128 byte reads works better.
I
> > don’t think this is a USB stack issue, I think your device is
out-pacing
> > your read algorithm.
> >
> Yes, but my problem is different. I understand some data are lost when
read this way and I tried it just for test purposes. What I don’t
understand is why USB stack completes read IRP with STATUS_SUCCESS, URB
status 0 and data length 0. I’d understand if there is an error
returned. I’d understand if an USB frame if missed which is usual
situation. I have about 6 - 7% data loss at this speed using 64 bytes
read requests. Frames are just missing, one read frame has number N
(generated by device) and next N + 2. It is expected but why there are
success zero data reads? As I said, there are about 3 - 10 such reads
per million. I’d like to know what does it mean and if I can expect
something like this when using longer requests (I plan to use 64 frames
per request i.e. 4096 bytes).
>
> > For the second problem, there is no serialization between control
and
> > bulk endpoints in the USB stack. Obviously there is some
serialization>
> > in the host controller, as only one transfer can happen at a time,
but
> > this is packet serialization, not transfer serialization. So, as
soon
> > as a packet gets NAK’ed or ACK’ed, the host controller gives the
next
> > endpoint a shot. There are a lot of protocols that have always had
> > reads pending on one endpoint while doing other IO on different
> > endpoints. So, my guess is that your device is NAKing the control
> > transfer.
> >
> Well, the device is prototype and everything is possible. I can’t
check it just now but will try. It is rather hard to believe as it
starts to work immediatelly when the first read request is cancelled but
maybe device has a problem. I’ll ask hw guys to check.
>
> Anyway, thanks for confirmation it should work as I presumed.
>
> > My recommendation would be to get a USB Bus analyzer and looking at
your
> > device’s behavior. I think that will illuminate the problem.
> >
> You’re rigth. I just don’t have USB analyzer here and will try it when
possible.
>
> Thanks.
>
> Best regards,
>
> Michal Vodicka
> UPEK, Inc.
> [xxxxx@upek.com, http:]
> >
> >
> >
> >
> > —
> > Questions? First check the Kernel Driver FAQ at
> http:
> >
> > You are currently subscribed to ntdev as: unknown lmsubst tag
> argument: ‘’
> > To unsubscribe send a blank email to xxxxx@lists.osr.com
> >
> >
> > —
> > Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
> >
> > You are currently subscribed to ntdev as: unknown lmsubst tag
> argument: ‘’
> > To unsubscribe send a blank email to xxxxx@lists.osr.com
> > —
> > Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
> >
> > You are currently subscribed to ntdev as: unknown lmsubst tag
> argument: ‘’
> > To unsubscribe send a blank email to xxxxx@lists.osr.com
> >
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: unknown lmsubst tag argument:
> ‘’
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
> —
> Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
> To unsubscribe send a blank email to xxxxx@lists.osr.com
></http:></http:>

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Dmitriy Budko[SMTP:xxxxx@vmware.com]
Reply To: Windows System Software Devs Interest List
Sent: Tuesday, February 15, 2005 12:09 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

Dmitriy,

did you test your HW and the driver with a different USB host
controller? By different I mean a HC from a different HW vendor
and/or a different HC standard (UHCI, OHCI, EHCI)?

From my USB experience, USB HCs frequently have weird HW bugs.
Many of these bugs require workarounds in MS HC drivers or in
HC vendor filter drivers.

We tested it on 14 computers yesterday and problem can be reproduced on 3 of them. In all cases on VIA USB controller driven by usbuhci.sys. One computer had more USB controllers and problem could be reproduced on VIA controller only. It wouldn’t surprise me if there is some VIA hw problem, it wouldn’t be the first one. However, it can be pure coincidence and problem can be releated to timing which differs with controller and drivers. It doesn’t seem OS dependent, reproduced on XP SP0, SP2 and w2k3 SP1 as well.

For definitive answer it is necessary to capture USB traffic with analyser. Unfortunately, it is in our department in different country where problem can’t be reproduced.

> It seems as the first
> bulk URB is sent to hw and blocks subsequent requests. I’m
> probably missing something here, can anybody explain it?

Looks like a bug somewhere. A HW USB bus analyzer
is a very useful tool in such cases.

Yep. It is device problem which NAKs control endpoint requests once bulk read is started. I saw it in the log within 5 minutes. Great tool and it seems I need one, too :slight_smile:

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

Finally I have USB analyser and could investigate the second mystery (zero data reads). I examined several logs and results are convincing: there is no USB level problem. All lost frames were successfully read and ACKed. No ACK with no data, always 64 bytes of data transferred and ACKed. It means the problem is at host side; HC or OS drivers.

We’re able to reproduce problem at 7 computers, all with USB UHCI controllers. Six with VIA and one with NEC. All computers are relatively slow, CPU speeds up to 1.4 GHz, both Intel and AMD.

I guess there is some kind of race conditions somewhere. The only thing all lost reads have common is timing. If frame N is lost, the time between frame N and frame N + 1 is in interval 56.500 usec and 56.833 usec and the time between N’s ACK and N + 1 IN packet is about 7 usec. As I examined logs, it is uncommon. Usual times are longer than 57.1 usec and the time between ACK and IN is more than 8 usec. Hard to prove because analyser software doesn’t allow to display difference times.

I wonder if installation of checked usbuhci.sys version can help to reveal problem. If so, are there some debug flags to enable? Or ETW?

Ideas welcome. It isn’t critical problem as a workaround is easy and amount of lost data is small but I’d prefer to know what can cause it.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]


From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Michal Vodicka[SMTP:xxxxx@upek.com]
Reply To: Windows System Software Devs Interest List
Sent: Tuesday, February 15, 2005 9:31 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] USB mysteries

Randy,

one mystery is solved and you were right. Control endpoint data requests are NAKed once bulk read is started. Our hw developers are trying to find how is it possible just now. From USB analyser log it is clear and works and you described. After SOF host controller asks control endpoint as the first, it is NAKed and host controller queries builk endpoint which is NAKed until next SOF. And again and again. Bad device; fortunately, only prototype.

The second problem can’t be reproduced in the department where we have USB analyser. It can be reproduced on 3 computers of 12 here and it seems to be related to timing as all 3 have similar transfer rates. I guess you’re also right and device ACKs request with no data. I can imagine race conditions in internal device buffers. Needs more examination.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

> ----------
> From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Randy Aull[SMTP:xxxxx@windows.microsoft.com]
> Reply To: Windows System Software Devs Interest List
> Sent: Sunday, February 13, 2005 2:25 AM
> To: Windows System Software Devs Interest List
> Subject: RE: [ntdev] USB mysteries
>
> Actually, that should read “Reads with zero bytes are NOT necessarily failures”.
>
> _____
>
> From: xxxxx@lists.osr.com on behalf of Randy Aull
> Sent: Sat 2/12/2005 3:01 PM
> To: Windows System Software Devs Interest List
> Subject: RE: [ntdev] USB mysteries
>
>
> Reads with zero bytes are necessarily failures. In fact it can be fairly common. All the device has to do it ACK with no data. That would be a successful read with length = 0. The only times that there would be a failure (barring PnP events, etc) would be if the device issued a STALL, or the device sent more that MaxPacket bytes or some other USB violation.
>
> My guess is that your device is actually ACKing the IN without any data, thus completing your reads with zero bytes.
>
>
> _____
>
> From: xxxxx@lists.osr.com on behalf of Michal Vodicka
> Sent: Fri 2/11/2005 9:41 PM
> To: Windows System Software Devs Interest List
> Subject: RE: [ntdev] USB mysteries
>
>
>
> > ----------
> > From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Randy Aull[SMTP:xxxxx@windows.microsoft.com]
> > Reply To: Windows System Software Devs Interest List
> > Sent: Saturday, February 12, 2005 6:16 AM
> > To: Windows System Software Devs Interest List
> > Subject: RE: [ntdev] USB mysteries
> >
> > For the first problem, the timing factor makes me believe that your
> > device’s buffer is emptying when you request 64 byte requests. At 900
> > kB/s, you are pretty much maxing out the bandwidth of USB 1.1. There is
> > a high cost of scheduling a transfer on the bus and completing it. This
> > sometimes can actually cause the host software to need to pause the
> > schedule all together (thus losing some bandwidth). So, I don’t think
> > you will be able to keep up with the device by issuing 64 byte requests,
> > no matter how many you pend. The bigger the request, the less the
> > overhead will affect you, which is why 128 byte reads works better. I
> > don’t think this is a USB stack issue, I think your device is out-pacing
> > your read algorithm.
> >
> Yes, but my problem is different. I understand some data are lost when read this way and I tried it just for test purposes. What I don’t understand is why USB stack completes read IRP with STATUS_SUCCESS, URB status 0 and data length 0. I’d understand if there is an error returned. I’d understand if an USB frame if missed which is usual situation. I have about 6 - 7% data loss at this speed using 64 bytes read requests. Frames are just missing, one read frame has number N (generated by device) and next N + 2. It is expected but why there are success zero data reads? As I said, there are about 3 - 10 such reads per million.> I’d like to know what does it mean and if I can expect something like this when using longer requests (I plan to use 64 frames per request i.e. 4096 bytes).
>
> > For the second problem, there is no serialization between control and
> > bulk endpoints in the USB stack. Obviously there is some serialization
> > in the host controller, as only one transfer can happen at a time, but
> > this is packet serialization, not transfer serialization. So, as soon
> > as a packet gets NAK’ed or ACK’ed, the host controller gives the next
> > endpoint a shot. There are a lot of protocols that have always had
> > reads pending on one endpoint while doing other IO on different
> > endpoints. So, my guess is that your device is NAKing the control
> > transfer.
> >
> Well, the device is prototype and everything is possible. I can’t check it just now but will try. It is rather hard to believe as it starts to work immediatelly when the first read request is cancelled but maybe device has a problem. I’ll ask hw guys to check.
>
> Anyway, thanks for confirmation it should work as I presumed.
>
> > My recommendation would be to get a USB Bus analyzer and looking at your
> > device’s behavior. I think that will illuminate the problem.
> >
> You’re rigth. I just don’t have USB analyzer here and will try it when possible.
>
> Thanks.
>
> Best regards,
>
> Michal Vodicka
> UPEK, Inc.
> [xxxxx@upek.com, http:]
></http:>

Michal,

I have seen two different situations that will cause a USB host
controller to ACK a received data packet and discard the contents. I’m
not sure what hardware you are using to implement your USB peripheral
device. So, forgive me if this information is not applicable.

  1. If the USB peripheral device errors the data toggle (DATA0/DATA1) it
    may send two DATA0 or two DATA1 packets in a row. Per the spec’, the
    host controller will ACK the second packet and discard the contents. It
    assumes that the packet is a duplicate.

  2. The host controller may react this way if your USB peripheral fails
    to implement bit stuffing properly. I witnessed a case where a full
    speed device failed to transmit a stuffed 0-bit when it was the last bit
    of a packet. Literally, if the last six bits of the data CRC are ones
    then the device must transmit a stuffed 0-bit before the EOP. If I
    remember correctly, VIA host controllers were the most sensitive to this
    problem. Most other manufacturer’s controllers would ignore it. But,
    the VIA chips would ACK the packet and discard the data.

Regarding your periodic measurements. A problem such as number two is
dependant on the data. Thus, periodicity in the data could potential
yield such measurements.

Good Luck,
Jim Allred
EVI Technology, LLC

Michal Vodicka wrote:

Finally I have USB analyser and could investigate the second mystery (zero data reads). I examined several logs and results are convincing: there is no USB level problem. All lost frames were successfully read and ACKed. No ACK with no data, always 64 bytes of data transferred and ACKed. It means the problem is at host side; HC or OS drivers.

We’re able to reproduce problem at 7 computers, all with USB UHCI controllers. Six with VIA and one with NEC. All computers are relatively slow, CPU speeds up to 1.4 GHz, both Intel and AMD.

I guess there is some kind of race conditions somewhere. The only thing all lost reads have common is timing. If frame N is lost, the time between frame N and frame N + 1 is in interval 56.500 usec and 56.833 usec and the time between N’s ACK and N + 1 IN packet is about 7 usec. As I examined logs, it is uncommon. Usual times are longer than 57.1 usec and the time between ACK and IN is more than 8 usec. Hard to prove because analyser software doesn’t allow to display difference times.

I wonder if installation of checked usbuhci.sys version can help to reveal problem. If so, are there some debug flags to enable? Or ETW?

Ideas welcome. It isn’t critical problem as a workaround is easy and amount of lost data is small but I’d prefer to know what can cause it.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

>----------
>From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Michal Vodicka[SMTP:xxxxx@upek.com]
>Reply To: Windows System Software Devs Interest List
>Sent: Tuesday, February 15, 2005 9:31 PM
>To: Windows System Software Devs Interest List
>Subject: RE: [ntdev] USB mysteries
>
>Randy,
>
>one mystery is solved and you were right. Control endpoint data requests are NAKed once bulk read is started. Our hw developers are trying to find how is it possible just now. From USB analyser log it is clear and works and you described. After SOF host controller asks control endpoint as the first, it is NAKed and host controller queries builk endpoint which is NAKed until next SOF. And again and again. Bad device; fortunately, only prototype.
>
>The second problem can’t be reproduced in the department where we have USB analyser. It can be reproduced on 3 computers of 12 here and it seems to be related to timing as all 3 have similar transfer rates. I guess you’re also right and device ACKs request with no data. I can imagine race conditions in internal device buffers. Needs more examination.
>
>Best regards,
>
>Michal Vodicka
>UPEK, Inc.
>[xxxxx@upek.com, http://www.upek.com]
>
>
>
>>----------
>>From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Randy Aull[SMTP:xxxxx@windows.microsoft.com]
>>Reply To: Windows System Software Devs Interest List
>>Sent: Sunday, February 13, 2005 2:25 AM
>>To: Windows System Software Devs Interest List
>>Subject: RE: [ntdev] USB mysteries
>>
>>Actually, that should read “Reads with zero bytes are NOT necessarily failures”.
>>
>> _____
>>
>>From: xxxxx@lists.osr.com on behalf of Randy Aull
>>Sent: Sat 2/12/2005 3:01 PM
>>To: Windows System Software Devs Interest List
>>Subject: RE: [ntdev] USB mysteries
>>
>>
>>Reads with zero bytes are necessarily failures. In fact it can be fairly common. All the device has to do it ACK with no data. That would be a successful read with length = 0. The only times that there would be a failure (barring PnP events, etc) would be if the device issued a STALL, or the device sent more that MaxPacket bytes or some other USB violation.
>>
>>My guess is that your device is actually ACKing the IN without any data, thus completing your reads with zero bytes.
>>
>>
>> _____
>>
>>From: xxxxx@lists.osr.com on behalf of Michal Vodicka
>>Sent: Fri 2/11/2005 9:41 PM
>>To: Windows System Software Devs Interest List
>>Subject: RE: [ntdev] USB mysteries
>>
>>
>>
>>
>>>----------
>>>From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Randy Aull[SMTP:xxxxx@windows.microsoft.com]
>>>Reply To: Windows System Software Devs Interest List
>>>Sent: Saturday, February 12, 2005 6:16 AM
>>>To: Windows System Software Devs Interest List
>>>Subject: RE: [ntdev] USB mysteries
>>>
>>>For the first problem, the timing factor makes me believe that your
>>>device’s buffer is emptying when you request 64 byte requests. At 900
>>>kB/s, you are pretty much maxing out the bandwidth of USB 1.1. There is
>>>a high cost of scheduling a transfer on the bus and completing it. This
>>>sometimes can actually cause the host software to need to pause the
>>>schedule all together (thus losing some bandwidth). So, I don’t think
>>>you will be able to keep up with the device by issuing 64 byte requests,
>>>no matter how many you pend. The bigger the request, the less the
>>>overhead will affect you, which is why 128 byte reads works better. I
>>>don’t think this is a USB stack issue, I think your device is out-pacing
>>>your read algorithm.
>>>
>>
>>Yes, but my problem is different. I understand some data are lost when read this way and I tried it just for test purposes. What I don’t understand is why USB stack completes read IRP with STATUS_SUCCESS, URB status 0 and data length 0. I’d understand if there is an error returned. I’d understand if an USB frame if missed which is usual situation. I have about 6 - 7% data loss at this speed using 64 bytes read requests. Frames are just missing, one read frame has number N (generated by device) and next N + 2. It is expected but why there are success zero data reads? As I said, there are about 3 - 10 such reads per million.> I’d like to know what does it mean and if I can expect something like this when using longer requests (I plan to use 64 frames per request i.e. 4096 bytes).
>>
>>
>>>For the second problem, there is no serialization between control and
>>>bulk endpoints in the USB stack. Obviously there is some serialization
>>>in the host controller, as only one transfer can happen at a time, but
>>>this is packet serialization, not transfer serialization. So, as soon
>>>as a packet gets NAK’ed or ACK’ed, the host controller gives the next
>>>endpoint a shot. There are a lot of protocols that have always had
>>>reads pending on one endpoint while doing other IO on different
>>>endpoints. So, my guess is that your device is NAKing the control
>>>transfer.
>>>
>>
>>Well, the device is prototype and everything is possible. I can’t check it just now but will try. It is rather hard to believe as it starts to work immediatelly when the first read request is cancelled but maybe device has a problem. I’ll ask hw guys to check.
>>
>>Anyway, thanks for confirmation it should work as I presumed.
>>
>>
>>>My recommendation would be to get a USB Bus analyzer and looking at your
>>>device’s behavior. I think that will illuminate the problem.
>>>
>>
>>You’re rigth. I just don’t have USB analyzer here and will try it when possible.
>>
>>Thanks.
>>
>>Best regards,
>>
>>Michal Vodicka
>>UPEK, Inc.
>>[xxxxx@upek.com, http:]
>>
>
> —
> Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
> To unsubscribe send a blank email to xxxxx@lists.osr.com
></http:>

Jim,

thanks for the ideas. Comments inline:

  1. If the USB peripheral device errors the data toggle (DATA0/DATA1) it
    may send two DATA0 or two DATA1 packets in a row. Per the spec’, the
    host controller will ACK the second packet and discard the contents. It
    assumes that the packet is a duplicate.

I checked it and it never occured, at least for lost frames. DATA0/1 always toggle.

  1. The host controller may react this way if your USB peripheral fails
    to implement bit stuffing properly. I witnessed a case where a full
    speed device failed to transmit a stuffed 0-bit when it was the last bit
    of a packet. Literally, if the last six bits of the data CRC are ones
    then the device must transmit a stuffed 0-bit before the EOP. If I
    remember correctly, VIA host controllers were the most sensitive to this
    problem. Most other manufacturer’s controllers would ignore it. But,
    the VIA chips would ACK the packet and discard the data.

I hope analyser would catch it but checked it to be sure. In most cases, lost frames CRC didn’t contain six consecutive ones. Frame data except first two bytes also didn’t contain it.

Regarding your periodic measurements. A problem such as number two is
dependant on the data. Thus, periodicity in the data could potential
yield such measurements.

I’m not sure if I exactly understand what you mean. Let me describe how the device under test works. It generates stream of data according to specified interval. Host job is to read data as fast as possible because device circular buffer is very limited (8 frames). Test frames are numbered: first two bytes contain the number from 0 to 0x3fff so I can detect data loss. The remaining 62 bytes is some constant value as 0xf. The problem can be reproduced if device is faster than host i.e. it always has data to send when host asks. There is no NAK. With decreased device speed problem probability lowers and finally disappears.

As for times, it is the only pattern I see. I wasn’t quite clear before: the 56.xxx usec time is from lost frame transfer start to next frame transfer start. I tried to set device to generate data faster and used overlapped reads at PC side and although data stream was very regular (17 transfers per time frame), I never noticed the time difference shorter than 57 usec. It seems as a bit shorter times are unique for lost frames.

Well, I believe in this case our device doesn’t cause the problem. It can be HC problem but we reproduced it on HCs from two different manufacturers. Also, it seems to be related to CPU speed as we reproduced it on slower CPUs only. It leads to conclusion there is some OS drivers bug, probably very tight race conditions. Reproduced at XP SP0, SP2 and w2k3, always at USB UHCI controller. There were computers with more HCs of different types and problem was reproducible at UHCI only.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]