Trying to simulate serial (COM) port with WinUsb

I have a question about how WinUsb handles short, irregular reads and
timeouts.

I’ve been looking through the documentation and examples but I don’t
quite find the options that will do what I want. I am relatively new to
windows and usb development so I apologize if what I ask is basic and I
just don’t know how to look for the answer. I have found very little
information or examples about WinUSB compared to other USB development.

I have several devices from different manufacturers, each of them with
similar, but not quite the same, communication protocols. Each
apparently was originally designed for serial interface, but now every
one of them offers USB as a connection option.

I have code to talk to these devices when connected via serial cable
that basically does the following:

open serial port for non overlapped IO using a tiemout
loop (a long time) {

check if device has any data to read using ReadFile
read until read buffer is empty

/* This times out if there is nothing, or returns whatever
is in the read buffer at the time of the read call */

When user is ready, send a command to the device using WriteFile

checks for ACK/NAK

}

The read data is variable in length, including 0 bytes if there is no
data to read, and can vary from 1 byte (ACK/NAK) to hundreds of bytes. I
do not know the length ahead of time.

I want to mimic this behavior using WinUSB, but I can’t quite get it to
do what I want. I believe I need to set the ReadPipe options to RAW_IO,
but when I combine this with timeouts the code breaks in mysterious ways.

I am using non-overlapped reads/writes.

I have code similar to:

timeout=400;
retval = WinUsb_SetPipePolicy(handle, epin,
PIPE_TRANSFER_TIMEOUT, sizeof(timeout),
&timeout);

/* check for errors */
t = FALSE:
retval = WinUsb_SetPipePolicy(handle, epin, ALLOW_PARTIAL_READS,
sizeof(t), &t); /* check for errors */
raw=0;
retval = WinUsb_SetPipePolicy(handle, epin, RAW_IO, sizeof(raw), &raw);

/* check for errors */

later on, I have a loop where I send commands to the device and expect a
response from it. Similar to:

loop () {
retval = WinUsb_ReadPipe (handle, epin, buf, 64, &readbytes, NULL);
/* check for errors or timeout */

/* handle read data */

if (I need to send a command) {
retval = WinUsb_WritePipe(handle, epout, some_command,
command_len, &sentbytes, NULL);
/* check for errors */
}
}

I have no problems with my writes, but the reads are timing out without
returning any data when I know the device is ready to send small amounts
of data (for example, with ACKs). Also, for long reads I am missing part
of the read buffer - my guess is that at timeout the read request is
canceled and with it whatever data the device was getting ready to send
to me.

I created a sample script that is successful in communicating with the
device if I use raw IO without timeouts. Unfortunately I can’t use this
since I do not know exactly when I will have data nor the size of the
data received.

I need behavior similar to ReadFile where it will return right away with
whatever is in the buffer at the time I call it, and if it times out
with an empty buffer, it returns 0 bytes, but it doesn’t drop any data
if I cancel the timeout.

Any help or pointers to examples I could read would be very much
appreciated.

I also tried without RAW_IO, but I noticed then that ACKs would not be
received until the device had additional (64 bytes total) data to send.


Miguel Sosa

Sorry, my example there had some errors on the values part of set pipe
policy:

timeout=400;
retval = WinUsb_SetPipePolicy(usb_handle, usb_epin,
PIPE_TRANSFER_TIMEOUT, sizeof(timeout),
&timeout);

opt=1;
retval = WinUsb_SetPipePolicy(usb_handle, usb_epin,
ALLOW_PARTIAL_READS, sizeof(opt), opt);

retval = WinUsb_SetPipePolicy(usb_handle, usb_epin,
RAW_IO, sizeof(opt), &opt);

In any case, is winusb the right path to follow for this scenario or
would a UMDF driver be more appropriate?

Could the PIPE_TRANSFER_TIMEOUT be causing the issue with the dropped
ACKs even with RAW_IO? Should I move to use overlap in my reads?

Miguel Sosa wrote:

I have a question about how WinUsb handles short, irregular reads and
timeouts.

I’ve been looking through the documentation and examples but I don’t
quite find the options that will do what I want. I am relatively new
to windows and usb development so I apologize if what I ask is basic
and I just don’t know how to look for the answer. I have found very
little information or examples about WinUSB compared to other USB
development.

I have several devices from different manufacturers, each of them with
similar, but not quite the same, communication protocols. Each
apparently was originally designed for serial interface, but now every
one of them offers USB as a connection option.

I have code to talk to these devices when connected via serial cable
that basically does the following:

open serial port for non overlapped IO using a tiemout
loop (a long time) {

check if device has any data to read using ReadFile
read until read buffer is empty

/* This times out if there is nothing, or returns whatever
is in the read buffer at the time of the read call */

When user is ready, send a command to the device using WriteFile

checks for ACK/NAK

}

The read data is variable in length, including 0 bytes if there is no
data to read, and can vary from 1 byte (ACK/NAK) to hundreds of bytes.
I do not know the length ahead of time.

I want to mimic this behavior using WinUSB, but I can’t quite get it
to do what I want. I believe I need to set the ReadPipe options to
RAW_IO, but when I combine this with timeouts the code breaks in
mysterious ways.

I am using non-overlapped reads/writes.

I have code similar to:

timeout=400;
retval = WinUsb_SetPipePolicy(handle, epin,
PIPE_TRANSFER_TIMEOUT, sizeof(timeout),
&timeout);

/* check for errors */
t = FALSE:
retval = WinUsb_SetPipePolicy(handle, epin, ALLOW_PARTIAL_READS,
sizeof(t), &t); /* check for errors */
raw=0;
retval = WinUsb_SetPipePolicy(handle, epin, RAW_IO, sizeof(raw), &raw);

/* check for errors */

later on, I have a loop where I send commands to the device and expect
a response from it. Similar to:

loop () {
retval = WinUsb_ReadPipe (handle, epin, buf, 64, &readbytes, NULL);
/* check for errors or timeout */

/* handle read data */

if (I need to send a command) {
retval = WinUsb_WritePipe(handle, epout, some_command,
command_len, &sentbytes, NULL);
/* check for errors */
}
}

I have no problems with my writes, but the reads are timing out
without returning any data when I know the device is ready to send
small amounts of data (for example, with ACKs). Also, for long reads I
am missing part of the read buffer - my guess is that at timeout the
read request is canceled and with it whatever data the device was
getting ready to send to me.

I created a sample script that is successful in communicating with the
device if I use raw IO without timeouts. Unfortunately I can’t use
this since I do not know exactly when I will have data nor the size of
the data received.

I need behavior similar to ReadFile where it will return right away
with whatever is in the buffer at the time I call it, and if it times
out with an empty buffer, it returns 0 bytes, but it doesn’t drop any
data if I cancel the timeout.

Any help or pointers to examples I could read would be very much
appreciated.

I also tried without RAW_IO, but I noticed then that ACKs would not be
received until the device had additional (64 bytes total) data to send.


Miguel Sosa


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Miguel Sosa wrote:

I have a question about how WinUsb handles short, irregular reads and
timeouts.

I’ve been looking through the documentation and examples but I don’t
quite find the options that will do what I want. I am relatively new
to windows and usb development so I apologize if what I ask is basic
and I just don’t know how to look for the answer. I have found very
little information or examples about WinUSB compared to other USB
development.

WinUSB is relatively new. I’m starting to do more and more with it, but
I’m not sure it’s had the same level of user pounding that the kernel
USB APIs have had.

I have several devices from different manufacturers, each of them with
similar, but not quite the same, communication protocols. Each
apparently was originally designed for serial interface, but now every
one of them offers USB as a connection option.

What kind of pipes do these devices use? Bulk or interrupt? Don’t
these devices claim to be USB Communication Device Class devices, so
that they work with a standard driver?

I want to mimic this behavior using WinUSB, but I can’t quite get it
to do what I want. I believe I need to set the ReadPipe options to
RAW_IO, but when I combine this with timeouts the code breaks in
mysterious ways.

later on, I have a loop where I send commands to the device and expect
a response from it. Similar to:

loop () {
retval = WinUsb_ReadPipe (handle, epin, buf, 64, &readbytes, NULL);
/* check for errors or timeout */

/* handle read data */

if (I need to send a command) {
retval = WinUsb_WritePipe(handle, epout, some_command,
command_len, &sentbytes, NULL);
/* check for errors */
}
}

I have no problems with my writes, but the reads are timing out
without returning any data when I know the device is ready to send
small amounts of data (for example, with ACKs). Also, for long reads I
am missing part of the read buffer - my guess is that at timeout the
read request is canceled and with it whatever data the device was
getting ready to send to me.

So, “readbytes” is 0 after the ReadPipe call?

You might consider getting a USB sniffer tool like SnoopyPro
(http://sourceforge.net/projects/usbsnoop/) to monitor the low-level
transactions. I’m not sure that leads to a solution, but it will tell
you how the WinUSB calls are mapping to URBs and transfers.

I need behavior similar to ReadFile where it will return right away
with whatever is in the buffer at the time I call it, and if it times
out with an empty buffer, it returns 0 bytes, but it doesn’t drop any
data if I cancel the timeout.

I believe that’s how WinUSB_ReadPipe is supposed to work, but perhaps we
can get a note from Doron on that.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Miguel Sosa wrote:

Sorry, my example there had some errors on the values part of set pipe
policy:

timeout=400;
retval = WinUsb_SetPipePolicy(usb_handle, usb_epin,
PIPE_TRANSFER_TIMEOUT, sizeof(timeout),
&timeout);

opt=1;
retval = WinUsb_SetPipePolicy(usb_handle, usb_epin,
ALLOW_PARTIAL_READS, sizeof(opt), opt);

I assume you mean “&opt”, but it shouldn’t matter. ALLOW_PARTIAL_READS
should not be relevant for you, and I don’t think it has any effect when
you specify RAW_IO.

In any case, is winusb the right path to follow for this scenario or
would a UMDF driver be more appropriate?

UMDF drivers use WinUSB as their transport. It’s a different API, but
the same provider.

Could the PIPE_TRANSFER_TIMEOUT be causing the issue with the dropped
ACKs even with RAW_IO? Should I move to use overlap in my reads?

When you say “dropped ACKs”, do you mean a byte of data being returned
on the pipe? The reason I ask is that “ACK” has a very specific meaning
in the USB world, and I’m guessing that’s not what you mean.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Hi Tim,

Thanks for your email

Tim Roberts wrote:

Miguel Sosa wrote:

> I have a question about how WinUsb handles short, irregular reads and
> timeouts.
>
> I’ve been looking through the documentation and examples but I don’t
> quite find the options that will do what I want. I am relatively new
> to windows and usb development so I apologize if what I ask is basic
> and I just don’t know how to look for the answer. I have found very
> little information or examples about WinUSB compared to other USB
> development.
>

WinUSB is relatively new. I’m starting to do more and more with it, but
I’m not sure it’s had the same level of user pounding that the kernel
USB APIs have had.

I definitely do get that. I have found some good examples but not much
on the special cases.

> I have several devices from different manufacturers, each of them with
> similar, but not quite the same, communication protocols. Each
> apparently was originally designed for serial interface, but now every
> one of them offers USB as a connection option.
>

What kind of pipes do these devices use? Bulk or interrupt? Don’t
these devices claim to be USB Communication Device Class devices, so
that they work with a standard driver?

3 devices use Bulk pipes; One uses interrupt (although it has one
interrupt for reads and another for writes).

The manufacturers do provide their own drivers. The ones that have bulk
pipes have virtual COM drivers (so I can use them with the serial port
code i already have). The one that uses interrupts works like a HID
device. I wrote a basic HID client for that one which seems to work well.

Having said that, for simplicity to our customers I would prefer to have
just one way for all devices to communicate. I already use libusb in
linux to talk to all 4 devices and wanted to do something similar in
windows. I am following the progress in the libusb list of the winusb
port but I need something that can use right now.

> I want to mimic this behavior using WinUSB, but I can’t quite get it
> to do what I want. I believe I need to set the ReadPipe options to
> RAW_IO, but when I combine this with timeouts the code breaks in
> mysterious ways.
> …
> later on, I have a loop where I send commands to the device and expect
> a response from it. Similar to:
>
> loop () {
> retval = WinUsb_ReadPipe (handle, epin, buf, 64, &readbytes, NULL);
> /* check for errors or timeout */
>
> /* handle read data */
>
> if (I need to send a command) {
> retval = WinUsb_WritePipe(handle, epout, some_command,
> command_len, &sentbytes, NULL);
> /* check for errors */
> }
> }
>
> I have no problems with my writes, but the reads are timing out
> without returning any data when I know the device is ready to send
> small amounts of data (for example, with ACKs). Also, for long reads I
> am missing part of the read buffer - my guess is that at timeout the
> read request is canceled and with it whatever data the device was
> getting ready to send to me.
>

So, “readbytes” is 0 after the ReadPipe call?

yes. I get an ERROR_SEM_TIMEOUT and readbytes is 0.

You might consider getting a USB sniffer tool like SnoopyPro
(http://sourceforge.net/projects/usbsnoop/) to monitor the low-level
transactions. I’m not sure that leads to a solution, but it will tell
you how the WinUSB calls are mapping to URBs and transfers.

I am using sniffUSB, and have used Snoopy. It has been very helpful,
except that I am still learning how to parse it. I can recognize the
packets with read/write data, and some of the control requests, but most
of the ones without data are still a mystery to me.

Any links that would help with learning how to parse the USB traffic?

> I need behavior similar to ReadFile where it will return right away
> with whatever is in the buffer at the time I call it, and if it times
> out with an empty buffer, it returns 0 bytes, but it doesn’t drop any
> data if I cancel the timeout.
>

I believe that’s how WinUSB_ReadPipe is supposed to work, but perhaps we
can get a note from Doron on that.

Thanks for your help!

Tim Roberts wrote:

Miguel Sosa wrote:

> Sorry, my example there had some errors on the values part of set pipe
> policy:
>
> timeout=400;
> retval = WinUsb_SetPipePolicy(usb_handle, usb_epin,
> PIPE_TRANSFER_TIMEOUT, sizeof(timeout),
> &timeout);
>
> opt=1;
> retval = WinUsb_SetPipePolicy(usb_handle, usb_epin,
> ALLOW_PARTIAL_READS, sizeof(opt), opt);
>

I assume you mean “&opt”, but it shouldn’t matter. ALLOW_PARTIAL_READS
should not be relevant for you, and I don’t think it has any effect when
you specify RAW_IO.

Yes. &opt.

I have been trying as many option combinations as I can think of. I
previously had regular (non raw) IO, and believe I needed the
allow_partial_reads for packets longer than len. I didn’t remove it
*just in case*.

> In any case, is winusb the right path to follow for this scenario or
> would a UMDF driver be more appropriate?
>

UMDF drivers use WinUSB as their transport. It’s a different API, but
the same provider.

OK.

> Could the PIPE_TRANSFER_TIMEOUT be causing the issue with the dropped
> ACKs even with RAW_IO? Should I move to use overlap in my reads?
>

When you say “dropped ACKs”, do you mean a byte of data being returned
on the pipe? The reason I ask is that “ACK” has a very specific meaning
in the USB world, and I’m guessing that’s not what you mean.

Here I mean an actual 1 byte response form the device with 6 (ACK) or 15
(NAK).

Miguel Sosa wrote:

3 devices use Bulk pipes; One uses interrupt (although it has one
interrupt for reads and another for writes).

Same with the bulk pipes, right? USB pipes are unidirectional (except
the control pipe).

The manufacturers do provide their own drivers. The ones that have
bulk pipes have virtual COM drivers (so I can use them with the serial
port code i already have). The one that uses interrupts works like a
HID device. I wrote a basic HID client for that one which seems to
work well.

Having said that, for simplicity to our customers I would prefer to
have just one way for all devices to communicate. I already use libusb
in linux to talk to all 4 devices and wanted to do something similar
in windows. I am following the progress in the libusb list of the
winusb port but I need something that can use right now.

If it were me, I’d be tempted to write a wrapper DLL that hides the
details underneath, then use the virtual COM driver where appropriate
and the HID client where appropriate. I certainly do understand wanting
to be able to control things at the lowest level, however.

I am using sniffUSB, and have used Snoopy. It has been very helpful,
except that I am still learning how to parse it. I can recognize the
packets with read/write data, and some of the control requests, but
most of the ones without data are still a mystery to me.

Any links that would help with learning how to parse the USB traffic?

Remember that you see the whole exchange: request going down, response
coming back. If you have a screenshot or dump output that confuses you,
send it along and we can interpret.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Tim Roberts wrote:

Miguel Sosa wrote:
> 3 devices use Bulk pipes; One uses interrupt (although it has one
> interrupt for reads and another for writes).
Same with the bulk pipes, right? USB pipes are unidirectional (except
the control pipe).
Yes.
If it were me, I’d be tempted to write a wrapper DLL that hides the
details underneath, then use the virtual COM driver where appropriate
and the HID client where appropriate. I certainly do understand wanting
to be able to control things at the lowest level, however.
Well, the other part is that it is also easier to maintain if I have
just one winusb code base that I can use for everything. :slight_smile:

> I am using sniffUSB, and have used Snoopy. It has been very helpful,
> except that I am still learning how to parse it. I can recognize the
> packets with read/write data, and some of the control requests, but
> most of the ones without data are still a mystery to me.
>
> Any links that would help with learning how to parse the USB traffic?

Remember that you see the whole exchange: request going down, response
coming back. If you have a screenshot or dump output that confuses you,
send it along and we can interpret.

Yes. I noticed I have two entries (up/down) for each URB. A couple of
questions,

between the BULK_OR_INTERRUPT_TRANSFERS I see:

[1621871 ms] UsbSnoop - MyInternalIOCTLCompletion(b75e8126) :
fido=00000000, Irp=89d96568, Context=896d4f80, IRQL=2

Which seems to be the result of the bulk write operation.

I also see:

[1621877 ms] UsbSnoop - FilterDispatchAny(b75e7fd2) :
IRP_MJ_INTERNAL_DEVICE_CONTROL
[1621879 ms] UsbSnoop - FdoHookDispatchInternalIoctl(b75e81ea) :
fdo=8978e8b8, Irp=89d96568, IRQL=0

Which I believe is the actual read request to the device. (it’s followed
by BULK_OR_INTERRUPT_TRANSFERS with the IN endpoint.)

Is there any guide to parse those messages?

Miguel Sosa wrote:

Yes. I noticed I have two entries (up/down) for each URB. A couple of
questions,

between the BULK_OR_INTERRUPT_TRANSFERS I see:

[1621871 ms] UsbSnoop - MyInternalIOCTLCompletion(b75e8126) :
fido=00000000, Irp=89d96568, Context=896d4f80, IRQL=2

Which seems to be the result of the bulk write operation.

Right. That tells you that a previous request has been completed and
sent back to WinUSB. The IRP will let you associate that with the
request submission, earlier in the log.

I also see:

[1621877 ms] UsbSnoop - FilterDispatchAny(b75e7fd2) :
IRP_MJ_INTERNAL_DEVICE_CONTROL
[1621879 ms] UsbSnoop - FdoHookDispatchInternalIoctl(b75e81ea) :
fdo=8978e8b8, Irp=89d96568, IRQL=0

Which I believe is the actual read request to the device. (it’s
followed by BULK_OR_INTERRUPT_TRANSFERS with the IN endpoint.)

Is there any guide to parse those messages?

There’s not too much more to extract from that. Those are just trace
messages through the driver. USB requests (like
BULK_OR_INTERRUPT_TRANSFER) are sent wrapped in an
IRP_MJ_INTERNAL_DEVICE_CONTROL IRP. Don’t you get anything that
indicates how many bytes were transferred?


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Tim Roberts wrote:

Miguel Sosa wrote:

> Is there any guide to parse those messages?
>

There’s not too much more to extract from that. Those are just trace
messages through the driver. USB requests (like
BULK_OR_INTERRUPT_TRANSFER) are sent wrapped in an
IRP_MJ_INTERNAL_DEVICE_CONTROL IRP. Don’t you get anything that
indicates how many bytes were transferred?

Yes. I get some traffic with apparently empty transfers and one transfer
with data.

My test operation does the following:

Every read requests is for 64 bytes since I don’t know ahead of time how
many bytes to request.

  • read from device to see if there is anything. Nothing there so it
    times out. (io is canceled).

  • write a command to device

  • read (expects 0x06 - gets nothing unless in RAW_IO)
    Now my code is waiting for input from the user. While doing so, it
    periodically checks if there is something to read:

  • 1…N read (time passes, call times out since there is nothing on
    the pipe until there is input from the user)

  • User does something - there are 99 bytes to read

  • read (expects 99 bytes - gets 36).

With timeouts enabled, I only received the last 36 bytes. The first 63
bytes of the response are lost. I only received the second packet of
data. In linux, doing the same operation I get 3 responses:

  • ACK
  • 63 bytes
  • 36 bytes

So something in winusb is either not requesting or canceling the request
of the middle packet before the device sends the data. Makes no sense to
me, but with all the reads/timeouts it could very well be happening.

With timeouts disabled, using overlap, and raw IO I get the 06, but I
need to AbortPipe after any read with data or after some period of time
with no data received. It seems to work but it looks very ugly with all
the abort packets in there.

I need to investigate this further. It is not apparent to me what I am
doing incorrect in my code (maybe it’s a windows developer newbie error)
or why either winusb or the device would buffer data this way.

Thanks for all your help and hints.