How are the USB mass storage and USB hub driver devnodes and drivers loaded?

I have written an answer on stack overflow but there are a couple of inaccuracies I’d like to change that I’m not sure of.

With reference to the following devnode chain: https://docs.microsoft.com/en-us/windows-hardware/drivers/gettingstarted/driver-stacks

  1. How does the hub driver get inserted, is it done in StartDevice of the xHCI driver? It could call IoInvalidateDeviceRelations and say it only has one child, the hub device.

  2. How does the USB mass storage node get inserted? Does the PnP manager notice that the PDO is a mass storage device , see that the DIID contains a USB\ prefix and then load usbstor.sys and create the devnode for that and change the driver object for the disk PDO so it points to the driver of the usb mass storage devnode, before consulting the registry for the DIID string?

  3. What actually calls IoInvalidateDeviceRelations in the first place. It has to be the xHCI driver right? Because it handles the event ring 0 interrupt. Is my thinking right: the xHCI driver calls IoInvalidateDeviceRelations with the PDO of the hub driver and the hub driver responds to the IRP_MN_QUERY_DEVICE_RELATIONS but then it would have to send an IRP to the xHCI controller to see what port statuses have changed to know how many PDOs to create/remove. Potentially the IoInvalidateDeviceRelations is not called on the PDO of the hub driver but rather the xHCI driver and the xHCI driver handles the IRP_MN_QUERY_DEVICE_RELATIONS and then it can check these changes internally then and pair PDOs to SlotIDs. If this were the case then it would create a device object that points to the driver object of the hub driver so that the devnodes are correctly positioned. The diagram shows the PDO belonging to usbstor.sys

On Apr 29, 2019, at 6:58 PM, Terrydaktal wrote:
>
> With reference to the following devnode chain: https://docs.microsoft.com/en-us/windows-hardware/drivers/gettingstarted/driver-stacks
>
> 1) How does the hub driver get inserted, is it done in StartDevice of the xHCI driver? It could call IoInvalidateDeviceRelations and say it only has one child, the hub device.

Device stacks are always built from the bottom up. A bus FDO creates PDOs for its child devices. The PnP subsystem loads an FDO for each PDO, which in turn creates more PDOs. Devices are never inserted into an existing stack.

> 2) How does the USB mass storage node get inserted? Does the PnP manager notice that the PDO is a mass storage device , see that the DIID contains a USB\ prefix and then load usbstor.sys and create the devnode for that and change the driver object for the disk PDO so it points to the driver of the usb mass storage devnode, before consulting the registry for the DIID string?

The USB hub driver creates a PDO. The PDO has several names, including one with the VID and PID, and one that identifies the USB device class from the descriptors. If there is no specific INF match for the VID and PID, then the USB mass storage INF will match the generic device class name. That loads usbstor.sys.

> 3) What actually calls IoInvalidateDeviceRelations in the first place. It has to be the xHCI driver right?

Each bus driver calls IoInvalidateDeviceRelations when it updates its set of child PDOs, to trigger the loading of the next layer of FDOs.

Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

Of course, everything Mr. Roberts says is correct.

To be clear: The Host Controller Driver is not the “bus driver” for the USB Bus. He’s the Controller Driver. He does act as a bus driver, but only so far as to enumerate the USB hub driver… at least one of which is (always) integral to the host controller itself.

The USB hub driver, once instantiated, is “the guy” who enumerates devices on the USB bus. Every USB device is connected to a hub. When the hub notices the arrival or departure of a device on his hub, he calls IoInvalidateDeviceRelations, which triggers PnP to (re-)send his query to the USB hub for his current list of child devices.

Peter

@“Peter_Viscarola_(OSR)” @Tim_Roberts Yes, I know how it usually works but the issue is the sources saying that the hub driver calls IoInvalidateDeviceRelations. How can it when it’s the xHCI driver that set up an interrupt for event ring 0 and knows the address of the event ring that it created in memory so it can read from it. If the interrupt is actually registered by the hub driver then it would still have to know the address of the event ring to handle it, how would the drivers exchange this information? If the xHCI driver is the one that registers the interrupt and reads the event on ring 0, what’s the point in cramming this into an IRP that causes the hub driver to invoke IoInvalidateDeviceRelations when it could just call it on the PDO of the hub driver itself, job done. The issue here is, the hub driver will now be the one thst responds to the event but now it needs to know which ports have had an insertion / removal on and to do that it needs to read a host controller register but only the xHCI driver knows where this is in memory. Instead of the hub driver sending an IRP to perform this read action, the xHCI controller could do all the initialisation on the port and then send an IRP to the hub driver with the slot ID that has been added and removed and it will call IoInvalidateDeviceRelations and add and remove PDOs in the IRP it responds to that correspond to the slot IDs.

Now there is the issue of usbstor. If the prefix + VID +PID + instanceID combo leads to the PnP manager loading usbstor.sys for the PDO it queries then how does usbstor.sys know that it needs to load a child and call IoInvalidateDeviceRelations itself? My only guess is that the prefix + VID + PID + InstanceID returned by the usb hub driver is a dummy one that will get usbstor.sys loaded by the PnP manager if the hub driver notices the PDO is a mass storage type. Then, usbstor.sys has to load the correct device, how does it know it has a child? And somehow, it needs to be able to correctly respond to IRPs that query the ID of the device. Do you reckon there is a mechanism in which it can send a get_descriptor request in an IRP to the xHCI driver and attach the slotID to perform the action on, it passes it down to the hub driver, the hub doesn’t implement and the xHCI driver responds. It does seem like a bit of a tangle to me.

Regards, Lewis.

As some USB devices do not have this USB mass storage interstitial node, the hub driver must do the get_descriptor request on the slot ID (if they don’t share knowledge of an address space then the xHCI driver will have to expose an API through the means of IRPs), otherwise it leaves that to the usbstor.sys.

Also, the xHCI driver is able to send an IRP to the hub driver using the PDO it created for it. I assume PDOs are stored as globals.

I assume usbstor.sys knows by definition that it will have a child so it waits for an IRP from the hub driver that passes the slot ID of the new device before it sends a get_descriptor IRP to the XHCI driver for the slot ID.

Terrydaktal wrote:

Yes, I know how it usually works but the issue is the sources saying that the hub driver calls IoInvalidateDeviceRelations. How can it when it’s the xHCI driver that set up an interrupt for event ring 0 and knows the address of the event ring that it created in memory so it can read from it.

It’s not really that hard to understand, is it?  Just because the XHCI
driver talks to the wire does not mean that it is responsible for all
the devices that happen to appear on the wire.  I don’t know that this
is the exact sequence, but here’s one potential architecture.  The HCD
(the xHCI driver) gets a signal from a physical hub saying “a new device
has arrived”.  At that point, the HCD doesn’t know anything about it,
and it doesn’t even know the hub protocols.  It has child drivers that
are hub drivers (one for each hub) to handle those details.  It sends a
signal to the appropriate hub driver saying “hey, wake up, a device
arrived”. The hub driver then sends packets to the hub it controls
telling it to enumerate the device and assign an ID.  The HCD happily
sends those packets and returns the results to the hub driver.  At that
point, the hub driver knows the VID and PID for the new device, and
creates a child PDO.

If the interrupt is actually registered by the hub driver then it would still have to know the address of the event ring to handle it, how would the drivers exchange this information?

Interrupt ownership is totally irrelevant.  This is a protocol driver
stack with multiple layers.  The HCD doesn’t understand hub protocol. 
It understands packet queues and packet scheduling.  In a normal device
stack, the hub driver would submit an IRP to the HCD that remains
pending until some event happens.  When a device appears, the HCD will
complete that request, telling the hub driver to do investigate.

If the xHCI driver is the one that registers the interrupt and reads the event on ring 0, what’s the point in cramming this into an IRP that causes the hub driver to invoke IoInvalidateDeviceRelations when it could just call it on the PDO of the hub driver itself, job done.

Encapsulation,  division of labor, ease of future expansion. Each HCD
has many hubs.  Each hub has many devices (including more hubs).  Each
device has many interfaces.  The best architecture lets each of those
things be handled by one driver that understands its one function, and
lets other drivers handle the other protocols.  You can replace the HCD
with your own (for example, for USB-over-Ethernet) without having to
rewrite the entire device stack.  As long as you know how to tickle the
USB hub driver, it can continue to do the same job.

The issue here is, the hub driver will now be the one thst responds to the event but now it needs to know which ports have had an insertion / removal on and to do that it needs to read a host controller register but only the xHCI driver knows where this is in memory.

It’s not a host controller register.  It’s a packet from the hub.  The
HCD treats it the same as every other packet, handing it off to the
driver that knows how to handle it.  IRPs are cheap

Instead of the hub driver sending an IRP to perform this read action, the xHCI controller could do all the initialisation on the port and then send an IRP to the hub driver with the slot ID that has been added and removed and it will call IoInvalidateDeviceRelations and add and remove PDOs in the IRP it responds to that correspond to the slot IDs.

Why should the HCD have to know anything about hubs?  A hub is just
another device, with its own request protocols.  Let a hub driver handle
that.  And remember that a lower device cannot send an IRP to an upper
device.  Requests always flow down, towards the hardware.

Now there is the issue of usbstor. If the prefix + VID +PID + instanceID combo leads to the PnP manager loading usbstor.sys for the PDO it queries then how does usbstor.sys know that it needs to load a child and call IoInvalidateDeviceRelations itself? My only guess is that the prefix + VID + PID + InstanceID returned by the usb hub driver is a dummy one that will get usbstor.sys loaded by the PnP manager if the hub driver notices the PDO is a mass storage type. Then, usbstor.sys has to load the correct device, how does it know it has a child?

This is simply not how it works.  The hub driver doesn’t know anything
about audio or video or mass storage or mice or keyboards.  The hub
driver sees that a new device with VID=1234 and PID=5678 and DID=0100
and class 08 in its descriptors.  The hub driver creates a new PDO and
gives it the following hardware IDs:

    USB\VID_1234&PID_5678&DID_0100
    USB\VID_1234&PID_5678
    USB\Class_08&SubClass=02&Prot=50
    USB\Class_08&SubClass=02
    USB\Class_08

At that point, the hub driver calls IoInvalidateDeviceRelations to tell
PnP that there is a new PDO to deal with.  At that point, the hub
driver’s job is through.  It has done everything it needs to do.

Here, PnP takes over.  It starts at the top of that list of IDs and
searches for INF files that match it.  If there is no specific driver
for this device, the third ID, the one that identifies it as a generic
mass storage device, is found in usbstor.inf, so PnP loads usbstor.sys
and hands it the unique device identifier.

Usbstor.sys now creates URBs to go identify the device and start to
configure it.  It sends requests to the hub driver, which (possibly)
reformats them and sends them to the HCD driver, which turns them into
scheduled packets.  At some point, usbstor.sys will probably find a
volume on that device.  It will create a child PDO with an identifier
like SCSI\My_Device_Name.  That will trigger the loading of a driver
that understands SCSI protocol but doesn’t know anything about
communicating with hardware.

Every device knows its piece of the puzzle.  As long as they follow the
interface rules, it becomes easy to add new types of devices, or to use
existing drivers with new buses.

And somehow, it needs to be able to correctly respond to IRPs that query the ID of the device. Do you reckon there is a mechanism in which it can send a get_descriptor request in an IRP to the xHCI driver and attach the slotID to perform the action on, it passes it down to the hub driver, the hub doesn’t implement and the xHCI driver responds.

Each layer has a well defined protocol.  Usbstor accepts disk requests
at the top.  It translates them into URBs for its device, which it hands
off to the hub driver.  The hub driver might make some adjustments to
the request, and hands it off to the HCD.

It does seem like a bit of a tangle to me.

You just aren’t seeing the bigger picture.

Terrydaktal wrote:

As some USB devices do not have this USB mass storage interstitial mode, the hub driver must do the get_descriptor request on the slot ID (if they don’t share knowledge of an address space then the xHCI driver will have to expose an API through the means of IRPs), otherwise it leaves that to the usbstor.sys

I can’t guess what you mean by “USB mass storage interstitial mode”.

The “slot ID” is a detail that doesn’t have any meaning to any driver
above the hub.  Look, the HCD creates a series of hub PDOs, one driver
instance per hub.  Each PDO knows the USB device ID for the hub it has
been assigned.  The hub driver creates a series of device PDOs, one
instance per deivce.  Each PDO knows the slot ID of the device it has
been assigned.  If it is a composite device (with multiple interfaces),
usbccgp creates another set of PDOs, one per interface.  Each PDO knows
the interface it was assigned to handle (and lies to the driver above it
about the interface number).  So, when usbstor sends an URB, it doesn’t
give a whack what its low-level path is.  It doesn’t care which
interface it is, or which port it is on, or which hub on which
controller it is on, or even what type of controller it is.  It sends an
URB.  The composite PDO fixes up the interface and sends to the hub. 
The hub PDO adds information to the URB that identifies the slot ID and
hands it to the next level.  By the time it gets to the HCD, it knows
which USB device ID to place in the packet header.

It’s just that easy.  Every level handles its small part of the
equation, without having to worry about piddling details that don’t make
a difference.

I suppose you’re right. I get obsessed with the minute details but I do understand the general process but there are contradictions and ambiguities on the microscopic level that I feel the urge to iron out. Thanks for taking the time to reply. I’m going to read your answer again and further brainstorm and consolidate and hopefully make progress

Terrydaktal wrote:

I assume usbstor.sys knows by definition that it will have a child so it waits for an IRP from the hub driver that passes the slot ID of the new device before it sends a get_descriptor IRP to the XHCI driver for the slot ID.

I’ve already spewed forth today, but this explanation has things
backwards.  Usbstor.sys doesn’t sit around idle waiting for stuff to
do.  Usbstor.sys won’t even be loaded until a hub driver creates a PDO
that triggers its loading.  It doesn’t need to know the slot ID, because
the PDO that triggered it identifies a unique device, and the hub
already knows which slot it is.  By the time usbstor’s requests get down
to the HCD, the hub and slot number have been translated to a device ID
to go out on the wire.

Yes, I know how it usually works

I do understand the general process

I’m not sure that you do. In fact, if I had to guess, I’d say you don’t.

You should take one of our seminars.

I assume PDOs are stored as globals.

What?

there are contradictions and ambiguities on the microscopic level

What? This is software, things work the way they work. They don’t work ambiguously. Software instructions can’t be amgiguous (oh, Anton… cut me some slack here, please).

Mr. Roberts and I have already explained a great deal to you. You are either apparently not taking the time to fully read and absorb what we’ve written, or you’re not understanding what we’ve tried to explain for whatever reason.

Peter