Pci Express QoS support in windows Driver

NTDEV
1

Hi, All

I am doing Pci Express Technical support (but basically in pre-os
environment) and someone asked me this question which I find it hard to
answer. Can anyone here please give me some clues?

We know Pci Express support QoS by introducing TC (Traffic Class) and
VC(Virtual Channel) and other arbitration algorithm to dispense the
available bandwidth among all the requesters. In Pre-OS environment we
cared little about this capability. But it can be envisioned that this
is very useful technology in OS environment while data can be easily
classified by its nature such as video data, network data, etc. However,
since Pci Express is still somewhat new to OS (Microsoft official
documents indicate that Pci Express will be OS aware only in Vista and
its successors), I have no idea how this can be supported in WDM
framework. (Maybe in WDF?)

My original idea is that the chipset driver is supplied by the chipset
vendor which has the greatest knowledge about root complex. This driver
exposes certain interfaces with predefined GUID to let other registering
drivers know when the QoS premise is initialized and how it is supported
(e.g. how many TCs are there). The video driver, for example, can query
this interface and use whatever method that is native to the video card,
to direct the card to send data flow at escalated TC.

At least this is the working model that has been proposed by some new
Chipset DMA engines.

The puzzles obsessing me here is that:

  1. Is my guess here correct? There could be other working models
    anyway.

  2. Is any device driver using on modern platforms already doing so
    or the capability is just wasted?

  3. Any changes in WDF so this could be supported?

  4. Any insight in Vista and other upcoming OS how this capability
    will be supported and utilized?

Thanks very much in advance for anyone who can share me his insights.

Best regards,

Cody

2

Wu, Cody wrote:

Hi, All

I am doing Pci Express Technical support (but basically in pre-os
environment) and someone asked me this question which I find it hard
to answer. Can anyone here please give me some clues?

We know Pci Express support QoS by introducing TC (Traffic Class) and
VC(Virtual Channel) and other arbitration algorithm to dispense the
available bandwidth among all the requesters.

…virtually none of which is actually implemented in today’s PCI
Express. The specification made virtual channels optional, which means
no one is implementing them. There was talk a year or two ago of
removing the whole VC concept from the spec.

In Pre-OS environment we cared little about this capability. But it
can be envisioned that this is very useful technology in OS
environment while data can be easily classified by its nature such as
video data, network data, etc. However, since Pci Express is still
somewhat new to OS (Microsoft official documents indicate that Pci
Express will be OS aware only in Vista and its successors), I have no
idea how this can be supported in WDM framework. (Maybe in WDF?)

Nope. None of this is exposed to drivers, even in Vista. There may be
custom ioctls to the PCI Express bus driver, but I haven’t seen them.
There is a white paper on the WHDC web site on PCI Express support in
Vista. Do you have that?

My original idea is that the chipset driver is supplied by the chipset
vendor which has the greatest knowledge about root complex. This
driver exposes certain interfaces with predefined GUID to let other
registering drivers know when the QoS premise is initialized and how
it is supported (e.g. how many TCs are there). The video driver, for
example, can query this interface and use whatever method that is
native to the video card, to direct the card to send data flow at
escalated TC.

It’s not at all clear that this kind of management is worth the
trouble. Why is it up to the video driver to decide that it deserves to
be the highest traffic class? The only side effect of raising someone
to the highest class is that you lock out someone else. The only
reasonable way to run a bus is to share the bandwidth evenly. Remember
that bandwidth competition happens only at the root complex; the
individual slots are all point-to-point. As of now, the root complexes
are fast enough that sharing gives the best overall experience.

The puzzles obsessing me here is that:

  1. Is my guess here correct? There could be other working models
    anyway.

  2. Is any device driver using on modern platforms already doing
    so or the capability is just wasted?

I have heard rumors of drivers doing PCI Express isochronous transfers
by directly programming the root complex chipsets. I have not heard of
any traffic class management at all.

  1. Any changes in WDF so this could be supported?

How would you do that? What would the API look like? Couldn’t you
argue that this should be managed by the device rather than the driver?

  1. Any insight in Vista and other upcoming OS how this
    capability will be supported and utilized?

I suspect it won’t be. Too much trouble, not enough gain.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

3

Hi, Tim

Thanks for your reply and detailed explanations. After some
investigation I came to the following conclusions, maybe wrong or
inaccurate, but I would like to get your opinions:

  1. Vista has no support for virtual channel other than VC0 which is the
    enforced default in Pci Express Spec. This is equivalent to say QoS
    would not be supported in Vista at least.
  2. I am not clear about how many IHVs are building Pci Express devices
    with QoS support. The root complex in the recent models (both desktop
    and server board) all have this feature natively supported. However,
    IHVs may not benefit from this now that Vista has no intention to
    support it. This may just reduce their incentive to implement this in
    their cards.
  3. I doubt if the bandwidth inside the system is also abundant for
    everyone. Even if it is now, sooner or later people will devise new
    devices to consume that. In the long run, QoS may be a wise choice. And
    I don’t think the functionality of the bus driver is to evenly
    distribute the available bandwidth. You can tolerate a slow network
    connection but you may feel totally unacceptable with the slow stepping
    video frames. I bet we should not be too optimistic in this anyway.
    That’s why the pci Express spec allows you several different algorithms
    to tell how the bandwidth should be divided among competing requesters.
    You could, however, use round robin if you believe in fair share.
  4. Yes, I have done some searching on the internet and found some WinHec
    documents on Pci Express particularly informative. Thanks for the
    information.
  5. I don’t exclude the possibility for some determined IHV to do direct
    query towards root complex but this is certainly not the right way to do
    it.

Though it is hard to tell the future personally I do feel we will find
the feature useful someday. QoS in a WAN may be too difficult to
implement (yet still there are successful examples), but it will be more
practical for a much shrank computer system interconnect topology.

Thanks again for your explanation!

Best regards,
Cody

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tim Roberts
Sent: Tuesday, November 21, 2006 2:08 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Pci Express QoS support in windows Driver

Wu, Cody wrote:

Hi, All

I am doing Pci Express Technical support (but basically in pre-os
environment) and someone asked me this question which I find it hard
to answer. Can anyone here please give me some clues?

We know Pci Express support QoS by introducing TC (Traffic Class) and
VC(Virtual Channel) and other arbitration algorithm to dispense the
available bandwidth among all the requesters.

…virtually none of which is actually implemented in today’s PCI
Express. The specification made virtual channels optional, which means
no one is implementing them. There was talk a year or two ago of
removing the whole VC concept from the spec.

In Pre-OS environment we cared little about this capability. But it
can be envisioned that this is very useful technology in OS
environment while data can be easily classified by its nature such as
video data, network data, etc. However, since Pci Express is still
somewhat new to OS (Microsoft official documents indicate that Pci
Express will be OS aware only in Vista and its successors), I have no
idea how this can be supported in WDM framework. (Maybe in WDF?)

Nope. None of this is exposed to drivers, even in Vista. There may be
custom ioctls to the PCI Express bus driver, but I haven’t seen them.
There is a white paper on the WHDC web site on PCI Express support in
Vista. Do you have that?

My original idea is that the chipset driver is supplied by the chipset
vendor which has the greatest knowledge about root complex. This
driver exposes certain interfaces with predefined GUID to let other
registering drivers know when the QoS premise is initialized and how
it is supported (e.g. how many TCs are there). The video driver, for
example, can query this interface and use whatever method that is
native to the video card, to direct the card to send data flow at
escalated TC.

It’s not at all clear that this kind of management is worth the
trouble. Why is it up to the video driver to decide that it deserves to
be the highest traffic class? The only side effect of raising someone
to the highest class is that you lock out someone else. The only
reasonable way to run a bus is to share the bandwidth evenly. Remember
that bandwidth competition happens only at the root complex; the
individual slots are all point-to-point. As of now, the root complexes
are fast enough that sharing gives the best overall experience.

The puzzles obsessing me here is that:

  1. Is my guess here correct? There could be other working models
    anyway.

  2. Is any device driver using on modern platforms already doing
    so or the capability is just wasted?

I have heard rumors of drivers doing PCI Express isochronous transfers
by directly programming the root complex chipsets. I have not heard of
any traffic class management at all.

  1. Any changes in WDF so this could be supported?

How would you do that? What would the API look like? Couldn’t you
argue that this should be managed by the device rather than the driver?

  1. Any insight in Vista and other upcoming OS how this
    capability will be supported and utilized?

I suspect it won’t be. Too much trouble, not enough gain.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

4

Wu, Cody wrote:

  1. I doubt if the bandwidth inside the system is also abundant for
    everyone. Even if it is now, sooner or later people will devise new
    devices to consume that. In the long run, QoS may be a wise choice. And
    I don’t think the functionality of the bus driver is to evenly
    distribute the available bandwidth. You can tolerate a slow network
    connection but you may feel totally unacceptable with the slow stepping
    video frames. I bet we should not be too optimistic in this anyway.

As a counter-example, consider USB. There are now untold millions of
USB devices, plugged in to real-world systems in far wider variety and
vastly larger numbers than we are ever going to see for PCI Express.
And even with all of that variety and all of those demands, USB works
perfectly well without QoS.

In my own humble opinion, and I have been known to be wrong once or
twice, PCI Express QoS is simply not useful in the general case. In
very specific cases with very specific configurations, I believe it
could be useful for micro-optimization, but that’s just not the real
world for the Windows PC. You need to be able to apply rules, and PCs
are just too heterogeneous for rules to work.

  1. I don’t exclude the possibility for some determined IHV to do direct
    query towards root complex but this is certainly not the right way to do
    it.

As of now, it’s the only way. There are no APIs in XP or Server 2003
for doing this.

Though it is hard to tell the future personally I do feel we will find
the feature useful someday.

Perhaps. Color me “dubious”.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

5

Hi Tim,

Thanks very much for your information.

By the way, do you have any idea how MSI(MIS-X) is going to be supported
in Window Vista? The foil on the Microsoft website claims it is
supporting this officially. I believe it is pretty easy to device
drivers to generate interrupt by doing memory writes. What puzzles me is
who is going to take the responsibility of programming resources into
these capability registers. Will it be OS? As far as I know, current
BIOSes generally don’t take the effort to do this. Again there are two
questions:

  1. Before Vista, no one ever used it?
  2. How will Vista support it if 1) is true? I mean, any special handling
    aside from allocating resources and connecting the destination interrupt
    to its service routines?

Best regards,
Cody

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tim Roberts
Sent: Tuesday, November 21, 2006 10:14 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Pci Express QoS support in windows Driver

Wu, Cody wrote:

  1. I doubt if the bandwidth inside the system is also abundant for
    everyone. Even if it is now, sooner or later people will devise new
    devices to consume that. In the long run, QoS may be a wise choice.
    And
    I don’t think the functionality of the bus driver is to evenly
    distribute the available bandwidth. You can tolerate a slow network
    connection but you may feel totally unacceptable with the slow
    stepping
    video frames. I bet we should not be too optimistic in this anyway.

As a counter-example, consider USB. There are now untold millions of
USB devices, plugged in to real-world systems in far wider variety and
vastly larger numbers than we are ever going to see for PCI Express.
And even with all of that variety and all of those demands, USB works
perfectly well without QoS.

In my own humble opinion, and I have been known to be wrong once or
twice, PCI Express QoS is simply not useful in the general case. In
very specific cases with very specific configurations, I believe it
could be useful for micro-optimization, but that’s just not the real
world for the Windows PC. You need to be able to apply rules, and PCs
are just too heterogeneous for rules to work.

  1. I don’t exclude the possibility for some determined IHV to do
    direct
    query towards root complex but this is certainly not the right way to
    do
    it.

As of now, it’s the only way. There are no APIs in XP or Server 2003
for doing this.

Though it is hard to tell the future personally I do feel we will find
the feature useful someday.

Perhaps. Color me “dubious”.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

6

Hi Tim,

Please ignore the previous email as I have found the whitepaper on the
topic. The original link in the winHec foil is invalid so it took me
some time to find it. Thanks for your help! :slight_smile:

Best regards,
Cody

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Wu, Cody
Sent: Tuesday, November 21, 2006 4:58 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Pci Express QoS support in windows Driver

Hi Tim,

Thanks very much for your information.

By the way, do you have any idea how MSI(MIS-X) is going to be supported
in Window Vista? The foil on the Microsoft website claims it is
supporting this officially. I believe it is pretty easy to device
drivers to generate interrupt by doing memory writes. What puzzles me is
who is going to take the responsibility of programming resources into
these capability registers. Will it be OS? As far as I know, current
BIOSes generally don’t take the effort to do this. Again there are two
questions:

  1. Before Vista, no one ever used it?
  2. How will Vista support it if 1) is true? I mean, any special handling
    aside from allocating resources and connecting the destination interrupt
    to its service routines?

Best regards,
Cody

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tim Roberts
Sent: Tuesday, November 21, 2006 10:14 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Pci Express QoS support in windows Driver

Wu, Cody wrote:

  1. I doubt if the bandwidth inside the system is also abundant for
    everyone. Even if it is now, sooner or later people will devise new
    devices to consume that. In the long run, QoS may be a wise choice.
    And
    I don’t think the functionality of the bus driver is to evenly
    distribute the available bandwidth. You can tolerate a slow network
    connection but you may feel totally unacceptable with the slow
    stepping
    video frames. I bet we should not be too optimistic in this anyway.

As a counter-example, consider USB. There are now untold millions of
USB devices, plugged in to real-world systems in far wider variety and
vastly larger numbers than we are ever going to see for PCI Express.
And even with all of that variety and all of those demands, USB works
perfectly well without QoS.

In my own humble opinion, and I have been known to be wrong once or
twice, PCI Express QoS is simply not useful in the general case. In
very specific cases with very specific configurations, I believe it
could be useful for micro-optimization, but that’s just not the real
world for the Windows PC. You need to be able to apply rules, and PCs
are just too heterogeneous for rules to work.

  1. I don’t exclude the possibility for some determined IHV to do
    direct
    query towards root complex but this is certainly not the right way to
    do
    it.

As of now, it’s the only way. There are no APIs in XP or Server 2003
for doing this.

Though it is hard to tell the future personally I do feel we will find
the feature useful someday.

Perhaps. Color me “dubious”.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

7

Tim Roberts writes:

As a counter-example, consider USB. There are now untold millions of
USB devices, plugged in to real-world systems in far wider variety and
vastly larger numbers than we are ever going to see for PCI Express.
And even with all of that variety and all of those demands, USB works
perfectly well without QoS.

I agree, but with an important qualification. USB works because it *does* support QoS, but in such a way that the hardware designer and device driver developer don’t need to care about it. (I’m sure you know this, so I’m writing only for the benefit of others.)

USB works because there is a well-defined priority scheme, which controls how the USB is shared between control, isoch, interrupt, and bulk transfers. Because the USB spec clearly defines the types of transfers and their relative priorities, and because there are only a handful of HCI implementations (plus their associated bus drivers), the part of the overall picture that cares about QoS is simple enough to get it right. This frees the USB device designer from caring about it, or having any power over it (i.e. power to screw things up). Very few devices actually need any QoS, and those that do use isochronous transfers.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Tim Roberts
Sent: Monday, November 20, 2006 6:14 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Pci Express QoS support in windows Driver

Wu, Cody wrote:

  1. I doubt if the bandwidth inside the system is also abundant for
    everyone. Even if it is now, sooner or later people will devise new
    devices to consume that. In the long run, QoS may be a wise choice. And
    I don’t think the functionality of the bus driver is to evenly
    distribute the available bandwidth. You can tolerate a slow network
    connection but you may feel totally unacceptable with the slow stepping
    video frames. I bet we should not be too optimistic in this anyway.

As a counter-example, consider USB. There are now untold millions of
USB devices, plugged in to real-world systems in far wider variety and
vastly larger numbers than we are ever going to see for PCI Express.
And even with all of that variety and all of those demands, USB works
perfectly well without QoS.

In my own humble opinion, and I have been known to be wrong once or
twice, PCI Express QoS is simply not useful in the general case. In
very specific cases with very specific configurations, I believe it
could be useful for micro-optimization, but that’s just not the real
world for the Windows PC. You need to be able to apply rules, and PCs
are just too heterogeneous for rules to work.

  1. I don’t exclude the possibility for some determined IHV to do direct
    query towards root complex but this is certainly not the right way to do
    it.

As of now, it’s the only way. There are no APIs in XP or Server 2003
for doing this.

Though it is hard to tell the future personally I do feel we will find
the feature useful someday.

Perhaps. Color me “dubious”.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer