How dynamically re-assign bus numbers

Hi All,

I’m working on the driver for specific device connected to the system via ExpressCard. Operating system is Windows 7. Device itself represent two bridges with a secondary buse each. When ExpressCard is inserted by system startup, Windows enumerates buses properly:

[0:1C:0] PCIe Port1-> [0B - 0B]
[0:1C:1] PCIe Port2-> [0C - 0C]
[0:1C:2] PCIe Port3-> [0D - 0D]
[0:1C:3] PCIe Port4-> [0E - 10] -> [0E:0:0] Bridge1 -> [0F - 10] -> [0F:0:0] Bridge2 -> [10-10] - OK
[0:1C:4] PCIe Port5-> [11 - 11]
[0:1C:5] PCIe Port6-> [09 - 09]

However, when ExpressCard is hot inserted, Windows multi-level rebalancing works wrongly:

[0:1C:0] PCIe Port1-> [0B - 0B]
[0:1C:1] PCIe Port2-> [0C - 0C]
[0:1C:2] PCIe Port3-> [0D - 0B]
[0:1C:3] PCIe Port4-> [0E - 0F] -> [0E:0:0] Bridge1 -> [0F - 10] -> [0F:0:0] Bridge2 ->Fails
[0:1C:4] PCIe Port5-> [10 - 10]
[0:1C:5] PCIe Port6-> [09 - 09]

Visibly, in the second case, configuration fails because there is not enough buses reserved for PCIe Port 4. My questions:

  1. How can I from driver for Bridge1 force rebalancing on PCIe Port 4?
    Note: I tried already simple things like IoInvalidateDeviceState
  2. Where pci.sys saves information which secondary/subordinate bus to assign to each bridge in the system during the initial enumeration?

Thank you in advance for any hint.

With best regards,

Al

To short circuit this, your driver cannot change the way bus numbers are assigned, either through hw resource assignment or touching internal PCI.sys structures.

d

debt from my phone


From: xxxxx@hotmail.com
Sent: 3/26/2012 6:26 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] How dynamically re-assign bus numbers

Hi All,

I’m working on the driver for specific device connected to the system via ExpressCard. Operating system is Windows 7. Device itself represent two bridges with a secondary buse each. When ExpressCard is inserted by system startup, Windows enumerates buses properly:

[0:1C:0] PCIe Port1-> [0B - 0B]
[0:1C:1] PCIe Port2-> [0C - 0C]
[0:1C:2] PCIe Port3-> [0D - 0D]
[0:1C:3] PCIe Port4-> [0E - 10] -> [0E:0:0] Bridge1 -> [0F - 10] -> [0F:0:0] Bridge2 -> [10-10] - OK
[0:1C:4] PCIe Port5-> [11 - 11]
[0:1C:5] PCIe Port6-> [09 - 09]

However, when ExpressCard is hot inserted, Windows multi-level rebalancing works wrongly:

[0:1C:0] PCIe Port1-> [0B - 0B]
[0:1C:1] PCIe Port2-> [0C - 0C]
[0:1C:2] PCIe Port3-> [0D - 0B]
[0:1C:3] PCIe Port4-> [0E - 0F] -> [0E:0:0] Bridge1 -> [0F - 10] -> [0F:0:0] Bridge2 ->Fails
[0:1C:4] PCIe Port5-> [10 - 10]
[0:1C:5] PCIe Port6-> [09 - 09]

Visibly, in the second case, configuration fails because there is not enough buses reserved for PCIe Port 4. My questions:

  1. How can I from driver for Bridge1 force rebalancing on PCIe Port 4?
    Note: I tried already simple things like IoInvalidateDeviceState
  2. Where pci.sys saves information which secondary/subordinate bus to assign to each bridge in the system during the initial enumeration?

Thank you in advance for any hint.

With best regards,

Al


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Hi Doron,

Thank you for your reply.

However, Windows 7 on startup without Express Card assigns 2 buses to PCIe Port 4, while only one bus is assigned to other PCIe ports. Furtermore buses are not assigned in order (Bus 9 is assigned to PCIe Port 5).
I would assume that either Windows saves initial assignment of PCI buses somewhere in the registry. Question is where? Another possibility is that pci.sys uses configuration assigned by BIOS. In such case, it should be a possible to change this configuration even before pci.sys is started. Isn’t?

With best regards,

Al

A test on win8 CP, while you’re waiting for answers, may be helpful.

–pa

On 26-Mar-2012 15:25, xxxxx@hotmail.com wrote:

Hi All,

I’m working on the driver for specific device connected to the system via ExpressCard. Operating system is Windows 7. Device itself represent two bridges with a secondary buse each. When ExpressCard is inserted by system startup, Windows enumerates buses properly:

[0:1C:0] PCIe Port1-> [0B - 0B]
[0:1C:1] PCIe Port2-> [0C - 0C]
[0:1C:2] PCIe Port3-> [0D - 0D]
[0:1C:3] PCIe Port4-> [0E - 10] -> [0E:0:0] Bridge1 -> [0F - 10] -> [0F:0:0] Bridge2 -> [10-10] - OK
[0:1C:4] PCIe Port5-> [11 - 11]
[0:1C:5] PCIe Port6-> [09 - 09]

However, when ExpressCard is hot inserted, Windows multi-level rebalancing works wrongly:

[0:1C:0] PCIe Port1-> [0B - 0B]
[0:1C:1] PCIe Port2-> [0C - 0C]
[0:1C:2] PCIe Port3-> [0D - 0B]
[0:1C:3] PCIe Port4-> [0E - 0F] -> [0E:0:0] Bridge1 -> [0F - 10] -> [0F:0:0] Bridge2 ->Fails
[0:1C:4] PCIe Port5-> [10 - 10]
[0:1C:5] PCIe Port6-> [09 - 09]

Visibly, in the second case, configuration fails because there is not enough buses reserved for PCIe Port 4. My questions:

  1. How can I from driver for Bridge1 force rebalancing on PCIe Port 4?
    Note: I tried already simple things like IoInvalidateDeviceState
  2. Where pci.sys saves information which secondary/subordinate bus to assign to each bridge in the system during the initial enumeration?

Thank you in advance for any hint.

With best regards,

Al

Thank you, Pavel A. In my initial post is specified that I’m asking about Windows 7

On 26-Mar-2012 22:21, xxxxx@hotmail.com wrote:

Thank you, Pavel A. In my initial post is specified that I’m asking about Windows 7

So, it is useful for you to know about Win8.
If your card works with Win8, you can request the fix backported to
Win7. If not - file a bug against win8, it may get some attention.

– pa

Pavel, please stop it. We need solution for existing Windows 7 and not back-ported thing. If you know solution, please tell us. If not, please do not participate in this thread.

xxxxx@hotmail.com wrote:

Pavel, please stop it. We need solution for existing Windows 7 and not back-ported thing. If you know solution, please tell us. If not, please do not participate in this thread.

I think you are being unreasonable. You seem to be describing what
might be a problem in Windows. If you can show that the problem is
fixed in Windows 8, then I don’t see what’s wrong with pursuing a course
of action that tries to get Microsoft to port that fix back to Windows
7. That is SURELY a far better solution that coding up some half-assed
hack job that you downloaded off of a tricks web site somewhere.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

You are assuming you can fix this with a registry setting or API call. Not so. Rebalance is expensive. What you are seeing could very well be by design or a limitation of rebalance on win7. The suggestion to test on w8 is a very good one. If it still shows up in w8, it will probably be more difficult for you to get this fixed and thus realize you must take another course of action

d

debt from my phone


From: Tim Roberts
Sent: 3/26/2012 5:42 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] How dynamically re-assign bus numbers

xxxxx@hotmail.com wrote:

Pavel, please stop it. We need solution for existing Windows 7 and not back-ported thing. If you know solution, please tell us. If not, please do not participate in this thread.

I think you are being unreasonable. You seem to be describing what
might be a problem in Windows. If you can show that the problem is
fixed in Windows 8, then I don’t see what’s wrong with pursuing a course
of action that tries to get Microsoft to port that fix back to Windows
7. That is SURELY a far better solution that coding up some half-assed
hack job that you downloaded off of a tricks web site somewhere.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Gentlemen,

Thank you for your suggestions. We certainly will test under Win8. However right now we need a solution for Windows 7. I already understood from the very first reply from Doron that dynamic rebalancing will not be possible. However, it should not be a problem to reserve certain bus ranges at startup. Once again, is anybody know where Windows store the information used for initial buses configuration?

With best regards,

Al

Windows doesn’t renumber PCI buses. It will assign bus numbers to buses
which seem to have none. But it won’t change what the BIOS did.

This is a stance that we’ve come to after many years of debugging machines
where the BIOS has implicit assumptions that the bus numbers would never
change and/or machines where the “bridges” in the chipset weren’t really PCI
at all and they weren’t exactly PCI compliant.

So, to answer your question, many machines will function if you renumber the
PCI bus numbers before Windows takes control. Some others won’t. In either
case, Windows will think that the BIOS did it an preserve what you’ve done.
I think that you’re making a mistake with this strategy, however. The only
safe way to deal with bus numbers is to leave it to the BIOS.

Jake Oshins
(most recent guy to touch all this code in PCI.sys)
Windows Kernel Team

This post implies no warrantees and confers no rights.

wrote in message news:xxxxx@ntdev…

Hi Doron,

Thank you for your reply.

However, Windows 7 on startup without Express Card assigns 2 buses to PCIe
Port 4, while only one bus is assigned to other PCIe ports. Furtermore buses
are not assigned in order (Bus 9 is assigned to PCIe Port 5).
I would assume that either Windows saves initial assignment of PCI buses
somewhere in the registry. Question is where? Another possibility is that
pci.sys uses configuration assigned by BIOS. In such case, it should be a
possible to change this configuration even before pci.sys is started. Isn’t?

With best regards,

Al

Yes, I know exactly where Windows stores that information: in internal
undocumented data structures, in RAM. This isn’t something you can safely
manipulate.

Jake Oshins
Windows Kernel Team

This post implies no warrantees and confers no rights.

wrote in message news:xxxxx@ntdev…

Gentlemen,

Thank you for your suggestions. We certainly will test under Win8. However
right now we need a solution for Windows 7. I already understood from the
very first reply from Doron that dynamic rebalancing will not be possible.
However, it should not be a problem to reserve certain bus ranges at
startup. Once again, is anybody know where Windows store the information
used for initial buses configuration?

With best regards,

Al

Hi Jake,

Thanks a lot for a clear and finally an engineering reply. I got your point. In my understanding, if Windows re-enumerates buses for cold-insertion scenario and system after such re-enumeration works fine, we can safely assume that if we will re-program subordinate bus at the system startup it will work either.

Short extra question: pci.sys in Vista is more or less the same as in Win7?

Best regards, and thank you again.

Al

No, the driver between the two releases is different

d

debt from my phone


From: xxxxx@hotmail.com
Sent: 3/27/2012 7:40 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] How dynamically re-assign bus numbers

Hi Jake,

Thanks a lot for a clear and finally an engineering reply. I got your point. In my understanding, if Windows re-enumerates buses for cold-insertion scenario and system after such re-enumeration works fine, we can safely assume that if we will re-program subordinate bus at the system startup it will work either.

Short extra question: pci.sys in Vista is more or less the same as in Win7?

Best regards, and thank you again.

Al


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Thank you, Doron.

With best regards,

Al