PCI Express problem

NTDEV
1

Hi all,

I am trying to do the following.

We have an existing PCI bus with some devices attached, the bus also has a
bridge and a secondary bus attached to it. The secondary bus is hot
pluggable with down-stream ports.

We have a PCI card that further has two bridges each bridge has a
subordinate bus. On the subordinate bus there would be two HBAs attaached.

Well, the trouble is the OS doesn’t detect one of the busses!

I am ataching the ouput of lspci command in Linux ( it is quite useful in
this case)

Initial point, when the machine is booted without the card attached…

-[0000:00]-±00.0
±00.1
±01.0
±02.0-[0000:01-03]–±00.0-[0000:02]–±03.0
| | -03.1
| -00.2-[0000:03]–
±04.0-[0000:04]–
±05.0-[0000:05]–
±06.0-[0000:06-0e]----00.0-[0000:07-0e]–±00.0-[0000:08-0a]–±
00.0-[0000:09]–±01.0
| |
| ±01.1
| |
| ±03.0
| |
| -03.1
| | -
00.1-[0000:0a]----01.0
| ±01.0-[0000:0b]–
| ±02.0-[0000:0c]–
| ±03.0-[0000:0d]–
| -04.0-[0000:0e]–
±07.0-[0000:0f]–
±1d.0
±1d.1
±1d.2
±1d.3
±1d.7
±1e.0-[0000:10]----04.0
±1f.0
±1f.1
-1f.3

oooo:ob is the trouble maker.

Well, as you see the problem is that the secondary bus doesnt have any
resources alloacated to it and thus the further split is causing a problem.
One solution could be to reset the entire primary bus and do a reprobing,
but that is not a professional solution and the devices connected to the bus
will do for a reset causing havoc. Also this is a damned hot pluggable
device. So how to solve it.

–

  • Developer
2

I would be very, very interested in what you find out hare.

What is the platform? We have run into a similar problem on a Dell 6850.
As I recall, the basic machine with nothing plugged in (except for maybe
some riser cards) had something like 17 bridges in it.

Now, I’m wondering if you are seeing the same or a related problem. In
our case I believe that the problem could have been I/O space assignment.

I/O space has to be allocated to secondary and subordinate busses in
multiples of 4k. There is only 64k of I/O space available. In the case
we saw, with some other cards plugged in and before ours was plugged in,
the system BIOS had ALL of the I/O space from 0x3000 to 0xFFFF allocated.
The system had devices on the primary bus mapped with I/O addresses in the
0 - 0x2FFF range. If our card, which is a PCIe card with a dual PCIe to
PCI-X bridge on it, was plugged in, all kinds of PnP messages came up when
the system was booted. Our card would have required 4k of I.O space for
each of the secondary busses.

Looking at the allocations compared to what was plugged in to the
secondary busses, the system BIOS had allocated an extra 4k of I/O space
to every bus over what was needed by the actual devices on that bus. We
figured that maybe it was allocating space to things that MIGHT be plugged
in later.

We still have not resolved this issue, and we don’t have one of these
machines here to work with, so I can’t say what the solution to this
problem ever will be. But it would seem that the system BIOS should be
giving some kind of priority to the devices that actually are plugged in.

Jerry.

xxxxx@lists.osr.com wrote on 12/01/2005 04:35:49 AM:

Hi all,

I am trying to do the following.

We have an existing PCI bus with some devices attached, the bus also
has a bridge and a secondary bus attached to it. The secondary bus
is hot pluggable with down-stream ports.

We have a PCI card that further has two bridges each bridge has a
subordinate bus. On the subordinate bus there would be two HBAs
attaached.

Well, the trouble is the OS doesn’t detect one of the busses!

I am ataching the ouput of lspci command in Linux ( it is quite
useful in this case)

Initial point, when the machine is booted without the card attached…

-[0000:00]-±00.0
±00.1
±01.0
±02.0-[0000:01-03]–±00.0-[0000:02]–±03.0
| | -03.1
| -00.2-[0000:03]–
±04.0-[0000:04]–
±05.0-[0000:05]–
±06.0-[0000:06-0e]----00.0-[0000:07-0e]–±00.0-[0000:
08-0a]–±00.0-[0000:09]–±01.0
| |
| ±01.1
| |
| ±03.0
| |
| -03.1
| |
-00.1-[0000:0a]----01.0
| ±01.0-[0000:0b]–
| ±02.0-[0000:0c]–
| ±03.0-[0000:0d]–
| -04.0-[0000:0e]–
±07.0-[0000:0f]–
±1d.0
±1d.1
±1d.2
±1d.3
±1d.7
±1e.0-[0000:10]----04.0
±1f.0
±1f.1
-1f.3

oooo:ob is the trouble maker.

Well, as you see the problem is that the secondary bus doesnt have
any resources alloacated to it and thus the further split is causing
a problem. One solution could be to reset the entire primary bus and
do a reprobing, but that is not a professional solution and the
devices connected to the bus will do for a reset causing havoc. Also
this is a damned hot pluggable device. So how to solve it.

–

3

Oops, the list manager kicked my butt for posting a messege too long, I will
split it into three parts.

PART-1

Hello Jerry,

What is the platform? We have run into a similar problem on a Dell 6850.
As I recall, the basic machine with nothing plugged in (except for maybe
some riser cards) had something like 17 bridges in it.

It is a basic intel x86 here. We have two teams, one for Linux and the other
for Windows.

Now, I’m wondering if you are seeing the same or a related problem. In
our case I believe that the problem could have been I/O space assignment.

You are correct, it is resource allocation. When the machine booted up, the
devices got resources, since the secondary bus was empty it didn’t have any
resources except what it neeed. now when I tried to plug in the card, it
required the new resources. But couldn’t allocate. Now either we reset the
PCI bus ( we cannot just reset the secondary bus, as the resource alloc
needs the parent ). Else find a way to bypass this damned problem, I don’t
know how.

I/O space has to be allocated to secondary and subordinate busses in
multiples of 4k. There is only 64k of I/O space available. In the case
we saw, with some other cards plugged in and before ours was plugged in,
the system BIOS had ALL of the I/O space from 0x3000 to 0xFFFF allocated.
The system had devices on the primary bus mapped with I/O addresses in the
0 - 0x2FFF range.

Here are the outputs before the card was plugged in. - Show quoted text -

-[0000:00]-±00.0
±00.1
±01.0
±02.0-[0000:01-03]–±00.0-[0000:02]–±03.0
| | -03.1
| -00.2-[0000:03]–
±04.0-[0000:04]–
±05.0-[0000:05]–
±06.0-[0000:06-0e]----00.0-[0000:07-0e]–±00.0-[0000:08-0a]–±
00.0-[0000:09]–±01.0
| |
| ±01.1
| |
| ±03.0
| |
| -03.1
| | -
00.1-[0000:0a]----01.0
| ±01.0-[0000:0b]–
| ±02.0-[0000:0c]–
| ±03.0-[0000:0d]–
| -04.0-[0000:0e]–
±07.0-[0000:0f]–
±1d.0
±1d.1
±1d.2
±1d.3
±1d.7
±1e.0-[0000:10]----04.0
±1f.0
±1f.1
-1f.3

4

PART-II

and here is what I get after I put it in…

-[0000:00]-±00.0
±00.1
±01.0
±02.0-[0000:01-03]–±00.0-[0000:02]–±03.0
| | -03.1
| -00.2-[0000:03]–
±04.0-[0000:04]–
±05.0-[0000:05]–
±06.0-[0000:06-0e]----00.0-[0000:07-0e]–±00.0-[0000:08-0a]–±
00.0-[0000:09]–±01.0
| |
| ±01.1
| |
| ±03.0
| |
| -03.1
| | -
00.1-[0000:0a]----01.0
| ±01.0-[0000:0b]–±
00.0-[0000:00]–
| | -
00.2-[0000:00]–
| ±02.0-[0000:0c]–
| ±03.0-[0000:0d]–
| -04.0-[0000:0e]–
±07.0-[0000:0f]–
±1d.0
±1d.1
±1d.2
±1d.3
±1d.7
±1e.0-[0000:10]----04.0
±1f.0
±1f.1
-1f.3

The problem is very much visible, the naming of the devices are not what
should have been!!!

Here is the verbose output…

00:00.0 Host bridge: Intel Corp. E7520 Memory Controller Hub (rev 0c)
Subsystem: Intel Corp. E7520 Memory Controller Hub
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0
Capabilities: [40] Vendor Specific Information

00:00.1 Class ff00: Intel Corp. E7525/E7520 Error Reporting Registers (rev
0c)
Subsystem: Intel Corp.: Unknown device 3590
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-
00:01.0 System peripheral: Intel Corp. E7520 DMA Controller (rev 0c)
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Interrupt: pin A routed to IRQ 255
Region 0: Memory at d0000000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [b0] Message Signalled Interrupts: 64bit- Queue=0/1
Enable-
Address: fee00000 Data: 0000

00:02.0 PCI bridge: Intel Corp. E7525/E7520/E7320 PCI Express Port A (rev
0c) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0, Cache Line Size 08
Bus: primary=00, secondary=01, subordinate=03, sec-latency=0
I/O behind bridge: 00002000-00002fff
Memory behind bridge: d0100000-d01fffff
Prefetchable memory behind bridge: 00000000d1000000-00000000d1000000
Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort- BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Message Signalled Interrupts: 64bit- Queue=0/1
Enable-
Address: fee00000 Data: 0000
Capabilities: [64] Express Root Port (Slot-) IRQ 0
Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 2
Link: Latency L0s <4us, L1 unlimited
Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x8
Root: Correctable- Non-Fatal- Fatal- PME-
Capabilities: [100] Advanced Error Reporting

5

PART III

00:04.0 PCI bridge: Intel Corp. E7525/E7520 PCI Express Port B (rev 0c)
(prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR+ Latency: 0, Cache Line Size 08
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort- BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Message Signalled Interrupts: 64bit- Queue=0/1
Enable-
Address: fee00000 Data: 0000
Capabilities: [64] Express Root Port (Slot-) IRQ 0
Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 4
Link: Latency L0s <4us, L1 unlimited
Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x8
Root: Correctable- Non-Fatal- Fatal- PME-
Capabilities: [100] Advanced Error Reporting

00:05.0 PCI bridge: Intel Corp. E7520 PCI Express Port B1 (rev 0c) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR+ Latency: 0, Cache Line Size 08
Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort- BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Message Signalled Interrupts: 64bit- Queue=0/1
Enable-
Address: fee00000 Data: 0000
Capabilities: [64] Express Root Port (Slot-) IRQ 0
Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s, Port 5
Link: Latency L0s <4us, L1 unlimited
Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x4
Root: Correctable- Non-Fatal- Fatal- PME-
Capabilities: [100] Advanced Error Reporting

00:06.0 PCI bridge: Intel Corp. E7520 PCI Express Port C (rev 0c) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0, Cache Line Size 08
Bus: primary=00, secondary=06, subordinate=0e, sec-latency=0
I/O behind bridge: 00003000-00004fff
Memory behind bridge: d0200000-d03fffff
Prefetchable memory behind bridge: 00000000d1100000-00000000d11000 00
Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort- BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Message Signalled Interrupts: 64bit- Queue=0/1
Enable-
Address: fee00000 Data: 0000
Capabilities: [64] Express Root Port (Slot-) IRQ 0
Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 6
Link: Latency L0s <4us, L1 unlimited
Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x8
Root: Correctable- Non-Fatal- Fatal- PME-
Capabilities: [100] Advanced Error Reporting

00:07.0 PCI bridge: Intel Corp. E7520 PCI Express Port C1 (rev 0c) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR+ Latency: 0, Cache Line Size 08
Bus: primary=00, secondary=0f, subordinate=0f, sec-latency=0
Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort- BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Message Signalled Interrupts: 64bit- Queue=0/1
Enable-
Address: fee00000 Data: 0000
Capabilities: [64] Express Root Port (Slot-) IRQ 0
Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s, Port 7
Link: Latency L0s <4us, L1 unlimited
Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x4
Root: Correctable- Non-Fatal- Fatal- PME-
Capabilities: [100] Advanced Error Reporting

6

PART IV

00:1d.0 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI
Controller #1 (rev 02) (prog-if 00 [UHCI])
Subsystem: Intel Corp.: Unknown device 24d0
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
SERR- Latency: 0
Interrupt: pin A routed to IRQ 19
Region 4: I/O ports at 1400 [size=32]

00:1d.1 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI
Controller #2 (rev 02) (prog-if 00 [UHCI])
Subsystem: Intel Corp.: Unknown device 24d0
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
SERR- Latency: 0
Interrupt: pin B routed to IRQ 18
Region 4: I/O ports at 1420 [size=32]

00:1d.2 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI #3 (rev
02) (prog-if 00 [UHCI])
Subsystem: Intel Corp.: Unknown device 24d0
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
SERR- Latency: 0
Interrupt: pin C routed to IRQ 17
Region 4: I/O ports at 1440 [size=32]

00:1d.3 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB UHCI
Controller #4 (rev 02) (prog-if 00 [UHCI])
Subsystem: Intel Corp.: Unknown device 24d0
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
SERR- Latency: 0
Interrupt: pin A routed to IRQ 19
Region 4: I/O ports at 1460 [size=32]

00:1d.7 USB Controller: Intel Corp. 82801EB/ER (ICH5/ICH5R) USB2 EHCI
Controller (rev 02) (prog-if 20 [EHCI])
Subsystem: Intel Corp.: Unknown device 24d0
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
SERR- Latency: 0
Interrupt: pin D routed to IRQ 20
Region 0: Memory at d0001000 (32-bit, non-prefetchable) [size=1K]
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Debug port

00:1e.0 PCI bridge: Intel Corp. 82801 PCI Bridge (rev c2) (prog-if 00
[Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0
Bus: primary=00, secondary=10, subordinate=10, sec-latency=32
I/O behind bridge: 00005000-00005fff
Memory behind bridge: d0400000-d04fffff
Prefetchable memory behind bridge: d8000000-dfffffff
Secondary status: 66Mhz- FastB2B+ ParErr- DEVSEL=medium >TAbort-
BridgeCtl: Parity- SERR- NoISA+ VGA+ MAbort- >Reset- FastB2B-

00:1f.0 ISA bridge: Intel Corp. 82801EB/ER (ICH5/ICH5R) LPC Interface Bridge
(rev 02)
Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
SERR- Latency: 0

7

PART V

00:1f.1 IDE interface: Intel Corp. 82801EB/ER (ICH5/ICH5R) IDE Controller
(rev 02) (prog-if 8a [Master SecP PriP])
Subsystem: Intel Corp.: Unknown device 24d0
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
SERR- Latency: 0
Interrupt: pin A routed to IRQ 17
Region 0: I/O ports at
Region 1: I/O ports at
Region 2: I/O ports at
Region 3: I/O ports at
Region 4: I/O ports at 14a0 [size=16]
Region 5: Memory at d1200000 (32-bit, non-prefetchable) [size=1K]

00:1f.3 SMBus: Intel Corp. 82801EB/ER (ICH5/ICH5R) SMBus Controller (rev 02)
Subsystem: Intel Corp.: Unknown device 24d0
Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
SERR- Interrupt: pin B routed to IRQ 10
Region 4: I/O ports at 1480 [size=32]

01:00.0 PCI bridge: Intel Corp. 6700PXH PCI Express-to-PCI Bridge A (rev 09)
(prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0, Cache Line Size 08
Bus: primary=01, secondary=02, subordinate=02, sec-latency=96
I/O behind bridge: 00002000-00002fff
Memory behind bridge: d0100000-d01fffff
Prefetchable memory behind bridge: 00000000d1000000-00000000d10000 00
Secondary status: 66Mhz+ FastB2B+ ParErr- DEVSEL=medium >TAbort-
BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [44] Express PCI/PCI-X Bridge IRQ 0
Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 0
Link: Latency L0s unlimited, L1 unlimited
Link: ASPM Disabled CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x8
Capabilities: [5c] Message Signalled Interrupts: 64bit+ Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [6c] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [d8] PCI-X bridge device.
Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=1
Status: Bus=1 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, SCO-, SRD-
: Upstream: Capacity=65535, Commitment Limit=65535
: Downstream: Capacity=65535, Commitment Limit=65535
Capabilities: [100] Advanced Error Reporting
Capabilities: [300] Power Budgeting

01:00.2 PCI bridge: Intel Corp. 6700PXH PCI Express-to-PCI Bridge B (rev 09)
(prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0, Cache Line Size 08
Bus: primary=01, secondary=03, subordinate=03, sec-latency=32
Secondary status: 66Mhz+ FastB2B+ ParErr- DEVSEL=medium >TAbort-
BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [44] Express PCI/PCI-X Bridge IRQ 0
Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 0
Link: Latency L0s unlimited, L1 unlimited
Link: ASPM Disabled CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x8
Capabilities: [5c] Message Signalled Interrupts: 64bit+ Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [6c] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [d8] PCI-X bridge device.
Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=3
Status: Bus=1 Dev=0 Func=2 64bit- 133MHz- SCD- USC-, SCO-, SRD-
: Upstream: Capacity=65535, Commitment Limit=65535
: Downstream: Capacity=65535, Commitment Limit=65535
Capabilities: [100] Advanced Error Reporting
Capabilities: [300] Power Budgeting

02:03.0 SCSI storage controller: Adaptec AIC-7902B U320 (rev 10)
Subsystem: Adaptec AIC-7902B U320
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- SERR- Latency: 72 (10000ns min, 6250ns max), Cache Line Size 08
Interrupt: pin A routed to IRQ 17
Region 0: I/O ports at 2400 [disabled] [size=256]
Region 1: Memory at d0100000 (64-bit, non-prefetchable) [size=8K]
Region 3: I/O ports at 2000 [disabled] [size=256]
[virtual] Expansion ROM at d1000000 [disabled] [size=512K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [a0] Message Signalled Interrupts: 64bit+ Queue=0/1
Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [94] PCI-X non-bridge device.
Command: DPERE- ERO+ RBC=0 OST=4
Status: Bus=2 Dev=3 Func=0 64bit+ 133MHz+ SCD- USC-, DC=simple,
DMMRBC=0, DMOST=4, DMCRS=1, RSCEM-

02:03.1 SCSI storage controller: Adaptec AIC-7902B U320 (rev 10)
Subsystem: Adaptec AIC-7902B U320
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- SERR- Latency: 72 (10000ns min, 6250ns max), Cache Line Size 08
Interrupt: pin B routed to IRQ 18
Region 0: I/O ports at 2c00 [disabled] [size=256]
Region 1: Memory at d0102000 (64-bit, non-prefetchable) [size=8K]
Region 3: I/O ports at 2800 [disabled] [size=256]
[virtual] Expansion ROM at d1080000 [disabled] [size=512K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [a0] Message Signalled Interrupts: 64bit+ Queue=0/1
Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [94] PCI-X non-bridge device.
Command: DPERE- ERO+ RBC=0 OST=4
Status: Bus=2 Dev=3 Func=1 64bit+ 133MHz+ SCD- USC-, DC=simple,
DMMRBC=0, DMOST=4, DMCRS=1, RSCEM-

8

PART VI

06:00.0 PCI bridge: NEC Corporation: Unknown device 0124 (rev 02) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR+ Latency: 0, Cache Line Size 08
Bus: primary=06, secondary=07, subordinate=0e, sec-latency=0
I/O behind bridge: 00003000-00004fff
Memory behind bridge: d0200000-d03fffff
Prefetchable memory behind bridge: 00000000d1100000-00000000d11000 00
Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort- BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: 64bit+ Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [60] Express Upstream Port IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: AtnBtn- AtnInd- PwrInd-
Device: SlotPowerLimit 0.000000
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s L1, Port 0
Link: Latency L0s <1us, L1 <16us
Link: ASPM Disabled CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x8
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel

07:00.0 PCI bridge: NEC Corporation: Unknown device 0124 (rev 02) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0, Cache Line Size 08
Bus: primary=07, secondary=08, subordinate=0a, sec-latency=0
I/O behind bridge: 00003000-00004fff
Memory behind bridge: d0200000-d03fffff
Prefetchable memory behind bridge: 00000000d1100000-00000000d1100000
Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort- BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: 64bit+ Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [60] Express Downstream Port (Slot-) IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s L1, Port 1
Link: Latency L0s <1us, L1 <16us
Link: ASPM Disabled CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x8
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel

07:01.0 PCI bridge: NEC Corporation: Unknown device 0124 (rev 02) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0, Cache Line Size 08
Bus: primary=07, secondary=0b, subordinate=0b, sec-latency=0
Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort- BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: 64bit+ Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [60] Express Downstream Port (Slot+) IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s L1, Port 2
Link: Latency L0s <1us, L1 <16us
Link: ASPM Disabled CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x4
Slot: AtnBtn- PwrCtrl- MRL- AtnInd- PwrInd- HotPlug+ Surpise-
Slot: Number 10, PowerLimit 10.000000
Slot: Enabled AtnBtn- PwrFlt- MRL- PresDet+ CmdCplt- HPIrq+
Slot: AttnInd Off, PwrInd Off, Power+
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel

07:02.0 PCI bridge: NEC Corporation: Unknown device 0124 (rev 02) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0, Cache Line Size 08
Bus: primary=07, secondary=0c, subordinate=0c, sec-latency=0
Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort- BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: 64bit+ Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [60] Express Downstream Port (Slot+) IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s L1, Port 3
Link: Latency L0s <1us, L1 <16us
Link: ASPM Disabled CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x4
Slot: AtnBtn- PwrCtrl- MRL- AtnInd- PwrInd- HotPlug+ Surpise-
Slot: Number 11, PowerLimit 10.000000
Slot: Enabled AtnBtn- PwrFlt- MRL- PresDet+ CmdCplt- HPIrq+
Slot: AttnInd Off, PwrInd Off, Power+
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel

07:03.0 PCI bridge: NEC Corporation: Unknown device 0124 (rev 02) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0, Cache Line Size 08
Bus: primary=07, secondary=0d, subordinate=0d, sec-latency=0
Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort- BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: 64bit+ Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [60] Express Downstream Port (Slot-) IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s L1, Port 4
Link: Latency L0s <1us, L1 <16us
Link: ASPM Disabled CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x4
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel

07:04.0 PCI bridge: NEC Corporation: Unknown device 0124 (rev 02) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0, Cache Line Size 08
Bus: primary=07, secondary=0e, subordinate=0e, sec-latency=0
Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort- BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: 64bit+ Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [60] Express Downstream Port (Slot-) IRQ 0
Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
Link: Supported Speed 2.5Gb/s, Width x4, ASPM L0s L1, Port 5
Link: Latency L0s <1us, L1 <16us
Link: ASPM Disabled CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x4
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel

9

PART VII

08:00.0 PCI bridge: NEC Corporation: Unknown device 0125 (rev 02) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR+ Latency: 0, Cache Line Size 08
Bus: primary=08, secondary=09, subordinate=09, sec-latency=201
I/O behind bridge: 00003000-00003fff
Memory behind bridge: d0200000-d02fffff
Secondary status: 66Mhz+ FastB2B- ParErr- DEVSEL=slow >TAbort- BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [40] Express PCI/PCI-X Bridge IRQ 0
Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s L1, Port 0
Link: Latency L0s <512ns, L1 <16us
Link: ASPM Disabled CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x8
Capabilities: [54] PCI-X bridge device.
Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=2
Status: Bus=8 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, SCO-, SRD-
: Upstream: Capacity=32, Commitment Limit=32
: Downstream: Capacity=16, Commitment Limit=16
Capabilities: [64] Power Management version 2
Flags: PMEClk- DSI- D1+ D2- AuxCurrent=55mA
PME(D0+,D1+,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Bridge: PM- B3+
Capabilities: [100] Advanced Error Reporting

08:00.1 PCI bridge: NEC Corporation: Unknown device 0125 (rev 02) (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0, Cache Line Size 08
Bus: primary=08, secondary=0a, subordinate=0a, sec-latency=36
I/O behind bridge: 00004000-00004fff
Memory behind bridge: d0300000-d03fffff
Prefetchable memory behind bridge: 00000000d1100000-00000000d11000 00
Secondary status: 66Mhz+ FastB2B- ParErr- DEVSEL=slow >TAbort- BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [40] Express PCI/PCI-X Bridge IRQ 0
Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s L1, Port 0
Link: Latency L0s <512ns, L1 <16us
Link: ASPM Disabled CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x8
Capabilities: [54] PCI-X bridge device.
Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=0
Status: Bus=8 Dev=0 Func=1 64bit- 133MHz- SCD- USC-, SCO-, SRD-
: Upstream: Capacity=32, Commitment Limit=32
: Downstream: Capacity=16, Commitment Limit=16
Capabilities: [64] Power Management version 2
Flags: PMEClk- DSI- D1+ D2- AuxCurrent=55mA
PME(D0+,D1+,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Bridge: PM- B3+
Capabilities: [100] Advanced Error Reporting

09:01.0 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller
(rev 03)
Subsystem: Intel Corp. PRO/1000 MT Dual Port Network Connection
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
SERR- Latency: 52 (63750ns min), Cache Line Size 08
Interrupt: pin A routed to IRQ 16
Region 0: Memory at d0200000 (64-bit, non-prefetchable) [size=128K]
Region 4: I/O ports at 3000 [size=64]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [e4] PCI-X non-bridge device.
Command: DPERE- ERO+ RBC=0 OST=0
Status: Bus=9 Dev=1 Func=0 64bit+ 133MHz+ SCD- USC-, DC=simple,
DMMRBC=2, DMOST=0, DMCRS=1, RSCEM-
Capabilities: [f0] Message Signalled Interrupts: 64bit+ Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000

09:01.1 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller
(rev 03)
Subsystem: Intel Corp. PRO/1000 MT Dual Port Network Connection
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
SERR- Latency: 52 (63750ns min), Cache Line Size 08
Interrupt: pin B routed to IRQ 17
Region 0: Memory at d0220000 (64-bit, non-prefetchable) [size=128K]
Region 4: I/O ports at 3040 [size=64]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [e4] PCI-X non-bridge device.
Command: DPERE- ERO+ RBC=0 OST=0
Status: Bus=9 Dev=1 Func=1 64bit+ 133MHz+ SCD- USC-, DC=simple,
DMMRBC=2, DMOST=0, DMCRS=1, RSCEM-
Capabilities: [f0] Message Signalled Interrupts: 64bit+ Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000

09:03.0 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller
(rev 03)
Subsystem: Intel Corp. PRO/1000 MT Dual Port Network Connection
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
SERR- Latency: 52 (63750ns min), Cache Line Size 08
Interrupt: pin A routed to IRQ 18
Region 0: Memory at d0240000 (64-bit, non-prefetchable) [size=128K]
Region 4: I/O ports at 3080 [size=64]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [e4] PCI-X non-bridge device.
Command: DPERE- ERO+ RBC=0 OST=0
Status: Bus=9 Dev=3 Func=0 64bit+ 133MHz+ SCD- USC-, DC=simple,
DMMRBC=2, DMOST=0, DMCRS=1, RSCEM-
Capabilities: [f0] Message Signalled Interrupts: 64bit+ Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000

09:03.1 Ethernet controller: Intel Corp. 82546GB Gigabit Ethernet Controller
(rev 03)
Subsystem: Intel Corp. PRO/1000 MT Dual Port Network Connection
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
SERR- Latency: 52 (63750ns min), Cache Line Size 08
Interrupt: pin B routed to IRQ 19
Region 0: Memory at d0260000 (64-bit, non-prefetchable) [size=128K]
Region 4: I/O ports at 30c0 [size=64]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [e4] PCI-X non-bridge device.
Command: DPERE- ERO+ RBC=0 OST=0
Status: Bus=9 Dev=3 Func=1 64bit+ 133MHz+ SCD- USC-, DC=simple,
DMMRBC=2, DMOST=0, DMCRS=1, RSCEM-
Capabilities: [f0] Message Signalled Interrupts: 64bit+ Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000

0a:01.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev
10)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
SERR- Latency: 66 (2000ns min, 14000ns max), Cache Line Size 08
Interrupt: pin A routed to IRQ 16
Region 0: Memory at d0320000 (32-bit, non-prefetchable) [size=4K]
Region 1: I/O ports at 4000 [size=64]
Region 2: Memory at d0300000 (32-bit, non-prefetchable) [size=128K]
[virtual] Expansion ROM at d1100000 [disabled] [size=64K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA
PME(D0+,D1+,D2+,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=2 PME-

0b:00.0 PCI bridge: Intel Corp. 41210 [Lanai] Serial to Parallel PCI Bridge
(rev 09) (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Bus: primary=00, secondary=00, subordinate=00, sec-latency=64
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 0000000000000000-0000000000000000
Secondary status: 66Mhz+ FastB2B+ ParErr- DEVSEL=medium >TAbort-
BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
Capabilities: [44] Express PCI/PCI-X Bridge IRQ 0
Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal+ Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 0
Link: Latency L0s unlimited, L1 unlimited
Link: ASPM Disabled CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x8
Capabilities: [5c] Message Signalled Interrupts: 64bit+ Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [6c] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [d8] PCI-X bridge device.
Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=3
Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-, SCO-, SRD-
: Upstream: Capacity=65535, Commitment Limit=65535
: Downstream: Capacity=65535, Commitment Limit=65535
Capabilities: [100] Advanced Error Reporting
Capabilities: [300] Power Budgeting

0b:00.2 PCI bridge: Intel Corp. 41210 [Lanai] Serial to Parallel PCI Bridge
(rev 09) (prog-if 00 [Normal decode])
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Bus: primary=00, secondary=00, subordinate=00, sec-latency=0
I/O behind bridge: 00000000-00000fff
Memory behind bridge: 00000000-000fffff
Prefetchable memory behind bridge: 0000000000000000-0000000000000000
Secondary status: 66Mhz+ FastB2B+ ParErr- DEVSEL=medium >TAbort-
BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
Capabilities: [44] Express PCI/PCI-X Bridge IRQ 0
Device: Supported: MaxPayload 256 bytes, PhantFunc 0, ExtTag-
Device: Latency L0s <64ns, L1 <1us
Device: AtnBtn- AtnInd- PwrInd-
Device: Errors: Correctable- Non-Fatal- Fatal+ Unsupported-
Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 0
Link: Latency L0s unlimited, L1 unlimited
Link: ASPM Disabled CommClk- ExtSynch-
Link: Speed 2.5Gb/s, Width x8
Capabilities: [5c] Message Signalled Interrupts: 64bit+ Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [6c] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [d8] PCI-X bridge device.
Secondary Status: 64bit+, 133MHz+, SCD-, USC-, SCO-, SRD- Freq=0
Status: Bus=0 Dev=0 Func=2 64bit- 133MHz- SCD- USC-, SCO-, SRD-
: Upstream: Capacity=65535, Commitment Limit=65535
: Downstream: Capacity=65535, Commitment Limit=65535
Capabilities: [100] Advanced Error Reporting
Capabilities: [300] Power Budgeting

10:04.0 VGA compatible controller: ATI Technologies Inc Radeon RV100 QY
[Radeon 7000/VE] (prog-if 00 [VGA])
Subsystem: ATI Technologies Inc Radeon RV100 QY [Radeon 7000/VE]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping+ SERR+ FastB2B+
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
SERR- Latency: 66 (2000ns min), Cache Line Size 08
Interrupt: pin A routed to IRQ 10
Region 0: Memory at d8000000 (32-bit, prefetchable) [size=128M]
Region 1: I/O ports at 5000 [size=256]
Region 2: Memory at d0400000 (32-bit, non-prefetchable) [size=64K]
[virtual] Expansion ROM at d0420000 [disabled] [size=128K]
Capabilities: [50] Power Management version 2
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-

> If our card, which is a PCIe card with a dual PCIe to
>PCI-X bridge on it, was plugged in, all kinds of PnP messages came up when
>the system was booted. Our card would have required 4k of I.O space for
>each of the secondary busses.

Our card is a simple one, it has a bridge with two busses on it, each bus
has two HBAs on it. I have seen such hardware before, but maybe where ours
differ is the hot plugging part.

>Looking at the allocations compared to what was plugged in to the
>secondary busses, the system BIOS had allocated an extra 4k of I/O space
>to every bus over what was needed by the actual devices on that bus. We
>figured that maybe it was allocating space to things that MIGHT be plugged
>in later.

One solution we were thinking of was to allocate some extra free allocatable
resource to each of the busses in advance, but it would be a shameful waste
of things considering we just have a few KBs!

>We still have not resolved this issue, and we don’t have one of these
>machines here to work with, so I can’t say what the solution to this
>problem ever will be. But it would seem that the system BIOS should be
>giving some kind of priority to the devices that actually are plugged in.

>Jerry.

Let me know what you infer from my outputs which I hav attached.

10

Jerry,
Let me clearly state what the problem is, in simple words…

We don’t think that is the problem in our case. What we are trying to do is
hot plug a card with a pci bridge on it ( SCSI HBAs present behind this PCI
bridge ). The bridge (corrsponding to the PCIe slot) at which we r trying to
hot plug the card doesn’t seem to have enough “bus” resources.
This can be seen from lspci output.
01.0-[0000:0b] bridge has only 1 bus behind it i.e. 0b and we require to put
a bridge(present on the card) on this bus and then the HBAs. Now the bus
corresponding to the HBAs need to have a bus number which will come from
01.0-[0000:0b].
For e.g.
B1---->B2-----C1

where B1 is the bridge corresponding to the hot-pluggable slot, while B2 is
the bridge on the add-in card and C1 is the HBA behind the bridge B2 on the
add-in card.
Now B1 is having only 1 bus resource numbered “0b”. Now our problem is how
the bus between B2 and C1 will get a bus resource (bus number) when
hot-plugged.

11

Ok, I have reviewed the voluminous output you provided and here’s what I
see. Using the verbose output, I assume your card is on bus 0b and the
two bridges ore labeled (0b:00.0) and (0b:00:2).

On the way to your bridges we go through:

(00:06.0) bridge from bus 0 to busses 06 rhtough 0e, I/O range allocated
3000-4FFF.
(06:00.0) bridge from bus 6 to busses 07 through 0e, I/O range
allocated 3000-4FFF
(07:01.0) bridge from bus 7 to bus 0b, apparently no I/O space
configured to pass through!

Why is there no I/O configured for (07:01.0)? Well, the system BIOS
didn’t configure any apparently because there was no need to, it thought.
Of the 3000-4FFF allocated to busses 07 through 0e, it has allocated
3000-3FFF to bus 9 via the bridge at (08:00.0) and 4000-4FFF to bus 0a via
the bridge at (08:00.1). There are actually devices on these two busses
using some of their ranges.

Note that the range 5000-5FFF has been allocated to bus 10 through the
bridge at (00:1e.0).

Next, from the output, neither of the bridges on bus 0b have been
configured at all - they appear to be in their power-on states with all
the registers still zero. So it appears that no attempt has been made to
configure them yet.

When you hot-plug the device after boot, some reconfiguration is going to
have to be done. I don’t know what OS you are trying to run. You did say
that the output was from a Linux utility. I don’t know much about Linux’s
capabilities. However, the first thing the OS would need to do to
reconfigure the bridges is to add two subordinate busses to the bridges at
(00:06.0), (06:00.0) and (07:01.0). To do this it is going to have to
bump all bus numbers higher than 0b by two, adjusting the bridge secondary
and subordinate bus numbers in the affected bridges. This in and of
itself is not too big of a problem. At least on Windows, devices are not
supposed to care what bus they are on because somewhere it has been stated
that these things can change due to hot plugging.

Here is where the real problem comes in, though. If your devices that are
on the secondary busses on your card need I/O or memory space, and the
bridges have not been configured to allow extra for this, then some pretty
serious adjustments are going to need to be made. Let’s say that your
devices on your secondary bus need just 4k of I/O space (the minimum that
can be allocated). Then the I/O allocations on the same bridges on the
way to your card will need to increase by 8k since the 8k passed through
by the first two bridges is already in use. So the ranges on these
bridges need to be changed to 3000-6FFF. Oops - the range from 5000-5FFF
is already in use by the VGA board on bus 10. The only way to accomplish
this is to shut down any device using I/O in the range 5000 and above,
reconfigure, then restart those affected devices. THIS IS A MESS! And I
don’t even know if Windows can do all this. Perhaps someone from
Microsoft or one of the other smart folks on the list can comment on this.

In your first email, I think you said the OS doesn’t detect ONE of the
busses. From what I can see, I don’t see how it could set up either of
them. IF the OS did configure one of the busses, what OS was it? The
only way I can think of might be that the OS reconfigured all the I/O and
possibly memory assignments when it started up, leaving some extra on each
bus for possible hot plugging.

In our case, the system BIOS did, in fact, configure extra I/O, memory and
bus number ranges. However, it just plain always did this and as a
result, our card could not be configured correctly at all.

I didn’t study your output from the viewpoint of memory space allocation;
perhaps the same issue exists there as will.

Jerry.

Developer
Sent by: xxxxx@lists.osr.com
12/01/2005 10:03 AM
Please respond to
“Windows System Software Devs Interest List”

To
“Windows System Software Devs Interest List”
cc

Subject
Re:[ntdev] PCI Express problem

Jerry,
Let me clearly state what the problem is, in simple words…

We don’t think that is the problem in our case. What we are trying to do
is hot plug a card with a pci bridge on it ( SCSI HBAs present behind this
PCI bridge ). The bridge (corrsponding to the PCIe slot) at which we r
trying to hot plug the card doesn’t seem to have enough “bus” resources.
This can be seen from lspci output.
01.0-[0000:0b] bridge has only 1 bus behind it i.e. 0b and we require to
put a bridge(present on the card) on this bus and then the HBAs. Now the
bus corresponding to the HBAs need to have a bus number which will come
from 01.0-[0000:0b].
For e.g.
B1---->B2-----C1

where B1 is the bridge corresponding to the hot-pluggable slot, while B2
is the bridge on the add-in card and C1 is the HBA behind the bridge B2 on
the add-in card.
Now B1 is having only 1 bus resource numbered “0b”. Now our problem is how
the bus between B2 and C1 will get a bus resource (bus number) when
hot-plugged.
— Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256 You are currently subscribed
to ntdev as: unknown lmsubst tag argument: ‘’ To unsubscribe send a blank
email to xxxxx@lists.osr.com

12

Don’t know how linux would handle this.

I used to solve (or workaround) the resource problem
in win2k by writing a PCI bus filter driver that
reserves extra resources for each bridge for cPCI
hotplug platforms. That was done before win2k RTM
(build 2195). It’s *HUGE AMOUNT* of work to get it
sort of work on some selected platforms/chipsets/bios
combination. I left the company shortly after code
complete so I don’t know how well the driver goes in
production environment or in later OS builds.

Did you try this on Windows Vista? The PCI multiple
level resource rebalancing feature is supposed to
solve this problem.

Calvin Guan (Windows DDK MVP)
NetXtreme Longhorn Miniport Prime
Broadcom Corp. www.broadcom.com

— xxxxx@attotech.com wrote:

Ok, I have reviewed the voluminous output you
provided and here’s what I
see. Using the verbose output, I assume your card
is on bus 0b and the
two bridges ore labeled (0b:00.0) and (0b:00:2).

On the way to your bridges we go through:

(00:06.0) bridge from bus 0 to busses 06 rhtough 0e,
I/O range allocated
3000-4FFF.
(06:00.0) bridge from bus 6 to busses 07 through
0e, I/O range
allocated 3000-4FFF
(07:01.0) bridge from bus 7 to bus 0b,
apparently no I/O space
configured to pass through!

Why is there no I/O configured for (07:01.0)? Well,
the system BIOS
didn’t configure any apparently because there was no
need to, it thought.
Of the 3000-4FFF allocated to busses 07 through 0e,
it has allocated
3000-3FFF to bus 9 via the bridge at (08:00.0) and
4000-4FFF to bus 0a via
the bridge at (08:00.1). There are actually devices
on these two busses
using some of their ranges.

Note that the range 5000-5FFF has been allocated to
bus 10 through the
bridge at (00:1e.0).

Next, from the output, neither of the bridges on bus
0b have been
configured at all - they appear to be in their
power-on states with all
the registers still zero. So it appears that no
attempt has been made to
configure them yet.

When you hot-plug the device after boot, some
reconfiguration is going to
have to be done. I don’t know what OS you are
trying to run. You did say
that the output was from a Linux utility. I don’t
know much about Linux’s
capabilities. However, the first thing the OS would
need to do to
reconfigure the bridges is to add two subordinate
busses to the bridges at
(00:06.0), (06:00.0) and (07:01.0). To do this it
is going to have to
bump all bus numbers higher than 0b by two,
adjusting the bridge secondary
and subordinate bus numbers in the affected bridges.
This in and of
itself is not too big of a problem. At least on
Windows, devices are not
supposed to care what bus they are on because
somewhere it has been stated
that these things can change due to hot plugging.

Here is where the real problem comes in, though. If
your devices that are
on the secondary busses on your card need I/O or
memory space, and the
bridges have not been configured to allow extra for
this, then some pretty
serious adjustments are going to need to be made.
Let’s say that your
devices on your secondary bus need just 4k of I/O
space (the minimum that
can be allocated). Then the I/O allocations on the
same bridges on the
way to your card will need to increase by 8k since
the 8k passed through
by the first two bridges is already in use. So the
ranges on these
bridges need to be changed to 3000-6FFF. Oops - the
range from 5000-5FFF
is already in use by the VGA board on bus 10. The
only way to accomplish
this is to shut down any device using I/O in the
range 5000 and above,
reconfigure, then restart those affected devices.
THIS IS A MESS! And I
don’t even know if Windows can do all this. Perhaps
someone from
Microsoft or one of the other smart folks on the
list can comment on this.

In your first email, I think you said the OS doesn’t
detect ONE of the
busses. From what I can see, I don’t see how it
could set up either of
them. IF the OS did configure one of the busses,
what OS was it? The
only way I can think of might be that the OS
reconfigured all the I/O and
possibly memory assignments when it started up,
leaving some extra on each
bus for possible hot plugging.

In our case, the system BIOS did, in fact, configure
extra I/O, memory and
bus number ranges. However, it just plain always
did this and as a
result, our card could not be configured correctly
at all.

I didn’t study your output from the viewpoint of
memory space allocation;
perhaps the same issue exists there as will.

Jerry.

Developer
> Sent by: xxxxx@lists.osr.com
> 12/01/2005 10:03 AM
> Please respond to
> “Windows System Software Devs Interest List”
>
>
>
> To
> “Windows System Software Devs Interest List”
>
> cc
>
> Subject
> Re:[ntdev] PCI Express problem
>
>
>
>
>
>
> Jerry,
> Let me clearly state what the problem is, in simple
> words…
>
> We don’t think that is the problem in our case. What
> we are trying to do
> is hot plug a card with a pci bridge on it ( SCSI
> HBAs present behind this
> PCI bridge ). The bridge (corrsponding to the PCIe
> slot) at which we r
> trying to hot plug the card doesn’t seem to have
> enough “bus” resources.
> This can be seen from lspci output.
> 01.0-[0000:0b] bridge has only 1 bus behind it i.e.
> 0b and we require to
> put a bridge(present on the card) on this bus and
> then the HBAs. Now the
> bus corresponding to the HBAs need to have a bus
> number which will come
> from 01.0-[0000:0b].
> For e.g.
> B1---->B2-----C1
>
> where B1 is the bridge corresponding to the
> hot-pluggable slot, while B2
> is the bridge on the add-in card and C1 is the HBA
> behind the bridge B2 on
> the add-in card.
> Now B1 is having only 1 bus resource numbered “0b”.
> Now our problem is how
> the bus between B2 and C1 will get a bus resource
> (bus number) when
> hot-plugged.
> — Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256 You are
> currently subscribed
> to ntdev as: unknown lmsubst tag argument: ‘’ To
> unsubscribe send a blank
> email to xxxxx@lists.osr.com
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as:
> xxxxx@yahoo.ca
> To unsubscribe send a blank email to
> xxxxx@lists.osr.com
>

__________________________________________________________
Find your next car at http://autos.yahoo.ca

13

> the system was booted. Our card would have required 4k of I.O space for

each of the secondary busses.

IO space is obsolete. Can you move to memory space?

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

14

Hi, Maxim.

Sure I/O space is “obsolete.” However, what pray tell are we supposed to
do when the chip we are using has a BAR for I/O space?

And as far as I know, there is no way to use memory mapped registers from
a good-ole “real mode” INT 13 BIOS driver, so we’re stuck with I/O until
INT 13 BIOSs finally die.

In all our OS drivers we use memory space, of course.

Jerry.

“Maxim S. Shatskih”
Sent by: xxxxx@lists.osr.com
12/01/2005 03:08 PM
Please respond to
“Windows System Software Devs Interest List”

To
“Windows System Software Devs Interest List”
cc

Subject
Re: [ntdev] PCI Express problem

> the system was booted. Our card would have required 4k of I.O space for

> each of the secondary busses.

IO space is obsolete. Can you move to memory space?

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@attotech.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

15

By the way, the same problem with I/O space exists in this system with
memory space:

(00:06.0) forwards d0200000-d03ffff to busses 6 through e
(06:00.0) forwareds the same range to busses 7 through e
(07:01.0) forwards no memory space.

Of the space forwarded to busses 7 through 0e, (08:00.0) has forwarded
d0200000-d02fffff to bus 9 and (08:00.1) has forwarded d0300000-d03fffff
to bus 0a. Of course there’s nothing left for (07:01.0) to forward to bus
0b, so the memory forwarded by (00:06.0) and (06:00.0) needs to be
increased. (Memory space is forwarded by bridges in multiples of 1 meg,
or 0x00100000.)

To reconfigure, the space allocated to bus 10 by (00:1e.0) is currently
d0400000-d04fffff; and it would have to move, and this space is currently
used by the VGA controller on bus 10. So the VGA driver would have to be
stopped, allocations reconfigured, and the VGA driver restarted. (Of
course the problem could be much worse; in the general case, many devices
may need to be reconfigured.)

So the same problem exists with memory space as I/O space.

A hot-plug aware BIOS should allocate extra resources for each of the
busses. Those resources must include memory as well as I/O space; bus
numbers can be adjusted at run time if it is understood that device
drivers are not supposed to care what bus they are on. But how much extra
space to allocate? Who knows what will be plugged in and what the
requirements will be?

Windows could reallocate all the memory and I/O assignments when it starts
up. But even so, we would still be left with the basic question - how
much extra space to allocate would be just a guess.

Jerry.

xxxxx@attotech.com
Sent by: xxxxx@lists.osr.com
12/01/2005 02:21 PM
Please respond to
“Windows System Software Devs Interest List”

To
“Windows System Software Devs Interest List”
cc

Subject
Re:[ntdev] PCI Express problem

Ok, I have reviewed the voluminous output you provided and here’s what I
see. Using the verbose output, I assume your card is on bus 0b and the
two bridges ore labeled (0b:00.0) and (0b:00:2).

On the way to your bridges we go through:

(00:06.0) bridge from bus 0 to busses 06 rhtough 0e, I/O range allocated
3000-4FFF.
(06:00.0) bridge from bus 6 to busses 07 through 0e, I/O range
allocated 3000-4FFF
(07:01.0) bridge from bus 7 to bus 0b, apparently no I/O space
configured to pass through!

Why is there no I/O configured for (07:01.0)? Well, the system BIOS
didn’t configure any apparently because there was no need to, it thought.
Of the 3000-4FFF allocated to busses 07 through 0e, it has allocated
3000-3FFF to bus 9 via the bridge at (08:00.0) and 4000-4FFF to bus 0a via

the bridge at (08:00.1). There are actually devices on these two busses
using some of their ranges.

Note that the range 5000-5FFF has been allocated to bus 10 through the
bridge at (00:1e.0).

Next, from the output, neither of the bridges on bus 0b have been
configured at all - they appear to be in their power-on states with all
the registers still zero. So it appears that no attempt has been made to
configure them yet.

When you hot-plug the device after boot, some reconfiguration is going to
have to be done. I don’t know what OS you are trying to run. You did say

that the output was from a Linux utility. I don’t know much about Linux’s

capabilities. However, the first thing the OS would need to do to
reconfigure the bridges is to add two subordinate busses to the bridges at

(00:06.0), (06:00.0) and (07:01.0). To do this it is going to have to
bump all bus numbers higher than 0b by two, adjusting the bridge secondary

and subordinate bus numbers in the affected bridges. This in and of
itself is not too big of a problem. At least on Windows, devices are not
supposed to care what bus they are on because somewhere it has been stated

that these things can change due to hot plugging.

Here is where the real problem comes in, though. If your devices that are

on the secondary busses on your card need I/O or memory space, and the
bridges have not been configured to allow extra for this, then some pretty

serious adjustments are going to need to be made. Let’s say that your
devices on your secondary bus need just 4k of I/O space (the minimum that
can be allocated). Then the I/O allocations on the same bridges on the
way to your card will need to increase by 8k since the 8k passed through
by the first two bridges is already in use. So the ranges on these
bridges need to be changed to 3000-6FFF. Oops - the range from 5000-5FFF
is already in use by the VGA board on bus 10. The only way to accomplish
this is to shut down any device using I/O in the range 5000 and above,
reconfigure, then restart those affected devices. THIS IS A MESS! And I
don’t even know if Windows can do all this. Perhaps someone from
Microsoft or one of the other smart folks on the list can comment on this.

In your first email, I think you said the OS doesn’t detect ONE of the
busses. From what I can see, I don’t see how it could set up either of
them. IF the OS did configure one of the busses, what OS was it? The
only way I can think of might be that the OS reconfigured all the I/O and
possibly memory assignments when it started up, leaving some extra on each

bus for possible hot plugging.

In our case, the system BIOS did, in fact, configure extra I/O, memory and

bus number ranges. However, it just plain always did this and as a
result, our card could not be configured correctly at all.

I didn’t study your output from the viewpoint of memory space allocation;
perhaps the same issue exists there as will.

Jerry.

Developer
Sent by: xxxxx@lists.osr.com
12/01/2005 10:03 AM
Please respond to
“Windows System Software Devs Interest List”

To
“Windows System Software Devs Interest List”
cc

Subject
Re:[ntdev] PCI Express problem

Jerry,
Let me clearly state what the problem is, in simple words…

We don’t think that is the problem in our case. What we are trying to do
is hot plug a card with a pci bridge on it ( SCSI HBAs present behind this

PCI bridge ). The bridge (corrsponding to the PCIe slot) at which we r
trying to hot plug the card doesn’t seem to have enough “bus” resources.
This can be seen from lspci output.
01.0-[0000:0b] bridge has only 1 bus behind it i.e. 0b and we require to
put a bridge(present on the card) on this bus and then the HBAs. Now the
bus corresponding to the HBAs need to have a bus number which will come
from 01.0-[0000:0b].
For e.g.
B1---->B2-----C1

where B1 is the bridge corresponding to the hot-pluggable slot, while B2
is the bridge on the add-in card and C1 is the HBA behind the bridge B2 on

the add-in card.
Now B1 is having only 1 bus resource numbered “0b”. Now our problem is how

the bus between B2 and C1 will get a bus resource (bus number) when
hot-plugged.
— Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256 You are currently subscribed
to ntdev as: unknown lmsubst tag argument: ‘’ To unsubscribe send a blank
email to xxxxx@lists.osr.com

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@attotech.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

16

This is where dynamic resource rebalancing comes in, and I believe that it
is not available until the vista timeframe. Of course it will also break 90%
of device drivers out there that will not handle getting asked to release
their resources correctly, but that is a different story.

=====================
Mark Roddy DDK MVP
Windows 2003/XP/2000 Consulting
Hollis Technology Solutions 603-321-1032
www.hollistech.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@attotech.com
Sent: Friday, December 02, 2005 10:03 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] PCI Express problem

By the way, the same problem with I/O space exists in this
system with memory space:

(00:06.0) forwards d0200000-d03ffff to busses 6 through e
(06:00.0) forwareds the same range to busses 7 through e
(07:01.0) forwards no memory space.

Of the space forwarded to busses 7 through 0e, (08:00.0) has
forwarded d0200000-d02fffff to bus 9 and (08:00.1) has
forwarded d0300000-d03fffff to bus 0a. Of course there’s
nothing left for (07:01.0) to forward to bus 0b, so the
memory forwarded by (00:06.0) and (06:00.0) needs to be
increased. (Memory space is forwarded by bridges in
multiples of 1 meg, or 0x00100000.)

To reconfigure, the space allocated to bus 10 by (00:1e.0) is
currently d0400000-d04fffff; and it would have to move, and
this space is currently used by the VGA controller on bus 10.
So the VGA driver would have to be stopped, allocations
reconfigured, and the VGA driver restarted. (Of course the
problem could be much worse; in the general case, many
devices may need to be reconfigured.)

So the same problem exists with memory space as I/O space.

A hot-plug aware BIOS should allocate extra resources for
each of the busses. Those resources must include memory as
well as I/O space; bus numbers can be adjusted at run time if
it is understood that device drivers are not supposed to care
what bus they are on. But how much extra space to allocate?
Who knows what will be plugged in and what the requirements will be?

Windows could reallocate all the memory and I/O assignments
when it starts up. But even so, we would still be left with
the basic question - how much extra space to allocate would
be just a guess.

Jerry.

xxxxx@attotech.com
Sent by: xxxxx@lists.osr.com
12/01/2005 02:21 PM
Please respond to
“Windows System Software Devs Interest List”
>
>
> To
> “Windows System Software Devs Interest List”
> cc
>
> Subject
> Re:[ntdev] PCI Express problem
>
>
>
>
>
>
> Ok, I have reviewed the voluminous output you provided and
> here’s what I
> see. Using the verbose output, I assume your card is on bus
> 0b and the
> two bridges ore labeled (0b:00.0) and (0b:00:2).
>
> On the way to your bridges we go through:
>
> (00:06.0) bridge from bus 0 to busses 06 rhtough 0e, I/O
> range allocated
> 3000-4FFF.
> (06:00.0) bridge from bus 6 to busses 07 through 0e, I/O range
> allocated 3000-4FFF
> (07:01.0) bridge from bus 7 to bus 0b, apparently no I/O space
> configured to pass through!
>
> Why is there no I/O configured for (07:01.0)? Well, the system BIOS
> didn’t configure any apparently because there was no need to,
> it thought.
> Of the 3000-4FFF allocated to busses 07 through 0e, it has allocated
> 3000-3FFF to bus 9 via the bridge at (08:00.0) and 4000-4FFF
> to bus 0a via
>
> the bridge at (08:00.1). There are actually devices on these
> two busses
> using some of their ranges.
>
> Note that the range 5000-5FFF has been allocated to bus 10
> through the
> bridge at (00:1e.0).
>
> Next, from the output, neither of the bridges on bus 0b have been
> configured at all - they appear to be in their power-on
> states with all
> the registers still zero. So it appears that no attempt has
> been made to
> configure them yet.
>
> When you hot-plug the device after boot, some reconfiguration
> is going to
> have to be done. I don’t know what OS you are trying to run.
> You did say
>
> that the output was from a Linux utility. I don’t know much
> about Linux’s
>
> capabilities. However, the first thing the OS would need to do to
> reconfigure the bridges is to add two subordinate busses to
> the bridges at
>
> (00:06.0), (06:00.0) and (07:01.0). To do this it is going
> to have to
> bump all bus numbers higher than 0b by two, adjusting the
> bridge secondary
>
> and subordinate bus numbers in the affected bridges. This in and of
> itself is not too big of a problem. At least on Windows,
> devices are not
> supposed to care what bus they are on because somewhere it
> has been stated
>
> that these things can change due to hot plugging.
>
> Here is where the real problem comes in, though. If your
> devices that are
>
> on the secondary busses on your card need I/O or memory
> space, and the
> bridges have not been configured to allow extra for this,
> then some pretty
>
> serious adjustments are going to need to be made. Let’s say
> that your
> devices on your secondary bus need just 4k of I/O space (the
> minimum that
> can be allocated). Then the I/O allocations on the same
> bridges on the
> way to your card will need to increase by 8k since the 8k
> passed through
> by the first two bridges is already in use. So the ranges on these
> bridges need to be changed to 3000-6FFF. Oops - the range
> from 5000-5FFF
> is already in use by the VGA board on bus 10. The only way
> to accomplish
> this is to shut down any device using I/O in the range 5000
> and above,
> reconfigure, then restart those affected devices. THIS IS A
> MESS! And I
> don’t even know if Windows can do all this. Perhaps someone from
> Microsoft or one of the other smart folks on the list can
> comment on this.
>
> In your first email, I think you said the OS doesn’t detect
> ONE of the
> busses. From what I can see, I don’t see how it could set up
> either of
> them. IF the OS did configure one of the busses, what OS was
> it? The
> only way I can think of might be that the OS reconfigured all
> the I/O and
> possibly memory assignments when it started up, leaving some
> extra on each
>
> bus for possible hot plugging.
>
> In our case, the system BIOS did, in fact, configure extra
> I/O, memory and
>
> bus number ranges. However, it just plain always did this and as a
> result, our card could not be configured correctly at all.
>
> I didn’t study your output from the viewpoint of memory space
> allocation;
> perhaps the same issue exists there as will.
>
> Jerry.
>
>
>
>
>
> Developer
> Sent by: xxxxx@lists.osr.com
> 12/01/2005 10:03 AM
> Please respond to
> “Windows System Software Devs Interest List”
>
>
> To
> “Windows System Software Devs Interest List”
> cc
>
> Subject
> Re:[ntdev] PCI Express problem
>
>
>
>
>
>
> Jerry,
> Let me clearly state what the problem is, in simple words…
>
> We don’t think that is the problem in our case. What we are
> trying to do
> is hot plug a card with a pci bridge on it ( SCSI HBAs
> present behind this
>
> PCI bridge ). The bridge (corrsponding to the PCIe slot) at
> which we r
> trying to hot plug the card doesn’t seem to have enough “bus”
> resources.
> This can be seen from lspci output.
> 01.0-[0000:0b] bridge has only 1 bus behind it i.e. 0b and we
> require to
> put a bridge(present on the card) on this bus and then the
> HBAs. Now the
> bus corresponding to the HBAs need to have a bus number which
> will come
> from 01.0-[0000:0b].
> For e.g.
> B1---->B2-----C1
>
> where B1 is the bridge corresponding to the hot-pluggable
> slot, while B2
> is the bridge on the add-in card and C1 is the HBA behind the
> bridge B2 on
>
> the add-in card.
> Now B1 is having only 1 bus resource numbered “0b”. Now our
> problem is how
>
> the bus between B2 and C1 will get a bus resource (bus number) when
> hot-plugged.
> — Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256 You are currently
> subscribed
> to ntdev as: unknown lmsubst tag argument: ‘’ To unsubscribe
> send a blank
> email to xxxxx@lists.osr.com
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@attotech.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@hollistech.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>

17

Mark,

How is dynamic resource rebalancing different from the usual PnP process from
driver’s point of view? I hope that at least 90% of device drivers correctly
handle PnP.

Dmitriy Budko, VMware

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark Roddy
Sent: Friday, December 02, 2005 8:00 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] PCI Express problem

This is where dynamic resource rebalancing comes in, and I believe that it
is not available until the vista timeframe. Of course it will also break 90%
of device drivers out there that will not handle getting asked to release
their resources correctly, but that is a different story.

18

I thought that PCI Express was totally compatible to PCI at
driver level, am I mistaken ? It will be a pity if we have
hardware level compatibility up to Transaction Level, but then
Windows makes it incompatible at higher level! Now, it’s going
to be interesting to see if I can handle those dual-Opteron Tyan
motherboards with four 16-lane PCI connectors…

Alberto.

----- Original Message -----
From: “Dmitriy Budko”
To: “Windows System Software Devs Interest List”

Sent: Monday, December 05, 2005 9:17 PM
Subject: RE: [ntdev] PCI Express problem

Mark,

How is dynamic resource rebalancing different from the usual PnP
process from
driver’s point of view? I hope that at least 90% of device
drivers correctly
handle PnP.

Dmitriy Budko, VMware

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark
Roddy
Sent: Friday, December 02, 2005 8:00 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] PCI Express problem

This is where dynamic resource rebalancing comes in, and I
believe that it
is not available until the vista timeframe. Of course it will
also break 90%
of device drivers out there that will not handle getting asked
to release
their resources correctly, but that is a different story.

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag
argument: ‘’
To unsubscribe send a blank email to
xxxxx@lists.osr.com

19

While every driver handles start->remove transitions (hopefully), it is
difficult to say how many drivers handle start->stop->start properly
which is what dynamic resource rebalancing would send to a driver.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Dmitriy Budko
Sent: Monday, December 05, 2005 6:18 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] PCI Express problem

Mark,

How is dynamic resource rebalancing different from the usual PnP process
from
driver’s point of view? I hope that at least 90% of device drivers
correctly
handle PnP.

Dmitriy Budko, VMware

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark Roddy
Sent: Friday, December 02, 2005 8:00 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] PCI Express problem

This is where dynamic resource rebalancing comes in, and I believe that
it
is not available until the vista timeframe. Of course it will also break
90%
of device drivers out there that will not handle getting asked to
release
their resources correctly, but that is a different story.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

20

Alberto Moreira wrote:

I thought that PCI Express was totally compatible to PCI at driver
level, am I mistaken ?

I’m not sure that’s quite precise enough. There are features in PCI
Express that simply have no parallel in PCI. For example, hot
plugging, the extended configuration space, and message-based
interrupts. Current operating systems don’t understand those things in
the context of a PCI device, so a driver is never going to encounter them.

When we get to Vista, which is PCI Express aware, drivers for PCI
Express devices will have to handle conditions that just haven’t come up
before.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.