RE: PCI bus driver failed to identify device

I know many MSFT guys in this mail list, and hope there’s someone able to looking into the PCI code.
I tried this issue in Win7/8/8.1 OS, and has similar scenario that the bus_master bit is failed to be set, which cause the PCI device un-usable. If anyone knows the implementation detail, hope you could give me a shine lightness that what the PCI bus do before setting a PCI device as master?
===================Best Regards!Zhang Pei

From: xxxxx@live.com
To: xxxxx@lists.osr.com
Subject: [ntdev] PCI bus driver failed to identify device
Date: Mon, 16 May 2016 05:59:48 +0000

Hi Guys,
I??m working on a project for GPU device’s virtualization into guest OS (Windows, Linux). Recent weeks I met a PCI bus related issue which has me trap in deep crazy.
This is a x64 platform and 64bits guest OS is used. There are some BAR IO/mem resources which is assigned through the BARs. For BAR3/4, 1G size PCI aperture space is reserved. There??s no any issue if the assigned BAR3/4 address is under 4G address space, which means cfg space offset 0x1C’s value is 0. Issue only happens when its address is extended to upper 4G space. When entering OS, in device manger, the device??s status is error code 12: cannot find enough free resource that it can use.
I thought this issue would be related to memory map that upper 4G mapping failure from PCI BIOS side. But I’m wrong, I find I could successfully read/write the mapped upper 4G address through tool like RW-Everything, and also could get the BAR size successfully by writing 0xFFFFFFFF to BAR and then read back through the tool.
Another valuable thing is that, Linux guest don’t have such issue and could identify the device successfully and use the virtualized device. So I compared the cfg space data under the 2 guest OSs, and found the major difference is the COMMAND/STATUS value in offset 0x4. Windows guest??s PCI_COMMAND value is 0x103, and the bus_master bit is not set(but_master is bit 2). Linux guest??s PCI_COMMAND value is 0x407, which has the bus_master bit set.
I thought the bus_master bit is the key reason under Windows guest that make the system failed to identify the device. But I just don??t have any idea what??s the PCI bus driver??s execution logic during emulating PCI device. There must be something which don’t satisfy Windows PCI bus driver’s requirement, but I don’t know how to find it? I post the whole 256 bytes cfg space content below, if it could help.
Type:PCI Bus 00 Device 02 Function 00
Width:01
00=86 01=80 02=12 03=04 04=03 05=01 06=90 07=00
08=06 09=00 0A=00 0B=03 0C=00 0D=00 0E=00 0F=00
10=04 11=00 12=40 13=FE 14=00 15=00 16=00 17=00
18=0C 19=00 1A=00 1B=00 1C=01 1D=00 1E=00 1F=00
20=01 21=C0 22=00 23=00 24=00 25=00 26=00 27=00
28=00 29=00 2A=00 2B=00 2C=28 2D=10 2E=A4 2F=05
30=00 31=00 32=86 33=FE 34=90 35=00 36=00 37=00
38=00 39=00 3A=00 3B=00 3C=FF 3D=01 3E=00 3F=00
40=09 41=00 42=0C 43=01 44=61 45=A0 46=04 47=62
48=D0 49=00 4A=44 4B=56 4C=00 4D=00 4E=00 4F=00
50=01 51=02 52=00 53=00 54=31 55=00 56=00 57=00
58=00 59=00 5A=00 5B=00 5C=01 5D=00 5E=20 5F=DD
60=00 61=00 62=00 63=00 64=00 65=00 66=00 67=00
68=00 69=00 6A=00 6B=00 6C=00 6D=00 6E=00 6F=00
70=00 71=00 72=00 73=00 74=00 75=00 76=00 77=00
78=00 79=00 7A=00 7B=00 7C=00 7D=00 7E=00 7F=00
80=00 81=00 82=00 83=00 84=00 85=00 86=00 87=00
88=00 89=00 8A=00 8B=00 8C=00 8D=00 8E=00 8F=00
90=05 91=D0 92=00 93=00 94=00 95=00 96=00 97=00
98=00 99=00 9A=00 9B=00 9C=00 9D=00 9E=00 9F=00
A0=00 A1=00 A2=00 A3=00 A4=13 A5=00 A6=06 A7=03
A8=00 A9=00 AA=00 AB=00 AC=00 AD=00 AE=00 AF=00
B0=00 B1=00 B2=00 B3=00 B4=00 B5=00 B6=00 B7=00
B8=00 B9=00 BA=00 BB=00 BC=00 BD=00 BE=00 BF=00
C0=00 C1=00 C2=00 C3=00 C4=00 C5=00 C6=00 C7=00
C8=00 C9=00 CA=00 CB=00 CC=00 CD=00 CE=00 CF=00
D0=01 D1=A4 D2=22 D3=00 D4=00 D5=00 D6=00 D7=00
D8=00 D9=00 DA=00 DB=00 DC=00 DD=00 DE=00 DF=00
E0=00 E1=00 E2=00 E3=00 E4=00 E5=00 E6=00 E7=00
E8=00 E9=80 EA=00 EB=00 EC=00 ED=00 EE=00 EF=00
F0=00 F1=00 F2=00 F3=00 F4=00 F5=00 F6=00 F7=00
F8=00 F9=00 FA=06 FB=00 FC=00 FD=E0 FE=FF FF=BF
;Device/Vendor ID 0x04128086
;Revision ID 0x06
;Class Code 0x030000
;Cacheline Size 0x00
;Latency Timer 0x00
;Interrupt Pin INTA
;Interrupt Line None
;BAR1 0xFE400004
;BAR2 0x00000000
;BAR3 0x0000000C
;BAR4 0x00000001
;BAR5 0x0000C001
;BAR6 0x00000000
;Expansion ROM 0xFE860000
;Subsystem ID 0x05A41028

Sent from Windows Mail

NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!

Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:>

I vaguely remembered that the 4GB PCI space issue that I met. Several years
ago I was working on virtual WDDM driver that enables AERO in guest Windows
OS(Win7/Win8/Win8.1). One of the issue is >4GB aperture space mapping. We
were using QEMU/kvm as hypervisor on Linux host. “Seabios” update was the
answer back then. Not sure if this helps your problem.

2016-05-17 7:08 GMT+08:00 ZhangPei :

> I know many MSFT guys in this mail list, and hope there’s someone able to
> looking into the PCI code.
>
> I tried this issue in Win7/8/8.1 OS, and has similar scenario that the
> bus_master bit is failed to be set, which cause the PCI device un-usable.
> If anyone knows the implementation detail, hope you could give me a shine
> lightness that what the PCI bus do before setting a PCI device as master?
>
> ===================
> Best Regards!
> Zhang Pei
>
>
>
> ------------------------------
> From: xxxxx@live.com
> To: xxxxx@lists.osr.com
> Subject: [ntdev] PCI bus driver failed to identify device
> Date: Mon, 16 May 2016 05:59:48 +0000
>
>
> Hi Guys,
>
> I’m working on a project for GPU device’s virtualization into guest OS
> (Windows, Linux). Recent weeks I met a PCI bus related issue which has me
> trap in deep crazy.
>
> This is a x64 platform and 64bits guest OS is used. There are some BAR
> IO/mem resources which is assigned through the BARs. For BAR3/4, 1G size
> PCI aperture space is reserved. There’s no any issue if the assigned BAR3/4
> address is under 4G address space, which means cfg space offset 0x1C’s
> value is 0. Issue only happens when its address is extended to upper 4G
> space. When entering OS, in device manger, the device’s status is error
> code 12: cannot find enough free resource that it can use.
>
> I thought this issue would be related to memory map that upper 4G mapping
> failure from PCI BIOS side. But I’m wrong, I find I could successfully
> read/write the mapped upper 4G address through tool like RW-Everything, and
> also could get the BAR size successfully by writing 0xFFFFFFFF to BAR and
> then read back through the tool.
>
> Another valuable thing is that, Linux guest don’t have such issue and
> could identify the device successfully and use the virtualized device. So I
> compared the cfg space data under the 2 guest OSs, and found the major
> difference is the COMMAND/STATUS value in offset 0x4. Windows guest’s
> PCI_COMMAND value is 0x103, and the bus_master bit is not set(but_master is
> bit 2). Linux guest’s PCI_COMMAND value is 0x407, which has the bus_master
> bit set.
>
> I thought the bus_master bit is the key reason under Windows guest that
> make the system failed to identify the device. But I just don’t have any
> idea what’s the PCI bus driver’s execution logic during emulating PCI
> device. There must be something which don’t satisfy Windows PCI bus
> driver’s requirement, but I don’t know how to find it? I post the whole
> 256 bytes cfg space content below, if it could help.
>
> Type:PCI Bus 00 Device 02 Function 00
> Width:01
> 00=86 01=80 02=12 03=04 04=03 05=01 06=90 07=00
> 08=06 09=00 0A=00 0B=03 0C=00 0D=00 0E=00 0F=00
> 10=04 11=00 12=40 13=FE 14=00 15=00 16=00 17=00
> 18=0C 19=00 1A=00 1B=00 1C=01 1D=00 1E=00 1F=00
> 20=01 21=C0 22=00 23=00 24=00 25=00 26=00 27=00
> 28=00 29=00 2A=00 2B=00 2C=28 2D=10 2E=A4 2F=05
> 30=00 31=00 32=86 33=FE 34=90 35=00 36=00 37=00
> 38=00 39=00 3A=00 3B=00 3C=FF 3D=01 3E=00 3F=00
> 40=09 41=00 42=0C 43=01 44=61 45=A0 46=04 47=62
> 48=D0 49=00 4A=44 4B=56 4C=00 4D=00 4E=00 4F=00
> 50=01 51=02 52=00 53=00 54=31 55=00 56=00 57=00
> 58=00 59=00 5A=00 5B=00 5C=01 5D=00 5E=20 5F=DD
> 60=00 61=00 62=00 63=00 64=00 65=00 66=00 67=00
> 68=00 69=00 6A=00 6B=00 6C=00 6D=00 6E=00 6F=00
> 70=00 71=00 72=00 73=00 74=00 75=00 76=00 77=00
> 78=00 79=00 7A=00 7B=00 7C=00 7D=00 7E=00 7F=00
> 80=00 81=00 82=00 83=00 84=00 85=00 86=00 87=00
> 88=00 89=00 8A=00 8B=00 8C=00 8D=00 8E=00 8F=00
> 90=05 91=D0 92=00 93=00 94=00 95=00 96=00 97=00
> 98=00 99=00 9A=00 9B=00 9C=00 9D=00 9E=00 9F=00
> A0=00 A1=00 A2=00 A3=00 A4=13 A5=00 A6=06 A7=03
> A8=00 A9=00 AA=00 AB=00 AC=00 AD=00 AE=00 AF=00
> B0=00 B1=00 B2=00 B3=00 B4=00 B5=00 B6=00 B7=00
> B8=00 B9=00 BA=00 BB=00 BC=00 BD=00 BE=00 BF=00
> C0=00 C1=00 C2=00 C3=00 C4=00 C5=00 C6=00 C7=00
> C8=00 C9=00 CA=00 CB=00 CC=00 CD=00 CE=00 CF=00
> D0=01 D1=A4 D2=22 D3=00 D4=00 D5=00 D6=00 D7=00
> D8=00 D9=00 DA=00 DB=00 DC=00 DD=00 DE=00 DF=00
> E0=00 E1=00 E2=00 E3=00 E4=00 E5=00 E6=00 E7=00
> E8=00 E9=80 EA=00 EB=00 EC=00 ED=00 EE=00 EF=00
> F0=00 F1=00 F2=00 F3=00 F4=00 F5=00 F6=00 F7=00
> F8=00 F9=00 FA=06 FB=00 FC=00 FD=E0 FE=FF FF=BF
> ;Device/Vendor ID 0x04128086
> ;Revision ID 0x06
> ;Class Code 0x030000
> ;Cacheline Size 0x00
> ;Latency Timer 0x00
> ;Interrupt Pin INTA
> ;Interrupt Line None
> ;BAR1 0xFE400004
> ;BAR2 0x00000000
> ;BAR3 0x0000000C
> ;BAR4 0x00000001
> ;BAR5 0x0000C001
> ;BAR6 0x00000000
> ;Expansion ROM 0xFE860000
> ;Subsystem ID 0x05A41028
>
>
> Sent from Windows Mail
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: <
> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: <
> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:></http:>

You can try to disable the memory remapping in BIOS to force your PCI BAR always under 4G.

sorry for even later reply.
For my case, we were able to map several GB (> 4GB) of device RAM into
64bit PCIE BAR. The 64bit PCIE BAR was designed to support 16GB. I had no
problem mapping 4GB of device RAM into above 4GB address.The guest OS we’ve
tried were Win7(64bit) and Win8(64bit).

Back then we fixed an issue in SeaBIOS such that it allowed device RAM
mapped to address > 4GB, if my memory is correct.

2016-06-01 4:36 GMT+08:00 ZhangPei :

> Update even it’s late.
>
>
>
> Tried the latest seabios, issue still exits. Suppose your fix is to move
> the aperture below 4G, and aperture size is not very large. I still not get
> a final conclusion, but doubt it’s a OS’s limitation when processing the
> >4G 64Bit BAR address. I’m trying to get MS’s official support now.
>
>
>
> **************
> BRs,
> Pei, Zhang
>
>
>
> *From: *Lin JiaBang
> *Sent: *Tuesday, May 17, 2016 11:15
> *To: *Windows System Software Devs Interest List
> *Subject: *Re: [ntdev] PCI bus driver failed to identify device
>
>
> I vaguely remembered that the 4GB PCI space issue that I met. Several
> years ago I was working on virtual WDDM driver that enables AERO in guest
> Windows OS(Win7/Win8/Win8.1). One of the issue is >4GB aperture space
> mapping. We were using QEMU/kvm as hypervisor on Linux host. “Seabios”
> update was the answer back then. Not sure if this helps your problem.
>
>
>
> 2016-05-17 7:08 GMT+08:00 ZhangPei :
>
> I know many MSFT guys in this mail list, and hope there’s someone able to
> looking into the PCI code.
>
> I tried this issue in Win7/8/8.1 OS, and has similar scenario that the
> bus_master bit is failed to be set, which cause the PCI device un-usable.
> If anyone knows the implementation detail, hope you could give me a shine
> lightness that what the PCI bus do before setting a PCI device as master?
>
> ===================
> Best Regards!
> Zhang Pei
>
>
>
> ------------------------------
> From: xxxxx@live.com
> To: xxxxx@lists.osr.com
> Subject: [ntdev] PCI bus driver failed to identify device
> Date: Mon, 16 May 2016 05:59:48 +0000
>
>
> Hi Guys,
>
> I’m working on a project for GPU device’s virtualization into guest OS
> (Windows, Linux). Recent weeks I met a PCI bus related issue which has me
> trap in deep crazy.
>
> This is a x64 platform and 64bits guest OS is used. There are some BAR
> IO/mem resources which is assigned through the BARs. For BAR3/4, 1G size
> PCI aperture space is reserved. There’s no any issue if the assigned BAR3/4
> address is under 4G address space, which means cfg space offset 0x1C’s
> value is 0. Issue only happens when its address is extended to upper 4G
> space. When entering OS, in device manger, the device’s status is error
> code 12: cannot find enough free resource that it can use.
>
> I thought this issue would be related to memory map that upper 4G mapping
> failure from PCI BIOS side. But I’m wrong, I find I could successfully
> read/write the mapped upper 4G address through tool like RW-Everything, and
> also could get the BAR size successfully by writing 0xFFFFFFFF to BAR and
> then read back through the tool.
>
> Another valuable thing is that, Linux guest don’t have such issue and
> could identify the device successfully and use the virtualized device. So I
> compared the cfg space data under the 2 guest OSs, and found the major
> difference is the COMMAND/STATUS value in offset 0x4. Windows guest’s
> PCI_COMMAND value is 0x103, and the bus_master bit is not set(but_master is
> bit 2). Linux guest’s PCI_COMMAND value is 0x407, which has the bus_master
> bit set.
>
> I thought the bus_master bit is the key reason under Windows guest that
> make the system failed to identify the device. But I just don’t have any
> idea what’s the PCI bus driver’s execution logic during emulating PCI
> device. There must be something which don’t satisfy Windows PCI bus
> driver’s requirement, but I don’t know how to find it? I post the whole
> 256 bytes cfg space content below, if it could help.
>
> Type:PCI Bus 00 Device 02 Function 00
> Width:01
> 00=86 01=80 02=12 03=04 04=03 05=01 06=90 07=00
> 08=06 09=00 0A=00 0B=03 0C=00 0D=00 0E=00 0F=00
> 10=04 11=00 12=40 13=FE 14=00 15=00 16=00 17=00
> 18=0C 19=00 1A=00 1B=00 1C=01 1D=00 1E=00 1F=00
> 20=01 21=C0 22=00 23=00 24=00 25=00 26=00 27=00
> 28=00 29=00 2A=00 2B=00 2C=28 2D=10 2E=A4 2F=05
> 30=00 31=00 32=86 33=FE 34=90 35=00 36=00 37=00
> 38=00 39=00 3A=00 3B=00 3C=FF 3D=01 3E=00 3F=00
> 40=09 41=00 42=0C 43=01 44=61 45=A0 46=04 47=62
> 48=D0 49=00 4A=44 4B=56 4C=00 4D=00 4E=00 4F=00
> 50=01 51=02 52=00 53=00 54=31 55=00 56=00 57=00
> 58=00 59=00 5A=00 5B=00 5C=01 5D=00 5E=20 5F=DD
> 60=00 61=00 62=00 63=00 64=00 65=00 66=00 67=00
> 68=00 69=00 6A=00 6B=00 6C=00 6D=00 6E=00 6F=00
> 70=00 71=00 72=00 73=00 74=00 75=00 76=00 77=00
> 78=00 79=00 7A=00 7B=00 7C=00 7D=00 7E=00 7F=00
> 80=00 81=00 82=00 83=00 84=00 85=00 86=00 87=00
> 88=00 89=00 8A=00 8B=00 8C=00 8D=00 8E=00 8F=00
> 90=05 91=D0 92=00 93=00 94=00 95=00 96=00 97=00
> 98=00 99=00 9A=00 9B=00 9C=00 9D=00 9E=00 9F=00
> A0=00 A1=00 A2=00 A3=00 A4=13 A5=00 A6=06 A7=03
> A8=00 A9=00 AA=00 AB=00 AC=00 AD=00 AE=00 AF=00
> B0=00 B1=00 B2=00 B3=00 B4=00 B5=00 B6=00 B7=00
> B8=00 B9=00 BA=00 BB=00 BC=00 BD=00 BE=00 BF=00
> C0=00 C1=00 C2=00 C3=00 C4=00 C5=00 C6=00 C7=00
> C8=00 C9=00 CA=00 CB=00 CC=00 CD=00 CE=00 CF=00
> D0=01 D1=A4 D2=22 D3=00 D4=00 D5=00 D6=00 D7=00
> D8=00 D9=00 DA=00 DB=00 DC=00 DD=00 DE=00 DF=00
> E0=00 E1=00 E2=00 E3=00 E4=00 E5=00 E6=00 E7=00
> E8=00 E9=80 EA=00 EB=00 EC=00 ED=00 EE=00 EF=00
> F0=00 F1=00 F2=00 F3=00 F4=00 F5=00 F6=00 F7=00
> F8=00 F9=00 FA=06 FB=00 FC=00 FD=E0 FE=FF FF=BF
> ;Device/Vendor ID 0x04128086
> ;Revision ID 0x06
> ;Class Code 0x030000
> ;Cacheline Size 0x00
> ;Latency Timer 0x00
> ;Interrupt Pin INTA
> ;Interrupt Line None
> ;BAR1 0xFE400004
> ;BAR2 0x00000000
> ;BAR3 0x0000000C
> ;BAR4 0x00000001
> ;BAR5 0x0000C001
> ;BAR6 0x00000000
> ;Expansion ROM 0xFE860000
> ;Subsystem ID 0x05A41028
>
>
> Sent from Windows Mail
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: <
> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: <
> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
>
>
> — NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars
> on crash dump analysis, WDF, Windows internals and software drivers!
> Details at To unsubscribe, visit the List Server section of OSR Online at
></http:></http:>

I saw in your OP that you had problem with BAR3/BAR4 mapped above 4GB. The
BAR3 had PCI_BASE_ADDRESS_MEM_TYPE_64(0x04) set, but BAR4
had PCI_BASE_ADDRESS_SPACE_IO set. This might confuse OS.

I saw BAR1 is also 64bit capable, and BAR2 is 0. This looks perfect. I bet
you could do above 4GB address mapping in BAR1.

;BAR1 0xFE400004
;BAR2 0x00000000
;BAR3 0x0000000C
;BAR4 0x00000001

BAR 4 is incorrect.