Multifunction device BAR conflict

NTDEV
1

Hello,

Function 1 has 4 BARs specified as follows:

0: kd> !devext ffffc30971a72270
PDO Extension, Bus 0x1, Device 0, Function 1.
  DevObj 0xffffc30971a72120  Parent FDO DevExt 0xffffc30971a49190
  Device State = PciNotStarted
  ..
  Header Type 0, Class Base/Sub 06/80  (Bridge/PCI to 'Other')
  Programming Interface: 00, Revision: 00, IntPin: 00, RawLine ff
  Possible Decodes ((cmd & 7) = 7): BMI
  Capabilities: Ptr=40, power msi msix express 
  Express capabilities: (BIOS controlled) aer 
  Logical Device Power State: D0
  Device Wake Level:          D3
  WaitWakeIrp:                <none>
  Requirements:     Alignment Length    Minimum          Maximum
    BAR0    Mem:    00400000  00400000  0000000000000000 00000000ffffffff
    BAR2    Mem:    10000000  10000000  0000000000000000 ffffffffffffffff
    BAR4    Mem:    04000000  04000000  0000000000000000 00000000ffffffff
    BAR5    Mem:    00800000  00800000  0000000000000000 00000000ffffffff
  Resources: <none>
  Interrupt Requirement:
    Message Based: Type - Msi-X, 0x4 messages requested
  Interrupt Resource: <none>

Function 2 has 1 BAR whose size is equal to BAR5.

0: kd> !devext ffffdc8d4a80f1b0
PDO Extension, Bus 0x1, Device 0, Function 2.
  DevObj 0xffffdc8d4a80f060  Parent FDO DevExt 0xffffdc8d4a79e490
  Device State = PciNotStarted
  ..
  Header Type 0, Class Base/Sub 08/80  (Base System Device/'Other' base system device)
  Programming Interface: 00, Revision: 00, IntPin: 00, RawLine ff
  Possible Decodes ((cmd & 7) = 7): BMI
  Capabilities: Ptr=40, power msi msix express 
  Express capabilities: (BIOS controlled) aer 
  Logical Device Power State: D0
  Device Wake Level:          D3
  WaitWakeIrp:                <none>
  Requirements:     Alignment Length    Minimum          Maximum
    BAR0    Mem:    00800000  00800000  0000000000000000 00000000ffffffff
  Resources: <none>
  Interrupt Requirement:
    Message Based: Type - Msi-X, 0x3 messages requested
  Interrupt Resource: <none>

The problem is that when Function 2 is loaded 1st, BAR0 address is equal to BAR5 and reads into VA = MmMapIoSpace(BAR0) return -1. If Function 1 is loaded before Function 2, then there is no longer a conflict and all BARs are available.

The OS does not adjust BAR0 for Function 2 if it is the 1st function to be loaded with EvtDevicePrepareHardware. It however does adjust it if it is loaded after Function 1.

I think that the BIOS enumerates the functions and can cause the conflict. Why isn't there a computation for Function 2 BARs if it is the 1st function to be loaded?

This problem happens often on Windows 10 LTSC. Other editions load the functions in a good order. The problem is present with manual testing on those as well.

Thank you for taking the time.
Călin

Update: Instead of failing EvtDeviceD0Entry, the driver can keep the function online and rely on a 2nd pass.

Enter EvtDriverDeviceAdd
Found Function 1:0.2 PDO = FFFF800CD9644060
Exit EvtDriverDeviceAdd with code 0x00000000
Enter EvtDevicePrepareHardware 1:0.2
PCI BAR0 MmMapIoSpace success with phys addr: F4800000, virt addr: FFFFE078FD400000, len: 800000
Exit EvtDevicePrepareHardware with code 00000000
Enter EvtDeviceD0Entry 1.0.2
TestBAR0 failed
EvtDeviceD0Entry returned 00000000

Enter EvtDriverDeviceAdd
Found Function 1:0.1 PDO = FFFF800CD9619060
Exit EvtDriverDeviceAdd with code 0x00000000

Enter EvtDeviceQueryStop 1:0.2
Exit EvtDeviceQueryStop with code 00000000
Enter EvtDeviceD0Exit 1:0.2
Exit EvtDeviceD0Exit with code 0x00000000
Enter EvtDeviceReleaseHardware 1:0.2
Exit EvtDeviceReleaseHardware with code 0x00000000

Enter EvtDevicePrepareHardware 1:0.2
PCI BAR0 MmMapIoSpace success with phys addr: F4000000, virt addr: FFFFE07912A00000, len: 800000
Exit EvtDevicePrepareHardware with code 00000000
Enter EvtDeviceD0Entry 1:0.2
EvtDeviceD0Entry returned 00000000

Enter EvtDevicePrepareHardware 1:0.1
PCI BAR0 MmMapIoSpace success with phys addr: F5000000, virt addr: FFFFE07913200000, len: 400000
PCI BAR2 MmMapIoSpace success with phys addr: D0000000, virt addr: FFFFE07913600000, len: 10000000
PCI BAR4 MmMapIoSpace success with phys addr: F0000000, virt addr: FFFFE07923600000, len: 4000000
PCI BAR5 MmMapIoSpace success with phys addr: F4800000, virt addr: FFFFE07927600000, len: 800000
Exit EvtDevicePrepareHardware with code 00000000
Enter EvtDeviceD0Entry 1:0.1
EvtDeviceD0Entry returned 00000000
2

So, when you say "load a function", you mean "when the driver for the function comes up"? Are you saying that 1:0.2 BAR0 literally gets assigned the same physical address as 1:0.1 BAR5? Are you 100% sure this is not a problem in the hardware?

3

Haha, I am not sure that the problem is in the firmware or ASIC. Yes, by load I mean that the driver controlling the function is started.

4

Hmmm… I’d say either (a) fix the hardware, or (b) have your drivers start in an order that happens to work. Otherwise… I’m not sure what you could do about this.