I am developing a system controller DLL for a PXI system. I must get two PCI switches' bus numbers in that DLL. I think I can't access to PCI config space of PCI devices from user mode but I need the information at offset 0x18 of PCI config space.
I saw same implementations of a filter driver for PCI bus to read config space of devices on the PCI bus. Is it possible to create a filter driver (I read on some other topic that it is not possible with WDF) so that I can access to that filter driver from user space to read configuration space of any PCI device?
In a PXI system there may be multiple devices connected and I am not sure if bus numbers and subordinate bus numbers of PCI switches are persisting. That is why I want to read their bus and subordinate bus numbers.
KMDF certainly supports device filter drivers. It doesn't directly support 'bus filter drivers', but it is entirely possible to write a bus filter driver using WDF. It isn't clear to me that you need a bus filter driver. If your 'PCI switches' are discrete PCI devices then you can simply write a KMDF filter for those devices and read the config space for the device using the PCI interface returned from WdfFdoQueryForInterface.
If you can't get the required info from the the OS device database, you will have to pursue something lower level.
It sounds like your use case may be in a somewhat controlled environment. Instead of writing a filter driver for PCIe devices, you could also do one of two other alternatives. 1) Write a simple "devmem" driver that allows mapping device memory ranges into a user-mode program. Then read the ACPI table, using documented user-mode APIs to find the PCIe ECAM address(s), map them (using the devmem driver) and do whatever examination of the PCIe config space you want from a user-mode program. 2) Write a kernel driver that does this and exposes an ioctl/wmi interface. You might be able to write a UMDF driver that does ECAM access too.
A downside of reading the ECAM directly is it does not work in a VM, which uses a virtual PCIe bus that has no ECAM. I believe a filter driver will query for the config space interface and get something that works on the virtual PCIe bus.
If you need to get HLK driver certification, reading the ECAM may be frowned on, although I have seen some utility drivers (needed for firmware/configuration updates) from large datacenter hardware vendors do this. If you just need attestation signed drivers, then your kernel code can do whatever.
There has been some noise about Microsoft potentially restricting kernel mode access in the future (Apple MacOS has already done this). This is the fallout of the buggy CrowdStrike driver that caused many systems to become unbootable. The more you can do from user mode the better.
Could you clarify what kind of environment all this needs to work in? An embedded computer in a lab running your custom PCIe hardware? A cloud datacenter? Your grandmother's desktop computer?
Initially, I hesitated to develop a filter driver for the PCI switch devices on the controller board because I wasn't sure if we would use different models. However, I've now learned that we will be using Broadcom's 8750, and I believe developing a filter driver specifically for that is the better option.
I am relatively new to Windows driver development, so some concepts are still unfamiliar to me. While exploring Broadcom's PlxSdk, I noticed its driver uses the 0xCF8 and 0xCFC ports to read the PCI device configuration space. I've heard that this method can cause system crashes in certain situations, so I’m unsure if relying on PlxSdk or accessing the configuration space through the 0xCF8 port is the best approach. I just learned about ECAM yesterday after starting this discussion, and I realize that defining our requirements clearly before developing the driver is crucial.
The environment is a PXIe system, like National Instrument's PXI Systems. A system controller (essentially a computer) needs to communicate with other slots on the chassis via the PCI bus. Some vendors offer PXI resource managers that handle resource enumeration, address allocation, etc. These resource managers need to obtain bus information—such as the number of links connecting the system controller to the chassis, link widths, and their bus numbers—from the system module's DLL.
Thank you both for your suggestions. I'll continue researching before moving forward.
I could not imagine using the x64 0xCF8 and 0xCFC ports for PCI config space access on a modern system.
You can only access the first 256 bytes of config space and many PCIe devices now have important info/control at config addresses > 0xFF. This port access mechanism can only select a b:d:f address, not a s:b:d:f address (see #3).
Using the port is also problematic from a synchronization viewpoint. Accesses to the config address and value register must not be interrupted and generally on Windows you never fully disable interrupts (it uses a nested priority interrupt system instead). You also can't have concurrent core accesses using the port based config interface, so you would have to using an irql elevating lock around the config port access. It would be bad for core 1 to write the desired config address but before it can read the value, core 2 writes the address it wants to the same port. Using the ECAM is just a single read/write to access a config address (unless the device uses internal address indexing), so you have no concurrent core lock problem.
Some modern systems also have multiple PCIe segments (like the 192 core ARM64 servers I work on), so a full config address is segment:bus:device:function, not just bus:device:function. The ECAM may have multiple disjoint memory ranges to cover the full s:b:d:f config address space. These ranges are defined by the ACPI table for the ECAM. Systems with s:b:d:f also have some implications if you want to do peer-to-peer PCIe transfers, as PCIe requests don't use s:b:d:f addressing for response tlps. Systems may (or may not) allow intersegment PCIe to PCIe transfers by using a 64-bit mapped target bus address that goes through the SMMU/IOMMU, for a response tlp the SMMU/IOMMU does the correct routing for the requesting PCIe segment. These multi PCIe segment systems may have unique 32-bit PCIe target addresses for each segment, so have a translation applied, like 32-bit PCIe bus address 0x8000 on segment 4 may be mapped to a full 64-bit global address like 0x400000008000. Having multiple PCIe segments has the theoretical plus that you get a lot more potential 32-bit BAR space on a system, but also can cause a sticky little problem that the 32-bit bridge window addresses need to not overlap RAM addresses. This is so the bridge can tell if an upstream read/write tlp with a memory address should be routed upstream toward the root (i.e. global memory address) or downstream toward some peer port. Windows does not by default set a bit in the bridge that makes upstream requests always be forwarded to the SMMU/IOMMU. The PCIe bridge specs unfortunately only define a 32-bit address range for non-prefetchable addresses, even though a number of PCIe devices I've seen can work with 64-bit non-prefetchable BAR assignments.
I/O port instructions are not available on most processors except x64. PCIe has both memory and port read/write tlps, some processors will map a memory range to the 64K port x64 style port range so PCIe port transactions can be generated with memory accesses. Some processors will not have this extra bridge BAR range that generates port tlps, so a system may not be able to generate PCIe port tlps, only memory tlps.
PCIe config space tlps are a different type of request from memory or port tlps, which can be generated from ECAM accesses. Non-x64 systems are being used quite a bit now, and if you are writing a Windows driver it's appropriate to avoid things in the driver that are not portable across processor architectures. At the moment, Windows runs on x64 and ARM64 architectures. In the past it also ran on Alpha, MIPS, POWER, and Itanium architectures, and someday it may run on something else like RISC-V.
I thought of another downside of using a bus filter driver for config access. The filter will only work if the OS can detect the PCIe device. This is not always the case, like if it's a FPGA/Firmware controlled device and the firmware image is damaged. The device may not correctly respond to the normal PCIe enumeration, even though it can still be partially accessed (like it's firmware update interface) by PCIe config accesses. It' more fun to recover a bricked device (or running diagnostics) by running some software than it is to require taking it out and connecting JTAG or similar hardware debug interface so you can flash the firmware, so the OS can detect it again.
If the device does not respond to normal enumeration there is no windows pnp driver that can access it without violating the rules.
The point of using a filter driver (either a device filter driver or a bus filter driver) is to get access to the supported methods for reading/writing pci* config space.
I created a software driver to access ECAM address space. It can successfully read bus number and subordinate bus number of PCI devices. I leave the code here in case any one needs it:
You need to provide b.d.f bus parameters to read the config space. This does not exactly solves my problem, but I have some other plans to achieve it.
PHYSICAL_ADDRESS ecamAddress;
ecamAddress.QuadPart = 0xE0000000 + (busInfo->b << 20) + (busInfo->d << 15) + (busInfo->f << 12);
// Map ECAM base address
PVOID configSpace = MmMapIoSpace(ecamAddress, 256, MmNonCached);
if (configSpace == NULL) {
status = STATUS_INSUFFICIENT_RESOURCES;
goto Done;
}
// Read a 32-bit value
ULONG value = READ_REGISTER_ULONG(static_cast<PULONG>(configSpace));
USHORT vendorId = value & 0xFFFF;
USHORT deviceId = (value >> 16) & 0xFFFF;
#if DBG
DbgPrint("Vendor ID: 0x%04X\n", vendorId);
DbgPrint("Device ID: 0x%04X\n", deviceId);
#endif
// Read a 32-bit value
value = READ_REGISTER_ULONG(reinterpret_cast<PULONG>(static_cast<PUCHAR>(configSpace) + 0x18));
#if DBG
DbgPrint("Bus info ID: 0x%08X\n", value);
#endif
MmUnmapIoSpace(configSpace, 256);
Edit: You also need to find the ECAM base address (0xE0000000 in this case)
This might work on your specific computer, but there is no way you can put this in a production driver that you intend to release into the wild. There are ways to get this information using supported system methods that do the proper interlocking.
Honestly, I would fire a programmer that hard-coded a physical address into a kernel driver.
I know ECAM base address might be different for systems. I am not going to use this in my driver, I just wanted to share this code if any one needs it.