Hello,
I am writing a driver to give a user-space program the contents of the MMIO space of a PCI SPI device. However, when the user-space program reads it, it shows as all 0xff. I'm running Windows in a VM, if that's relevant.
I use MmMapIoSpace with a hardcoded physical address, and then RtlCopyBytes to copy the data into a buffer to send to user-space. Of course in the future I will programmatically retrieve the physical address.
PHYSICAL_ADDRESS addr = {.QuadPart = 0xc9206000ULL};
The PHYSICAL_ADDRESS was taken from lspci of the device:
00: 86 80 a4 a0 06 00 00 00 20 00 80 0c 00 00 00 00
10: 00 60 20 c9 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 3c 10 fe 87
30: 00 00 00 00 00 00 00 00 00 00 00 00 ff 00 00 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 cf ff 00 00 aa 08 00 10
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 b5 0f 21 01 00 00 00 00
According to the manual for the chipset I am using, the PCI BAR should be bits 12:31 of the DWORD at 0x10- 0xc92060.
When my user-space program reads the buffer given from the kernel module, it is all 0xff. Can anyone help me with this?
Yes, that's a 32-bit memory BAR at 0xC9206000. It's not bits 12:31, it's all 32 bits with the bottom 4 bits forced to 0. That's all part of the PCI spec.
0xff is what happens when there is a read that no device responded to. Have you used your VM software to assign that hardware to the VM? You don't automatically inherit all of the host hardware. That would cause chaos. Ownership has to be assigned, so the device can be removed from the host.
Yes, I have assigned the hardware- lspci shows the configuration space properly from within the VM, as seen in my original post. The emulator used is QEMU.
According to the Intel chipset datasheet (not posting here because the forum doesn't seem to allow links), bits 12:31 contain the address with 0:11 being used for other information. Although in this case it shouldn't matter because they are zeroed out anyway.
Is your host system Linux? Are you able to read that region from Linux? It should be exposed in /sys.
Yes, it is readable on Linux. Is it possible that QEMU does not support the device?
Also worth noting that the address at 0x10 is different when read from the VM and the host. On the host, it is 0x80400000, and I can read that from the host, but the 0xc9206000 is not readable from the VM.
I can think of many reasons why this wouldn't work. It is a core system device -- part of the Tiger Lake chipset. Perhaps the host operating system is not actually giving up control of it and you're getting a fake. This is just one function of a huge multifunction PCI device; perhaps the host will not allow the one function to be redirected.
I am not very surprised that it doesn't work.
I've tried again on a bare-metal Windows installation (Comet Lake machine- SPI datasheet is practically identical) and get the same problem.
lspci:
00: 86 80 a4 02 02 04 00 00 00 00 80 0c 00 00 00 00
10: 00 00 01 fe 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 28 10 51 09
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 cf ff 00 00 aa 08 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 b5 0f 00 01 00 00 00 00
I set the PHYSICAL_ADDRESS to 0xfe010000 and still receive all 0xff.
My driver's HandleIoctl routine (this is just a PoC):
PCHAR ptr = 0x0;
[...]
_Use_decl_annotations_
NTSTATUS HandleIoctl(PDEVICE_OBJECT DeviceObject, PIRP Irp)
{
NTSTATUS status = STATUS_SUCCESS;
PIO_STACK_LOCATION irpSp = IoGetCurrentIrpStackLocation(Irp);
if (irpSp->Parameters.DeviceIoControl.IoControlCode == IOCTL_TEST)
{
PCHAR buffer = MmGetSystemAddressForMdlSafe(Irp->MdlAddress, NormalPagePriority);
size_t outbuflen = irpSp->Parameters.DeviceIoControl.OutputBufferLength;
if (outbuflen < sizeof(PVOID) || !buffer) {
status = STATUS_INSUFFICIENT_RESOURCES;
goto ret;
}
PHYSICAL_ADDRESS addr = {.QuadPart = 0xfe010000ULL};
if (ptr == 0x0){
ptr = MmMapIoSpace(addr, 0x100, MmNonCached);
}
RtlCopyMemory(buffer, ptr, outbuflen);
Irp->IoStatus.Information = outbuflen;
} else {
status = STATUS_INVALID_DEVICE_REQUEST;
}
ret:
Irp->IoStatus.Status = status;
IoCompleteRequest(Irp, IO_NO_INCREMENT);
return status;
}
I tried on a bare-metal Windows installation (Comet Lake, SPI configuration registers identical) and it still doesn't work.
lspci:
00: 86 80 a4 02 02 04 00 00 00 00 80 0c 00 00 00 00
10: 00 00 01 fe 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 28 10 51 09
30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 cf ff 00 00 aa 08 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 b5 0f 00 01 00 00 00 00
My driver's HandleIoctl (just a PoC):
_Use_decl_annotations_
NTSTATUS HandleIoctl(PDEVICE_OBJECT DeviceObject, PIRP Irp)
{
NTSTATUS status = STATUS_SUCCESS;
PIO_STACK_LOCATION irpSp = IoGetCurrentIrpStackLocation(Irp);
if (irpSp->Parameters.DeviceIoControl.IoControlCode == IOCTL_TEST)
{
PCHAR buffer = MmGetSystemAddressForMdlSafe(Irp->MdlAddress, NormalPagePriority);
size_t outbuflen = irpSp->Parameters.DeviceIoControl.OutputBufferLength;
if (outbuflen < sizeof(PVOID) || !buffer) {
status = STATUS_INSUFFICIENT_RESOURCES;
goto ret;
}
PHYSICAL_ADDRESS addr = {.QuadPart = 0xfe010000ULL};
if (ptr == 0x0){
ptr = MmMapIoSpace(addr, 0x100, MmNonCached);
}
RtlCopyMemory(buffer, ptr, outbuflen);
Irp->IoStatus.Information = outbuflen;
} else {
status = STATUS_INVALID_DEVICE_REQUEST;
}
ret:
Irp->IoStatus.Status = status;
IoCompleteRequest(Irp, IO_NO_INCREMENT);
return status;
}
When userspace reads the returned buffer it is still all 0xff.
How is your IOCTL_TEST defined? Which METHOD did you choose?
And presumably you know that you need a matching MmUnmapIoSpace for every call to MmMapIoSpace.
You aren't doing any debug prints here. Do you know that ptr is getting a real address? Have you tried dumping the memory in kernel mode?
I added logging statements and it turns out MmMapIoSpace works fine (the pointer points to a buffer with the proper data), but after RtlCopyMemory finishes operation the output buffer contains all 0xff. Both ptr and buffer have real addresses.
#define IOCTL_TEST CTL_CODE(TEST_TYPE, 0xeee, METHOD_OUT_DIRECT, FILE_ANY_ACCESS)
This would be easier if you would just post a link to your driver and test app repository. Is HandleIoctl assigned to MajorFunction[IRP_DEVICE_CONTROL]? There's no reason why this would be running in the system process? Have you printed outbuflen to make sure it has a reasonable/expected value?
RtlCopyMemory is just a macro that calls memcpy, which is a compiler intrinsic. There aren't many ways for it to go wrong.