Answers Needed
- Is there a limit on the boundaries one can align a common buffer?
- Is there a way to translate a physical address to a virtual address?
Details?
We wrote a driver for XPe driver to talk to our PCI-based PLX switch, which has been working well. The driver allocates 3 buffers:
- Command flags (24 bytes) that reside in the scratchpad on the device,
- Mailbox for commands and replies (16MB), and
- Data buffer (32MB) for transferring large amounts of data.
The device requires the data buffers be aligned to 16MB and 32MB respectively.
We ported the driver to WES7 and had to reverse the allocation of the mailbox and data buffers to achieve the proper alignment. Now we are trying to port this driver to Win10 and are having problems so I believe we were just lucky in WES7?
The command flags work fine. We use WdfCommonBufferCreate to allocate the mailbox and data buffers. See the code at the bottom of this post.
Fundamental Problem…
We call WdfDeviceSetAlignmentRequirement to request the alignment before calling WdfCommonBufferCreate. We use WdfDeviceGetAlignmentRequirement and NTSTATUS to ensure the proper size buffers are allocated and the alignment has been properly set and both seem to be in order; however, the physical/logical addresses returned via WdfCommonBufferGetAlignedLogicalAddress are never properly aligned.
We also see unexpectedly close virtual addresses for the two requested buffers. We understand that the virtual addresses will NOT necessarily also have the same alignment as the physical addresses since our buffer size > our page size.
Our Approach to Address Alignment Problem…
The code post below is for function AllocatePCIBuffer2 which requests 2x the necessary buffer space and then determine the base address that has the desired alignment. I then computed the delta from the original base address and added that to the base virtual address returned by WdfCommonBufferGetAlignedVirtualAddress. The driver loads fine at boot but this updated driver code crashes the system the first time an app calls in to request the virtual addresses for its user address space. I believe this is because virtual address space isn?t contiguous but actually references PDEs/PTEs. I cannot find any examples/description of how to translate physical addresses to virtual ? probably because there is no good way to do this and there is probably a better way to go about this; is there?
Questions
- What is the limit on common buffer alignment? Our page size is 4K; I know we can configure it to be 2M but that is still shy of the buffer sizes we need.
- Is there a way to translate a physical address to a virtual address? I suppose one may be able to run thru all the PTEs!
- Most importantly, Should we be using a different approach?
A sample WinDbg output our driver generates is at the bottom of this post.
// ==============================================================================
// ==============================================================================
NTSTATUS AllocatePCIBuffer2(WDFDEVICE device, size_t size, WDFCOMMONBUFFER* buffer,
PVOID* virtualAddress, PHYSICAL_ADDRESS* physicalAddress)
{
NTSTATUS status = STATUS_SUCCESS;
// Entry housekeeping
KdPrintEx((DPFLTR_IHVDRIVER_ID, DPFLTR_INFO_LEVEL,“\n->AllocatePCIBuffer2\n”));
PAGED_CODE();
// No scatter/gather, 32-bit addressing, non-duplex, 4096-byte alignment.
WDFDMAENABLER enabler;
WDF_DMA_ENABLER_CONFIG dmaConfig;
WDF_DMA_ENABLER_CONFIG_INIT(&dmaConfig, WdfDmaProfilePacket, size);
WdfDeviceSetAlignmentRequirement(device, size-1);
if (!NT_SUCCESS(status))
{
KdPrintEx((DPFLTR_IHVDRIVER_ID, DPFLTR_INFO_LEVEL," *** WdfDeviceSetAlignmentRequirement failed with status 0x%08X\n", status));
return status;
}
size_t alignment = WdfDeviceGetAlignmentRequirement(device);
KdPrintEx((DPFLTR_IHVDRIVER_ID, DPFLTR_INFO_LEVEL,“AllocatePCIBuffer2 size = 0x%08X (%lu); alignment = 0x%08X (%lu); status 0x%08X\n”, size, size, alignment, alignment, status));
WdfDmaEnablerCreate(device, &dmaConfig, WDF_NO_OBJECT_ATTRIBUTES, &enabler);
if (!NT_SUCCESS(status))
{
KdPrintEx((DPFLTR_IHVDRIVER_ID, DPFLTR_INFO_LEVEL," *** WdfDmaEnablerCreate failed with status 0x%08X\n", status));
return status;
}
// Allocate the buffer (2X what is needed)
WdfCommonBufferCreate(enabler, size*2, WDF_NO_OBJECT_ATTRIBUTES, buffer);
if (!NT_SUCCESS(status))
{
KdPrintEx((DPFLTR_IHVDRIVER_ID, DPFLTR_INFO_LEVEL," *** WdfCommonBufferCreate failed with status 0x%08X\n", status));
return status;
}
size_t bufferLength = WdfCommonBufferGetLength(*buffer);
KdPrintEx((DPFLTR_IHVDRIVER_ID, DPFLTR_INFO_LEVEL,“bufferLength = 0x%08X (%lu)\n”, bufferLength, bufferLength));
*virtualAddress = WdfCommonBufferGetAlignedVirtualAddress(*buffer);
*physicalAddress = WdfCommonBufferGetAlignedLogicalAddress(*buffer);
// Align physical address
size_t mask = size - 1;
UINT32 qp = (UINT32)physicalAddress->u.LowPart;
void* base = (void*)qp;
void* alignedPhysicalAddress = (void*)(((uintptr_t)base+mask) & ~(uintptr_t)mask);
physicalAddress->QuadPart = (UINT32) alignedPhysicalAddress;
KdPrintEx((DPFLTR_IHVDRIVER_ID, DPFLTR_INFO_LEVEL,“Base physical address = 0x%08X; aligned physical address = 0x%08X\n”, base, alignedPhysicalAddress));
// Align virtual address
size_t baseOffset = (uintptr_t)alignedPhysicalAddress - (uintptr_t)base;
char* alignedVirtualAddress = (char*)virtualAddress + baseOffset;
KdPrintEx((DPFLTR_IHVDRIVER_ID, DPFLTR_INFO_LEVEL,“Base virtual address = 0x%08X; aligned virtual address = 0x%08X\n”, virtualAddress, alignedVirtualAddress));
*virtualAddress = (PVOID)alignedVirtualAddress;
KdPrintEx((DPFLTR_IHVDRIVER_ID, DPFLTR_INFO_LEVEL," Leaving AllocatePCIBuffer2…\n"));
return status;
}
// ======================================================================
// WinDbg outout
// ======================================================================
->EventDeviceAdd WdfDeviceCreate successful AddDevice PDO (0x89F21030) FDO (0xB316B9E8), pDeviceExt (0x89E1E638)->RegisterCallbacks Leaving RegisterCallbacks…
->AllocatePCIBuffer2
AllocatePCIBuffer2 size = 0x02000000 (33554432); alignment = 0x01FFFFFF (33554431); status 0x00000000
bufferLength = 0x04000000 (67108864)
Base physical address = 0xCD60C000; aligned physical address = 0xCE000000
Base virtual address = 0xB264408C; aligned virtual address = 0xB303808C
Leaving AllocatePCIBuffer2…
->AllocatePCIBuffer2
AllocatePCIBuffer2 size = 0x01000000 (16777216); alignment = 0x00FFFFFF (16777215); status 0x00000000
bufferLength = 0x02000000 (33554432)
Base physical address = 0xD2846000; aligned physical address = 0xD3000000
Base virtual address = 0xB2644074; aligned virtual address = 0xB2DFE074
Leaving AllocatePCIBuffer2…