I’m new to WDF driver programming, but I’ve been learning quite a bit from this site. Hopefully, someone can point me in the right direction. I have an IOCTL based DMA driver that I wrote for interfacing with PCIe FPGAs. It works well and is reliable until I add support for IO cancellation. Once I add the WdfRequestMarkCancelableEx, WdfRequestUnmarkCancelable calls I get the BSOD .
The driver code is rather long, but here’s what I think is relevant. The entry point is RiffaEvtIoDeviceControl. This starts the IO request for say the send direction, by calling RiffaIoctlSend. Both are listed below. I’ve commented out the WdfRequestMarkCancelableEx calls in the RiffaIoctlSend because it’s causing the BSOD. The EvtRequestCancel callback (RiffaEvtRequestCancel) is also listed below. After the send operation completes, the device causes an interrupt which eventually causes the RiffaCompleteRequest function (listed last) to be called. This calls WdfRequestCompleteWithInformation. You can see that WdfRequestUnmarkCancelable has been commented out there. I’m following the pattern on the MSDN WdfRequestUnmarkCancelable page. So I cannot figure out what’s going on. To make things worse, this works most of the time. It’s only when I run numerous back to back calls through this “send” path that I can get it to BSOD. The dump from the BSOD is not terribly useful to me. I’ve appended that last.
I’ll note that I’ve already double checked that I’m not putting the EvtRequestHandle callback in a PAGEable area. I suspect that the problem lies in either marking or unmarking, but the dump doesn’t help me narrow it down. Any direction or ideas would be greatly appreciated.
Thanks
Matt
=============================
/**
* This event is called when the framework receives IRP_MJ_DEVICE_CONTROL
* requests from the system.
* .
* Queue - Handle to the framework queue object that is associated with the
* I/O request.
*
* Request - Handle to a framework request object.
*
* OutputBufferLength - Length of the request’s output buffer,
* if an output buffer is available.
*
* InputBufferLength - Length of the request’s input buffer,
* if an input buffer is available.
*
* IoControlCode - the driver-defined or system-defined I/O control code
* (IOCTL) that is associated with the request.
*/
VOID RiffaEvtIoDeviceControl(IN WDFQUEUE Queue, IN WDFREQUEST Request,
IN size_t OutputBufferLength, IN size_t InputBufferLength,
IN ULONG IoControlCode) {
PDEVICE_EXTENSION devExt;
devExt = RiffaGetDeviceContext(WdfIoQueueGetDevice(Queue));
// Determine which I/O control code was specified.
switch (IoControlCode) {
case IOCTL_RIFFA_SEND: // (METHOD_OUT_DIRECT)
RiffaIoctlSend(devExt, Request, OutputBufferLength, InputBufferLength);
break;
case IOCTL_RIFFA_RECV: // (METHOD_OUT_DIRECT)
RiffaIoctlRecv(devExt, Request, OutputBufferLength, InputBufferLength);
break;
case IOCTL_RIFFA_LIST: // (METHOD_OUT_DIRECT)
RiffaIoctlList(devExt, Request, OutputBufferLength, InputBufferLength);
break;
case IOCTL_RIFFA_RESET: // (METHOD_OUT_DIRECT)
RiffaIoctlReset(devExt, Request);
break;
default:
// The specified I/O control code is unrecognized by this driver.
DbgPrintEx(DPFLTR_IHVDRIVER_ID, DPFLTR_ERROR_LEVEL,
“riffa: fpga:%s, invalid ioctl request type\n”, devExt->Name);
WdfRequestCompleteWithInformation(Request, STATUS_INVALID_DEVICE_REQUEST, 0);
break;
}
}
/**
* Handles IRP_MJ_DEVICE_CONTROL requests for IOCTL_RIFFA_SEND.
* .
* DevExt - Handle to the device extension object.
*
* Request - Handle to a framework request object.
*
* OutputBufferLength - Length of the request’s output buffer,
* if an output buffer is available.
*
* InputBufferLength - Length of the request’s input buffer,
* if an input buffer is available.
*/
VOID RiffaIoctlSend(IN PDEVICE_EXTENSION DevExt, IN WDFREQUEST Request,
IN size_t OutputBufferLength, IN size_t InputBufferLength) {
NTSTATUS status = STATUS_SUCCESS;
PREQUEST_EXTENSION reqExt;
PRIFFA_FPGA_CHNL_IO io;
UINT64 length;
PCHAR buf = NULL;
size_t bufSize;
// Input should be non-zero
if(!InputBufferLength) {
DbgPrintEx(DPFLTR_IHVDRIVER_ID, DPFLTR_ERROR_LEVEL,
“riffa: fpga:%s, ioctl send zero length input buffer\n”, DevExt->Name);
WdfRequestCompleteWithInformation(Request, STATUS_INVALID_PARAMETER, 0);
return;
}
// Get the input buffer
status = WdfRequestRetrieveInputBuffer(Request, 0, &buf, &bufSize);
if (!NT_SUCCESS(status)) {
DbgPrintEx(DPFLTR_IHVDRIVER_ID, DPFLTR_ERROR_LEVEL,
“riffa: fpga:%s, ioctl send WdfRequestRetrieveInputBuffer failed\n”,
DevExt->Name);
WdfRequestCompleteWithInformation(Request, status, 0);
return;
}
if (bufSize < sizeof(RIFFA_FPGA_CHNL_IO)) {
DbgPrintEx(DPFLTR_IHVDRIVER_ID, DPFLTR_ERROR_LEVEL,
“riffa: fpga:%s, ioctl send input buffer too small to contain RIFFA_FPGA_CHNL_IO\n”,
DevExt->Name);
WdfRequestCompleteWithInformation(Request, STATUS_INVALID_PARAMETER, 0);
return;
}
io = (PRIFFA_FPGA_CHNL_IO)buf;
// Validate the length, last, chnl
length = (io->Length < (OutputBufferLength>>2) ? io->Length : (OutputBufferLength>>2));
if (io->Last > 1) {
DbgPrintEx(DPFLTR_IHVDRIVER_ID, DPFLTR_ERROR_LEVEL,
“riffa: fpga:%s, ioctl send invalid last: %d\n”, DevExt->Name, io->Last);
WdfRequestCompleteWithInformation(Request, STATUS_INVALID_PARAMETER, 0);
return;
}
if (io->Chnl >= DevExt->NumChnls) {
DbgPrintEx(DPFLTR_IHVDRIVER_ID, DPFLTR_ERROR_LEVEL,
“riffa: fpga:%s, ioctl send invalid channel: %d, device only has %d channel(s)\n”,
DevExt->Name, io->Chnl, DevExt->NumChnls);
WdfRequestCompleteWithInformation(Request, STATUS_INVALID_PARAMETER, 0);
return;
}
// Check that this isn’t an already running transaction
if (InterlockedExchange(&DevExt->Chnl[io->Chnl].InUse, 1) == 1) {
DbgPrintEx(DPFLTR_IHVDRIVER_ID, DPFLTR_ERROR_LEVEL,
“riffa: fpga:%s, ioctl send already in use on channel: %d\n”,
DevExt->Name, io->Chnl);
WdfRequestCompleteWithInformation(Request, STATUS_INVALID_PARAMETER, 0);
return;
}
// Set the channel number in the request context for RiffaEvtRequestCancel.
reqExt = RiffaGetRequestContext(Request);
reqExt->Chnl = io->Chnl;
// Start a send transaction.
if (length) {
// Start a DMA transaction for sending.
DevExt->Chnl[io->Chnl].Timeout = io->Timeout;
DevExt->Chnl[io->Chnl].Length = (length<<2);
DevExt->Chnl[io->Chnl].SpillAfter = (length<<2);
DevExt->Chnl[io->Chnl].Offset = (io->Offset<<2);
DevExt->Chnl[io->Chnl].Last = io->Last;
DevExt->Chnl[io->Chnl].Provided = 0;
DevExt->Chnl[io->Chnl].ProvidedPrev = 0;
DevExt->Chnl[io->Chnl].Confirmed = 0;
DevExt->Chnl[io->Chnl].ConfirmedPrev = 0;
DevExt->Chnl[io->Chnl].ActiveCount = 0;
DevExt->Chnl[io->Chnl].Cancel = 0;
DevExt->Chnl[io->Chnl].Request = Request;
InterlockedExchange(&DevExt->Chnl[io->Chnl].ReqdDone, 0);
status = RiffaStartDmaTransaction(DevExt, io->Chnl, (length<<2),
0, WdfDmaDirectionWriteToDevice);
if (!NT_SUCCESS(status)) {
DbgPrintEx(DPFLTR_IHVDRIVER_ID, DPFLTR_ERROR_LEVEL,
“riffa: fpga:%s, unable to start DMA transaction on channel: %d\n”,
DevExt->Name, io->Chnl);
WdfRequestCompleteWithInformation(Request, status, 0);
return;
}
//else {
// Mark the WDFREQUEST as cancellable.
//status = WdfRequestMarkCancelableEx(Request, RiffaEvtRequestCancel);
//if (!NT_SUCCESS(status)) {
// DbgPrintEx(DPFLTR_IHVDRIVER_ID, DPFLTR_ERROR_LEVEL,
// “riffa: fpga:%s, ioctl send WdfRequestMarkCancelableEx failed\n”,
// DevExt->Name);
// WdfRequestCompleteWithInformation(Request, status, 0);
//}
//}
}
else if (io->Last) {
// Program the device for zero length send (device RX) and complete
DbgPrintEx(DPFLTR_IHVDRIVER_ID, DPFLTR_ERROR_LEVEL,
“riffa: fpga:%s, ioctl send zero length with last set\n”, DevExt->Name);
RiffaProgramSend(DevExt, io->Chnl, (UINT32)length, io->Offset, io->Last);
WdfRequestCompleteWithInformation(Request, STATUS_SUCCESS, 0);
}
else {
// Invalid request, results in no send
DbgPrintEx(DPFLTR_IHVDRIVER_ID, DPFLTR_ERROR_LEVEL,
“riffa: fpga:%s, ioctl send invalid, no length or last\n”, DevExt->Name);
WdfRequestCompleteWithInformation(Request, STATUS_INVALID_PARAMETER, 0);
}
}
/**
* EvtRequestCancel handler for WDFREQUEST objects for IOCTL sends/receives.
* Called if the IO Manager or calling application needs to cancel the send.
* In practice this should only happen when the user application hangs and the
* user CTRL+C signals or Task Ends the application.
*
* Request - WDFREQUEST object from the IOCTL queue, representing the send
*/
VOID RiffaEvtRequestCancel(IN WDFREQUEST Request) {
PDEVICE_EXTENSION devExt;
PREQUEST_EXTENSION reqExt;
UINT32 chnl;
BOOLEAN canCancel;
devExt = RiffaGetDeviceContext(WdfIoQueueGetDevice(WdfRequestGetIoQueue(Request)));
reqExt = RiffaGetRequestContext(Request);
KdPrintEx((DPFLTR_IHVDRIVER_ID, DPFLTR_TRACE_LEVEL,
“riffa: fpga:%s chnl:%d, request being cancelled\n”, devExt->Name,
(reqExt->Chnl >= RIFFA_MAX_NUM_CHNLS ? reqExt->Chnl - RIFFA_MAX_NUM_CHNLS : reqExt->Chnl)));
// Acquire the channel lock
WdfSpinLockAcquire(devExt->Chnl[reqExt->Chnl].SpinLock);
// See if we can cancel right now.
canCancel = (devExt->Chnl[reqExt->Chnl].ActiveCount == 0);
if (canCancel) {
// NULL out the request so that no other threads use it
devExt->Chnl[reqExt->Chnl].Request = NULL;
}
else {
// Set the cancel flag so that the last active thread cancels for us.
devExt->Chnl[reqExt->Chnl].Cancel = 1;
}
// Release the channel lock
WdfSpinLockRelease(devExt->Chnl[reqExt->Chnl].SpinLock);
// Cancel the request
if (canCancel) {
InterlockedExchange(&devExt->Chnl[reqExt->Chnl].Ready, 0);
InterlockedExchange(&devExt->Chnl[reqExt->Chnl].InUse, 0);
devExt->Chnl[reqExt->Chnl].ActiveCount = 0;
devExt->Chnl[reqExt->Chnl].Cancel = 0;
WdfDmaTransactionRelease(devExt->Chnl[reqExt->Chnl].DmaTransaction);
WdfTimerStop(devExt->Chnl[reqExt->Chnl].Timer, FALSE);
WdfRequestCompleteWithInformation(Request, STATUS_CANCELLED,
(ULONG_PTR)(devExt->Chnl[reqExt->Chnl].ConfirmedPrev +
devExt->Chnl[reqExt->Chnl].Confirmed)>>2);
}
}
/**
* Called when the WDFREQUEST object for the specified channel should be
* completed with the specified status. The WDFREQUEST has been marked
* cancelable so it must first be unmarked cancelable. After completion, the
* pointer to the WDFREQUEST is NULL’d out to indicate that the WDFREQUEST is
* no longer valid.
*
* DevExt - Pointer to the Device Extension
*
* Chnl - Channel number on which the DMA is taking place
*
* Status - NTSTATUS to set for completion
*/
VOID RiffaCompleteRequest(IN PDEVICE_EXTENSION DevExt, IN UINT32 Chnl,
IN NTSTATUS Status) {
NTSTATUS status;
WDFREQUEST request;
if ((request = DevExt->Chnl[Chnl].Request) != NULL) {
// Try to complete the request
DbgPrintEx(DPFLTR_IHVDRIVER_ID, DPFLTR_ERROR_LEVEL,
“riffa: fpga:%s chnl:%d, completing request\n”,
DevExt->Name, (Chnl < RIFFA_MAX_NUM_CHNLS ? Chnl : Chnl - RIFFA_MAX_NUM_CHNLS));
//status = WdfRequestUnmarkCancelable(request);
//if (status != STATUS_CANCELLED) {
// Complete the request
DevExt->Chnl[Chnl].Request = NULL;
WdfRequestCompleteWithInformation(request, Status,
(ULONG_PTR)(DevExt->Chnl[Chnl].ConfirmedPrev +
DevExt->Chnl[Chnl].Confirmed)>>2);
//}
//else {
// DbgPrintEx(DPFLTR_IHVDRIVER_ID, DPFLTR_ERROR_LEVEL,
// “riffa: fpga:%s chnl:%d, request already cancelled\n”,
// DevExt->Name, (Chnl < RIFFA_MAX_NUM_CHNLS ? Chnl : Chnl - RIFFA_MAX_NUM_CHNLS));
//}
}
else {
DbgPrintEx(DPFLTR_IHVDRIVER_ID, DPFLTR_ERROR_LEVEL,
“riffa: fpga:%s chnl:%d, request already nulled out\n”,
DevExt->Name, (Chnl < RIFFA_MAX_NUM_CHNLS ? Chnl : Chnl - RIFFA_MAX_NUM_CHNLS));
}
}
=================================
Loading Dump File [F:\MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available
Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols;C:\WinDDK\7600.16385.1\src\general\riffa\driver\windows\sys\objchk_win7_amd64\amd64
Executable search path is: C:\WinDDK\7600.16385.1\src\general\riffa\driver\windows\sys\objchk_win7_amd64\amd64
Windows 7 Kernel Version 7601 (Service Pack 1) MP (12 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 7601.17514.amd64fre.win7sp1_rtm.101119-1850
Machine Name:
Kernel base = 0xfffff80002a11000 PsLoadedModuleList = 0xfffff800
02c56e90
Debug session time: Tue Feb 17 03:38:55.592 2015 (UTC - 8:00)
System Uptime: 0 days 0:04:00.436
Loading Kernel Symbols
…
…
…
Loading User Symbols
PEB is paged out (Peb.Ldr = 000007ff`fffd3018). Type “.hh dbgerr001” for details
Loading unloaded module list
…
The context is partially valid. Only x86 user-mode context is available.
The wow64exts extension must be loaded to access 32-bit state.
.load wow64exts will do this if you haven’t loaded it already.
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
Use !analyze -v to get detailed debugging information.
BugCheck D1, {8, 2, 1, fffff88000f5c670}
Page 7c2028 not present in the dump file. Type “.hh dbgerr004” for details
Probably caused by : Unknown_Image ( Wdf01000!FxIrpQueue::RemoveIrpFromQueueByContext+30 )
Followup: MachineOwner
16.6: kd:x86> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************
DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 0000000000000008, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000001, value 0 = read operation, 1 = write operation
Arg4: fffff88000f5c670, address which referenced memory
Debugging Details:
Page 7c2028 not present in the dump file. Type “.hh dbgerr004” for details
WRITE_ADDRESS: 0000000000000008
CURRENT_IRQL: 0
FAULTING_IP:
Wdf01000!FxIrpQueue::RemoveIrpFromQueueByContext+30
fffff880`00f5c670 48 dec eax
DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT
BUGCHECK_STR: 0xD1
LAST_CONTROL_TRANSFER: from 0000000000000000 to 0000000000000000
STACK_TEXT:
00000000 00000000 00000000 00000000 00000000 0x0
STACK_COMMAND: .bugcheck ; kb
FOLLOWUP_IP:
Wdf01000!FxIrpQueue::RemoveIrpFromQueueByContext+30
fffff880`00f5c670 48 dec eax
SYMBOL_NAME: Wdf01000!FxIrpQueue::RemoveIrpFromQueueByContext+30
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: Unknown_Module
IMAGE_NAME: Unknown_Image
DEBUG_FLR_IMAGE_TIMESTAMP: 0
BUCKET_ID: INVALID_KERNEL_CONTEXT
Followup: MachineOwner