Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Home NTDEV
Before Posting...
Please check out the Community Guidelines in the Announcements and Administration Category.

More Info on Driver Writing and Debugging


The free OSR Learning Library has more than 50 articles on a wide variety of topics about writing and debugging device drivers and Minifilters. From introductory level to advanced. All the articles have been recently reviewed and updated, and are written using the clear and definitive style you've come to expect from OSR over the years.


Check out The OSR Learning Library at: https://www.osr.com/osr-learning-library/


Deleting Control Devices before the FDO

PlusquamPlusquam Member Posts: 2

Hi, I am debugging a KMDF-based PNP driver that creates dozens of Control Devices (CDOs) per FDO for legacy compatibility reasons. The driver creates the CDOs in EvtDevicePrepareHardware with WdfDeviceCreate and deletes them in EvtDeviceReleaseHardware with WdfObjectDelete. Apparently, WDF waits forever for the state machine of the m_DefaultQueueForCreates if a IRP_MJ_CREATE is completed on a CDO that is about to be deleted.

https://docs.microsoft.com/en-us/windows-hardware/drivers/wdf/using-control-device-objects:

If your driver creates control device objects (which do not support PnP or power management), and if the driver also creates framework device objects that support PnP and power management, the driver must eventually call WdfObjectDelete at IRQL = PASSIVE_LEVEL to delete the control device objects.

If the driver creates both types of device objects, the operating system cannot unload your driver until the driver has deleted the control device objects.

However, the driver must not delete the control device objects until after the framework has deleted the other device objects. To determine when the framework has deleted the other device objects, your driver should provide EvtCleanupCallback functions for those objects.

The last paragraph irritates me: to my understanding the paragraph means that the only valid point to delete the CDOs is the EvtCleanupCallback of the FDO (as demonstrated in the toastDrv sideband sample), because (1) the CDO must outlive the FDO and (2) the PNP manager will not unload the driver if DOs remain after the FDO is cleaned up, as discussed in https://community.osr.com/discussion/128797/control-device-inhibits-unload.

My current understanding contradicts (1): CDOs and FDOs are unrelated and therefore have independent lifetimes, so the driver may delete CDOs at any time. Is this correct or does the paragraph hint at a restriction that I am not aware of? In case it is relevant: the FDO and each CDO have their own WdfIoQueue and all Ioctls are completed in the IrpPreprocessCallback.

Thanks for helping.

Comments

  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 7,913

    Ah! You have entered that space which is the cross-product of PnP, NON-PnP, and KMDF. Things get... tricky.

    There is a positively wonderful (if ancient) thread on this topic that will explain to you what's going on. See it here.

    Peter

    Peter Viscarola
    OSR
    @OSRDrivers

  • PlusquamPlusquam Member Posts: 2

    I linked to that thread in the original post. The primary topic of the thread is that PNP drivers will not unload if the last open handle is on a CDO. This is a known and accepted issue of the driver that I work on. I am more concerned with the documentation. Is the statement “However, the driver must not delete the control device objects until after the framework has deleted the other device objects.” correct? If this is actually a requirement of WDF, I would at least consider moving the deletes of the CDOs from EvtDeviceReleaseHardware to EvtCleanupCallback, although deleting in EvtDeviceReleaseHardware apparently worked fine for many years. The only related information in the linked thread is that Doron_Holan states “The correct pattern is to delete in the last fdo's cleanup”.


    Anyhow, the problem I have does not appear to be related with the time when the CDOs are deleted. Instead WDF has (in my opinion) a bug that it blocks forever if the last reference on a device is released by a WdfRequestComplete in the EvtDeviceFileCreate callback of a CDO. The result is not just that the driver does not unload, but that a process, or in the case I have even another kernel service, hangs forever in the EvtFileCreate. Consequently, the system does not shutdown anymore and with Driver Verifier enabled, bluescreens. In that case I get the following callstack:

    nt!DbgBreakPoint
    Wdf01000!_FX_DRIVER_GLOBALS::WaitForSignal+0xc6 [minkernel\wdf\framework\shared\object\globals.cpp @ 1972]
    Wdf01000!FxIoQueue::Dispose+0xdb [minkernel\wdf\framework\shared\irphandlers\io\fxioqueue.cpp @ 552]
    Wdf01000!FxObject::DisposeChildrenWorker+0x1e68e [minkernel\wdf\framework\shared\object\fxobjectstatemachine.cpp @ 1212]
    Wdf01000!FxObject::PerformDisposingDisposeChildrenLocked+0x35 [minkernel\wdf\framework\shared\object\fxobjectstatemachine.cpp @ 846]
    Wdf01000!FxObject::PerformEarlyDisposeWorkerAndUnlock+0x6a [minkernel\wdf\framework\shared\object\fxobjectstatemachine.cpp @ 926]
    Wdf01000!FxObject::PerformEarlyDispose+0x8d [minkernel\wdf\framework\shared\object\fxobjectstatemachine.cpp @ 592]
    Wdf01000!FxObject::DisposeChildrenWorker+0x1e629 [minkernel\wdf\framework\shared\object\fxobjectstatemachine.cpp @ 1191]
    Wdf01000!FxObject::DeleteObject+0x26d [minkernel\wdf\framework\shared\object\fxobjectstatemachine.cpp @ 124]
    Wdf01000!FxPkgGeneral::DecrementOpenHandleCount+0x23 [minkernel\wdf\framework\shared\irphandlers\general\fxpkggeneral.cpp @ 1983]
    Wdf01000!FxRequest::CompleteInternal+0x1e855 [minkernel\wdf\framework\shared\core\fxrequest.cpp @ 862]
    Wdf01000!imp_WdfRequestComplete+0x8b [minkernel\wdf\framework\shared\core\fxrequestapi.cpp @ 436]
    cdoBug!WdfRequestComplete+0x45 [C:\Program Files (x86)\Windows Kits\10\Include\wdf\kmdf\1.15\wdfrequest.h @ 996]
    cdoBug!cdoBugEvtFileCreate+0x22 [cdoBug.c @ 14]
    Wdf01000!FxPkgGeneral::OnCreate+0xa68 [minkernel\wdf\framework\shared\irphandlers\general\fxpkggeneral.cpp @ 1302]
    Wdf01000!FxPkgGeneral::Dispatch+0x44c [minkernel\wdf\framework\shared\irphandlers\general\fxpkggeneral.cpp @ 785]
    Wdf01000!FxDevice::DispatchWithLock+0x112 [minkernel\wdf\framework\shared\core\fxdevice.cpp @ 1430]
    VerifierExt!xdv_IRP_MJ_CREATE_wrapper+0x99

    Relevant member variables of FxIoQueue are:
    m_Deleted = 0, m_Disposing = 1, m_Dispatching = 0, m_DriverIoCount = 1, m_Queue.m_RequestCount = 0

    To my understanding the bug is that FxIoQueue::DispatchEvents is called with m_DriverIoCount = 1, resulting in a totalIoCount=1 that prevents the FxIoQueue::DispatchEvents from setting the m_FinishDisposing event.


    For reference, the driver written to trigger the issue is:

    typedef struct {
    PKTHREAD thread;
    volatile BOOLEAN threadShallRun;
    } DEVICE_CTX;
    WDF_DECLARE_CONTEXT_TYPE_WITH_NAME(DEVICE_CTX, GetDeviceCtx);
    void cdoBugEvtFileCreate(WDFDEVICE d, WDFREQUEST r, WDFFILEOBJECT fo) {
    WdfRequestComplete(r, STATUS_UNSUCCESSFUL);
    }
    void cdoBugThreadRoutine(void *ctxVoid) {
    DECLARE_CONST_UNICODE_STRING(ntDevName, L"\Device\cdoBug");
    DEVICE_CTX *ctx = ctxVoid;
    while (ctx->threadShallRun) {
    PWDFDEVICE_INIT devInit;
    devInit = WdfControlDeviceInitAllocate( WdfGetDriver(), &SDDL_DEVOBJ_SYS_ALL_ADM_RWX_WORLD_RWX_RES_RWX);
    if (!devInit) {
    return;
    }
    NTSTATUS status = WdfDeviceInitAssignName(devInit, &ntDevName);
    if (!NT_SUCCESS(status)) {
    WdfDeviceInitFree(devInit);
    return;
    }
    WDF_FILEOBJECT_CONFIG foc;
    WDF_FILEOBJECT_CONFIG_INIT(&foc, cdoBugEvtFileCreate, NULL, NULL);
    WdfDeviceInitSetFileObjectConfig(devInit, &foc, WDF_NO_OBJECT_ATTRIBUTES);
    WDFDEVICE cdo;
    status = WdfDeviceCreate(&devInit, WDF_NO_OBJECT_ATTRIBUTES, &cdo);
    if (!NT_SUCCESS(status)) {
    WdfDeviceInitFree(devInit);
    return;
    }
    WdfControlFinishInitializing(cdo);
    LARGE_INTEGER li;
    li.QuadPart = WDF_REL_TIMEOUT_IN_MS(10);
    KeDelayExecutionThread(Executive, FALSE, &li);
    WdfObjectDelete(cdo);
    }
    }
    NTSTATUS cdoBugEvtD0Entry(WDFDEVICE device, WDF_POWER_DEVICE_STATE pds) {
    DEVICE_CTX *ctx = GetDeviceCtx(device);
    ctx->threadShallRun = TRUE;
    HANDLE hThread = 0;
    OBJECT_ATTRIBUTES objAttr;
    InitializeObjectAttributes(&objAttr, NULL, OBJ_KERNEL_HANDLE, NULL, NULL);
    NTSTATUS status = PsCreateSystemThread(&hThread, THREAD_ALL_ACCESS, &objAttr, 0, NULL, cdoBugThreadRoutine, ctx);
    if (NT_SUCCESS(status)) {
    status = ObReferenceObjectByHandle(hThread, SYNCHRONIZE, NULL, KernelMode, &ctx->thread, NULL);
    ZwClose(hThread);
    }
    return status;
    }
    NTSTATUS cdoBugEvtD0Exit(WDFDEVICE device, WDF_POWER_DEVICE_STATE pds) {
    DEVICE_CTX *ctx = GetDeviceCtx(device);
    ctx->threadShallRun = FALSE;
    KeWaitForSingleObject(ctx->thread, Executive, KernelMode, FALSE, NULL);
    ctx->thread = NULL;
    return STATUS_SUCCESS;
    }
    NTSTATUS cdoBugEvtDeviceAdd(WDFDRIVER d, PWDFDEVICE_INIT deviceInit) {
    WDF_OBJECT_ATTRIBUTES attributes;
    WDF_OBJECT_ATTRIBUTES_INIT_CONTEXT_TYPE(&attributes, DEVICE_CTX);
    WDF_PNPPOWER_EVENT_CALLBACKS callbacks;
    WDF_PNPPOWER_EVENT_CALLBACKS_INIT(&callbacks);
    callbacks.EvtDeviceD0Entry = cdoBugEvtD0Entry;
    callbacks.EvtDeviceD0Exit = cdoBugEvtD0Exit;
    WdfDeviceInitSetPnpPowerEventCallbacks(deviceInit, &callbacks);
    WDFDEVICE device;
    return WdfDeviceCreate(&deviceInit, &attributes, &device);
    }
    NTSTATUS DriverEntry(PDRIVER_OBJECT driverObject,
    PUNICODE_STRING registryPath) {
    WDF_DRIVER_CONFIG config;
    WDF_DRIVER_CONFIG_INIT(&config, cdoBugEvtDeviceAdd);
    return WdfDriverCreate(driverObject, registryPath, WDF_NO_OBJECT_ATTRIBUTES, &config, WDF_NO_HANDLE);
    }

    and a single userspace process calls CreateFile(L"\\\\.\\globalroot\\Device\\cdoBug", GENERIC_READ, 0, NULL, OPEN_EXISTING, 0, NULL); in an infinite loop.

Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Upcoming OSR Seminars
OSR has suspended in-person seminars due to the Covid-19 outbreak. But, don't miss your training! Attend via the internet instead!
Kernel Debugging 30 Mar 2020 OSR Seminar Space
Developing Minifilters 15 Jun 2020 LIVE ONLINE
Writing WDF Drivers 22 June 2020 LIVE ONLINE
Internals & Software Drivers 28 Sept 2020 Dulles, VA