Waiting for a queued work item to complete before driver unload

My driver allocates stuff from nonpaged pool. The DispatchClose routine does some stuff by queuing a DPC, and this queues a work item to do final stuff requiring PASSIVE_LEVEL, such as disconnecting interrupts and releasing the memory back to the pool.
If this is the last file handle for the driver that has just closed, the DriverUnload will be called.
Does the I/O manager check that any work items associated with the driver have been dequeued and have completed? Or does my
driver have to do that by having the work item post some event that the DriverUnload waits on?
Windows Internals 6th Ed says on pg 205

Device drivers should use only the latter [that is, IoQueueWorkItem] (because this associates the work item with a Device object, allowing for greater accountability and the handling of scenarios in which a driver unloads while its work item is active).
But there’s no details on how the kernel handles such scenarios. I’m hoping that it is done seamlessly.

To be honest, the whole post seems (at least to me) to be nonsensical from A to Z…

To begin with, disconnecting interrupts is totally unrelated to IRP_MJ_CLOSE, and is done either when the device is paused or removed ( in case of PnP driver) or when the driver is being unloaded (in case of a legacy one). In either case, queuing a DPC in context of an IRP that is guaranteed to be received at PASSIVE_LEVEL, although technically does not constitute a violation, seems at least strange.

Furthermore, closing the last handle to a device most certainly does not result in DriverUnload() invocation…

I suggest you should at least read MSDN and look at WDK samples before posting your questions here…

Anton Bassov

The kernel does not track your work items and run them down for you. If you use Io work items (not Ex work items), the kernel will keep an Ob reference on your device object until the work item runs and returns back to the kernel, which will keep your image loaded if driverunload happened previously. This will not maintain state for you though and the work item callback has to be implemented such that it can handle unload happening first. If it cannot, you must track when you have outstanding work and wait for it to finish during driverunload.

d

Bent from my phone


From: Michael_Rolle
Sent: Tuesday, February 5, 2019 12:12:31 AM
To: Doron Holan
Subject: [NTDEV] Waiting for a queued work item to complete before driver unload

OSR https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommunity.osr.com%2F&data=02|01|doron.holan%40microsoft.com|ab2be6ac9a624bfb4e5208d68b41ad93|72f988bf86f141af91ab2d7cd011db47|1|1|636849511537719548&sdata=DIdbQ%2BT%2BN7Fkrh9Xj1rr%2FZPaSRYP8eYW4fFGs7M4Jek%3D&reserved=0
Michael_Rolle started a new discussion: Waiting for a queued work item to complete before driver unload

My driver allocates stuff from nonpaged pool. The DispatchClose routine does some stuff by queuing a DPC, and this queues a work item to do final stuff requiring PASSIVE_LEVEL, such as disconnecting interrupts and releasing the memory back to the pool.

If this is the last file handle for the driver that has just closed, the DriverUnload will be called.

Does the I/O manager check that any work items associated with the driver have been dequeued and have completed? Or does my

driver have to do that by having the work item post some event that the DriverUnload waits on?

Windows Internals 6th Ed says on pg 205

> Device drivers should use only the latter [that is, IoQueueWorkItem] (because this associates the work item with a Device object, allowing for greater accountability and the handling of scenarios in which a driver unloads while its work item is active).

But there’s no details on how the kernel handles such scenarios. I’m hoping that it is done seamlessly.

One option is to use a remove lock. When your thread ebeters, have it to
acquire the remove lock. When the thread exits (i.e. your queue is empty),
have the thread to release the remove lock.

On Tue, Feb 5, 2019 at 10:57 AM Doron_Holan
wrote:

> OSR http://osr.vanillacommunities.com/
> Doron_Holan commented on Waiting for a queued work item to complete before
> driver unload
>
> The kernel does not track your work items and run them down for you. If
> you use Io work items (not Ex work items), the kernel will keep an Ob
> reference on your device object until the work item runs and returns back
> to the kernel, which will keep your image loaded if driverunload happened
> previously. This will not maintain state for you though and the work item
> callback has to be implemented such that it can handle unload happening
> first. If it cannot, you must track when you have outstanding work and wait
> for it to finish during driverunload.
>
> d
>
> Bent from my phone
> ________________________________
> From: Michael_Rolle
> Sent: Tuesday, February 5, 2019 12:12:31 AM
> To: Doron Holan
> Subject: [NTDEV] Waiting for a queued work item to complete before driver
> unload
>
> OSR
> https://nam06.safelinks.protection.outlook.com/?url=https://community.osr.com/&data=02|01|doron.holan@microsoft.com|ab2be6ac9a624bfb4e5208d68b41ad93|72f988bf86f141af91ab2d7cd011db47|1|1|636849511537719548&sdata=DIdbQ+T+N7Fkrh9Xj1rr/ZPaSRYP8eYW4fFGs7M4Jek=&reserved=0
> https:
> Michael_Rolle started a new discussion: Waiting for a queued work item to
> complete before driver unload
>
> My driver allocates stuff from nonpaged pool. The DispatchClose routine
> does some stuff by queuing a DPC, and this queues a work item to do final
> stuff requiring PASSIVE_LEVEL, such as disconnecting interrupts and
> releasing the memory back to the pool.
>
> If this is the last file handle for the driver that has just closed, the
> DriverUnload will be called.
>
> Does the I/O manager check that any work items associated with the driver
> have been dequeued and have completed? Or does my
>
> driver have to do that by having the work item post some event that the
> DriverUnload waits on?
>
> Windows Internals 6th Ed says on pg 205
>
> > Device drivers should use only the latter [that is, IoQueueWorkItem]
> (because this associates the work item with a Device object, allowing for
> greater accountability and the handling of scenarios in which a driver
> unloads while its work item is active).
>
> But there’s no details on how the kernel handles such scenarios. I’m
> hoping that it is done seamlessly.
>
> –
> Reply to this email directly or follow the link below to check it out:
> http://osr.vanillacommunities.com/discussion/comment/292450#Comment_292450
>
> Check it out:
> http://osr.vanillacommunities.com/discussion/comment/292450#Comment_292450
></https:>

@Doron_Holan said:
The kernel does not track your work items and run them down for you. If you use Io work items (not Ex work items), the kernel will keep an Ob reference on your device object until the work item runs and returns back to the kernel, which will keep your image loaded if driverunload happened previously. This will not maintain state for you though and the work item callback has to be implemented such that it can handle unload happening first. If it cannot, you must track when you have outstanding work and wait for it to finish during driverunload.

My driver has only one Device, which may be shared by many File handles. Each File may, from time to time, allocate some hardware resources exclusively on a particular CPU, and the Device keeps track of what CPU is being used by which File. The Device also allocates nonpaged memory to go along with the CPU allocation.
The process of releasing the CPU can happen by a specific request, or implicitly on closing the File. This process entails (1) a DPC targeted at the CPU to deal with releasing the hardware on that CPU, followed by (2) an IoWorkItem to do things that have to be at PASSIVE_LEVEL. Releasing of the pool allocation happens in (1) or preferably in (2). Concurrently, the File may get closed, and then the Driver may get unloaded by request made to the SCM.
If I understand you correctly, DispatchClose and DriverUnload are OK if they don’t depend on anything in (1) or (2) having completed. And (1) and (2) can depend on the Driver’s image and the Device object still being resident, due to Ob references held first by the DPC and then (which is queued before the DPC finishes) by the WorkItem.
Now DriverUnload will call IoDeleteDevice, so this may remove the Device from the Driver’s Device list immediately, but the memory occupied by the Device will stay around.

One question: I got a bugcheck from DriverVerifier once because I forgot to free an allocation from nonpaged pool made by the Device, before unloading the driver. Would I get this error at the time DriverUnload completes, or at the time the Driver image is unloaded. In the second case, I’m OK, but in the first case, I have to synchronize DriverUnload with all of the pool allocations.

Each File may, from time to time, allocate some hardware resources exclusively on a particular CPU

OMG…

I just forgot about your previous thread - you are the guy who tries to allocate his own interrupt vector and arbitrarily assign an IRQL to ISR that services it,right? I guess the other posters may want to check the following thread before trying to help you

https://community.osr.com/discussion/290986/why-is-my-interrupt-not-getting-serviced

Anton Bassov

Michael_Rolle wrote:

One question: I got a bugcheck from DriverVerifier once because I forgot to free an allocation from nonpaged pool made by the Device, before unloading the driver. Would I get this error at the time DriverUnload completes, or at the time the Driver image is unloaded. In the second case, I’m OK, but in the first case, I have to synchronize DriverUnload with all of the pool allocations.

This post has boggled me.  First, the driver image is unloaded as soon
as DriverUnload completes.  There is no time delay there.

Second, you’re saying you are OK with Driver Verifier bugchecks? Some
particularly suspicious users run with Driver Verifier on most of the
time.  You should NOT be releasing drivers that are known to cause a
bugcheck!

Non-paged pool is a limited resource.  If you are leaking non-paged
pool, then your driver is broken.  The reason Verifier bugchecks is
because, unlike user-mode process allocations, the system cannot clean
up non-paged pool.  If you don’t release them, the memory is lost.  Forever.

> This post has boggled me. First, the driver image is unloaded as soon as DriverUnload completes. There is no time delay there.

This is not true. If there is an outstanding Ob ref on a device or driver object, the image will unload when the Ob ref count drops to zero. Normal operation is that this happens after DriverUnload returns to the IO manager and the io manager drops its reference. Trust me, I had to fix these bugs in WDF and other drivers over the years, the problem is real and occurs with regularity.

d

The reason Verifier bugchecks is because, unlike user-mode process allocations, the system cannot clean
up non-paged pool. If you don’t release them, the memory is lost. Forever.

Actually, the same holds true for driver allocations from the paged pool as well - if you don’t free it it is going to be lost forever. Furthermore, assuming that we are speaking about the user-mode process never terminates (csrss.exe is the very first example that gets inti my head), the same may hold true for the userland as well…

First, the driver image is unloaded as soon as DriverUnload completes. There is no time delay there.

Consider the scenario when DEVICE_OBJECT’s refcount is not zero by the time DriverUnload completes. Although the device object may have been deleted, it is still going to be physically resident in RAM while its refcount is non-zero. As long as some DEVICE_OBJECT’s refcount is not zero, its DRIVER_OBJECT’s refcount is going to be non-zero as well, which means driver image will still be resident in RAM. Certainly,in practical terms, it may be already “not-so-functional” if a clean-up has been done by DriverUnload(), but it is still going to be resident in RAM until DRIVER_OBJECT’s refcount goes down to zero…

Anton Bassov