IRP cancel routine and MJ_CLEANUP

NTDEV
1

Not familiar with these, my question might be a little bit confusion.
Basically in the following case, when are IRP cancel routine and MJ_CLEANUP get called ?

i have an app, the process creates the 2nd thread, the main thread is suspended.
After some time, the 2nd thread finishes its work and calls ExitProcess.

So how many times does MJ_CLEANUP get called ? i have 2 threads and 1 process, so 3 times ?

Can i say since the 2nd thread calls ExitProcess, it’s forcing the suspended main thread to close, so IRP cancel routine gets called at that time ?

Both IRP cancel routine and MJ_CLEANUP will call the same MyFinalCleanup(). In my case, there’re synchronization issues. Can i use SpinLock to protect MyFinalCleanup() ?

Thanks very much !

2

These are quite different concepts. The IRP_MJ_CLEANUP is called when a
file objects handle count drops to zero, so the typical case is when all
handles to a device in a given process are closed. The IRP cancel code
is called whenever an IRP is canceled, note the user space CancelIo call
will cancel the IRP without the close or termination of the process.

Your cleanup for the two should also be different, the IRP cancel code
should only handle items related to cleaning up the processing of the
given IRP. The IRP_MJ_CLEANUP code should handle the cleanup associated
with the device being closed for a given file object.

Assuming you have IRP’s pending at the time of exit of the process you
will get a cleanup call for each IRP then a IRP_MJ_CANCEL call. If
there are no IRP’s pending only the IRP_MJ_CANCEL call will occur.
Design your code appropriately.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

“xxxxx@hotmail.com” wrote in message
news:xxxxx@ntdev:

> Not familiar with these, my question might be a little bit confusion.
> Basically in the following case, when are IRP cancel routine and MJ_CLEANUP get called ?
>
> i have an app, the process creates the 2nd thread, the main thread is suspended.
> After some time, the 2nd thread finishes its work and calls ExitProcess.
>
> So how many times does MJ_CLEANUP get called ? i have 2 threads and 1 process, so 3 times ?
>
> Can i say since the 2nd thread calls ExitProcess, it’s forcing the suspended main thread to close, so IRP cancel routine gets called at that time ?
>
> Both IRP cancel routine and MJ_CLEANUP will call the same MyFinalCleanup(). In my case, there’re synchronization issues. Can i use SpinLock to protect MyFinalCleanup() ?
>
> Thanks very much !

3

Look at the WDK documentation for IRP_MJ_CLEANUP.

When the handle reference count reaches 0 IRP_MJ_CLEANUP will be called
exactly once for the handle.

If a thread makes the final CloseHandle call from your application, then
IRP_MJ_CLEANUP will be called then. If the handle is never explicitly closed
(bad idea…), then the system will close the handle during ExitProcess.

Thomas F. Divine
http://www.pcausa.com

From:
Sent: Sunday, October 24, 2010 10:47 AM
To: “Windows System Software Devs Interest List”
Subject: [ntdev] IRP cancel routine and MJ_CLEANUP

> Not familiar with these, my question might be a little bit confusion.
> Basically in the following case, when are IRP cancel routine and
> MJ_CLEANUP get called ?
>
> i have an app, the process creates the 2nd thread, the main thread is
> suspended.
> After some time, the 2nd thread finishes its work and calls ExitProcess.
>
> So how many times does MJ_CLEANUP get called ? i have 2 threads and 1
> process, so 3 times ?
>
> Can i say since the 2nd thread calls ExitProcess, it’s forcing the
> suspended main thread to close, so IRP cancel routine gets called at that
> time ?
>
> Both IRP cancel routine and MJ_CLEANUP will call the same
> MyFinalCleanup(). In my case, there’re synchronization issues. Can i use
> SpinLock to protect MyFinalCleanup() ?
>
> Thanks very much !
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

4

And when a thread exits, all io it has sent if still pending will be canceled.

d

dent from a phpne with no keynoard

-----Original Message-----
From: Thomas F. Divine
Sent: October 24, 2010 8:16 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] IRP cancel routine and MJ_CLEANUP

Look at the WDK documentation for IRP_MJ_CLEANUP.

When the handle reference count reaches 0 IRP_MJ_CLEANUP will be called
exactly once for the handle.

If a thread makes the final CloseHandle call from your application, then
IRP_MJ_CLEANUP will be called then. If the handle is never explicitly closed
(bad idea…), then the system will close the handle during ExitProcess.

Thomas F. Divine
http://www.pcausa.com

--------------------------------------------------
From:
Sent: Sunday, October 24, 2010 10:47 AM
To: “Windows System Software Devs Interest List”
Subject: [ntdev] IRP cancel routine and MJ_CLEANUP

> Not familiar with these, my question might be a little bit confusion.
> Basically in the following case, when are IRP cancel routine and
> MJ_CLEANUP get called ?
>
> i have an app, the process creates the 2nd thread, the main thread is
> suspended.
> After some time, the 2nd thread finishes its work and calls ExitProcess.
>
> So how many times does MJ_CLEANUP get called ? i have 2 threads and 1
> process, so 3 times ?
>
> Can i say since the 2nd thread calls ExitProcess, it’s forcing the
> suspended main thread to close, so IRP cancel routine gets called at that
> time ?
>
> Both IRP cancel routine and MJ_CLEANUP will call the same
> MyFinalCleanup(). In my case, there’re synchronization issues. Can i use
> SpinLock to protect MyFinalCleanup() ?
>
> Thanks very much !
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

5

Reading the WDK topic “Handling IRPs” will probably help you.

Thomas F. Divine
http://www.pcausa.com

From: “Don Burn”
Sent: Sunday, October 24, 2010 11:01 AM
Newsgroups: ntdev
To: “Windows System Software Devs Interest List”
Subject: Re:[ntdev] IRP cancel routine and MJ_CLEANUP

> These are quite different concepts. The IRP_MJ_CLEANUP is called when a
> file objects handle count drops to zero, so the typical case is when all
> handles to a device in a given process are closed. The IRP cancel code is
> called whenever an IRP is canceled, note the user space CancelIo call will
> cancel the IRP without the close or termination of the process.
>
> Your cleanup for the two should also be different, the IRP cancel code
> should only handle items related to cleaning up the processing of the
> given IRP. The IRP_MJ_CLEANUP code should handle the cleanup associated
> with the device being closed for a given file object.
>
> Assuming you have IRP’s pending at the time of exit of the process you
> will get a cleanup call for each IRP then a IRP_MJ_CANCEL call. If there
> are no IRP’s pending only the IRP_MJ_CANCEL call will occur. Design your
> code appropriately.
>
>
> Don Burn (MVP, Windows DKD)
> Windows Filesystem and Driver Consulting
> Website: http://www.windrvr.com
> Blog: http://msmvps.com/blogs/WinDrvr
>
>
>
> “xxxxx@hotmail.com” wrote in message
> news:xxxxx@ntdev:
>
>> Not familiar with these, my question might be a little bit confusion.
>> Basically in the following case, when are IRP cancel routine and
>> MJ_CLEANUP get called ?
>>
>> i have an app, the process creates the 2nd thread, the main thread is
>> suspended.
>> After some time, the 2nd thread finishes its work and calls ExitProcess.
>>
>> So how many times does MJ_CLEANUP get called ? i have 2 threads and 1
>> process, so 3 times ?
>>
>> Can i say since the 2nd thread calls ExitProcess, it’s forcing the
>> suspended main thread to close, so IRP cancel routine gets called at that
>> time ?
>>
>> Both IRP cancel routine and MJ_CLEANUP will call the same
>> MyFinalCleanup(). In my case, there’re synchronization issues. Can i use
>> SpinLock to protect MyFinalCleanup() ?
>>
>> Thanks very much !
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

6

> So how many times does MJ_CLEANUP get called ? i have 2 threads and 1 process, so 3 times ?

Once per handle.

Both IRP cancel routine and MJ_CLEANUP will call the same MyFinalCleanup().

Why IRP cancel routine is calling MJ_CLEANUP handler? it must only detach the IRP from any lists/contexts and complete it, nothing else.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

7

> Assuming you have IRP’s pending at the time of exit of the process you

will get a cleanup call for each IRP

Sorry, not so. You mixed “cancel” and “cleanup”.

The correct is - one cancel call for each IRP, then a MJ_CLEANUP call.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

8

> After some time, the 2nd thread finishes its work and calls ExitProcess.

Note that internally ExitProcess is just a) calling the DllMain routines and b) TerminateProcess.

The latter call enters the kernel and sends the “exit special APC” to all threads in the process.

After receiving this APC, the thread will exit. This “receive the APC” situation occurs:

a) for a thread running in user mode - immediately
b) for a thread running in kernel mode - on next return to user mode
c) for a thread waiting for a non-overlapped IO operation (inside Read/WriteFile/DeviceIoControl) - the thread is awaken by this special APC, and goes on trying to cancel the request. If cancellation fails, then the thread waits for the IRP to complete its normal way with a huge timeout (minute or so). In case of timeout, the hard error is raised, with event logging and baloon/message box UI.
d) for a thread waiting in the driver with KernelMode/Alertable=FALSE, the APC is not delivered till the wait will be satisfied.

Then, when the thread receives the APC, its exit procedure starts.

The exit procedure tries to cancel all overlapped IO IRPs still pending off this thread, then waits for them for some large time (minute or so), then, when no IRPs are left - the thread is really destroyed. On timeout, the same occurs but with a dialog box/maybe event logging.

The process is actually killed (with MJ_CLEANUP sent to all handles) when the last thread in it exits.

So note that, if you have a thread blocked by KeWaitForSingleObject in kernel mode, this can really make an “unkillable” process which will not die till this wait will be satisfied. Avoid doing such things, pend the IRP instead and provide the cancel routine for it.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

9

Much good, specific, advice had been given to the OP. Let’s step back a bit and give him a bit of more general advice.

It’s not typically necessary – and in fact it is not typically desirable – for most drivers to implement processing routines to handle IRP_MJ_CLEANUP. There are few, and very specific, cases when an IRP_MJ_CLEANUP handler is needed; Unless your driver falls into one of those categories you should not implement cleanup.

Anytime your driver keeps IRPs in progress (either waiting on a queue or active on a device) for a long time or indeterminate amount of time, then you have to implement cancel processing. If you must have a guideline, let’s say “long” here is… oh… 3 or 4 seconds (that number is entirely arbitrary and not tied in any way to a limitation required for proper system operation).

Cancel is one of the two most bug prone areas of Windows driver development. Unless your driver have very unique needs, you *definitely* want to use Cancel Safe Queues – see the IoCsqXxxx functions in the WDK. I strongly urge you not to try to invent your own cancel processing outside of the Cancel Safe Queue constructs provided by Windows. There are myriad ways to implement cancel… most are wrong, and the ones that are correct then to be extraordinarily involved and subtle.

Hope that helps,

Peter
OSR

10

>have very unique needs, you *definitely* want to use Cancel Safe Queues – see the IoCsqXxxx

functions in the WDK.

KMDF queues are also OK.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

11

Yes, sorry… I’ve been thinking WDM lately.

The IoCsqXxxx equivalent in KMDF is, ah, any KMDF queue.

Peter
OSR

12

Thanks for everyone’s reply, make more sense now.

Our case is very complicated, there’s a reason why IRP cancel and MJ_CKLEANUP both call the same routine eventually. And this is where the bug gets introduced. So i need to read more …

13

xxxxx@osr.com wrote:

…
It’s not typically necessary – and in fact it is not typically desirable – for most drivers to implement processing routines to handle IRP_MJ_CLEANUP. There are few, and very specific, cases when an IRP_MJ_CLEANUP handler is needed; Unless your driver falls into one of those categories you should not implement cleanup.

I find this interesting, and I’d like to explore this a bit. In my
experience, any driver that needs to carry per-instance state must
implement both IRP_MJ_CREATE and either IRP_MJ_CLOSE or IRP_MJ_CLEANUP.
I think of those as the constructor and destructor for a file handle.

Since the only real different between CLEANUP and CLOSE is that CLEANUP
is sent while the process still exists and CLOSE is sent after it is
gone, I almost always implement CREATE and CLEANUP, but not CLOSE.

Is there a flaw in my philosophy that I have missed?

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

14

Well, that is ONE difference. However, for ME, the key difference is that CLEANUP is sent in the context of the process that did the open. CLOSE is not.

If we define “per instance” as “per OPEN instance”, then sure… that’s pretty reasonable.

If the state your handling is process specific (consider the classic – and often seriously broken – design of a driver mapping a block of memory into the space of a user application), the you MUST undo the state in CLEANUP.

Well, only problem is that handling CLEANUP correctly can be a real, serious, mess.

The first big issue I try to warn people about is that you can still receive I/O requests for the cleaned-up file handle between receiving the IRP_MJ_CLEANUP and receiving the CLOSE. Therefore, as part of your CLEANUP processing, you need to be sure that your driver doesn’t do the “wrong thing” if a request arrives for a “cleaned-up” File Object. And… if you sit down to think of it for a bit… you’ll see that the problem leads to a pretty big race condition.

Let’s say you keep a state variable in your File Object that says “This File Object has been cleaned up” (you can use FsContext2 or even one of the File Object flags for this purpose). It’s obviously possible for your I/O initiation code to race against the maintenance of that state… unless ALL the per-instance information including the state flag is referenced under lock.

So… handing CLEANUP correctly can be tough.

The OTHER big issue I always like to ask people about when they say they handle CLEANUP correctly is Duplicate Handles. What do you do with handles that have been duplicated into another process? Not an easy problem to solve.

So… in case I didn’t say it earlier… handling CLEANUP correctly can be tough.

I, personally, try to avoid it at all costs.

Peter
OSR