IRP_MJ_CLEANUP is not called

NTDEV
1

I must be doing something stupid, but I’m not getting a call
to my IRP_MJ_CLEANUP dispatch routine.

The setup is that the application issues an ioctl to pin down
and map some memory for the device to scribble on. The driver
leaves the IRP for that ioctl pending, so that the pages stay
pinned.

I have a cleanup routine that I want to use to detect that the
application is killed, or closes the handle with the pending
ioctl. I have no cancel routine for the IRP, I need to do some
work in PASSIVE mode and hoped that the cleanup routine will
allow me to cleanup pending IRPs.

But it’s not happening that way. I do not get a cleanup irp
(and close of the handle hangs) if my map ioctl is pending.
What is the cleanup irp waiting for?

Steve Williams “The woods are lovely, dark and deep.
steve at icarus.com But I have promises to keep,
http://www.icarus.com and lines to code before I sleep,
http://www.picturel.com And lines to code before I sleep.”

2

The cancel routine to run.

“Stephen Williams” wrote in message
news:xxxxx@ntdev…
>
> I must be doing something stupid, but I’m not getting a call
> to my IRP_MJ_CLEANUP dispatch routine.
>
> The setup is that the application issues an ioctl to pin down
> and map some memory for the device to scribble on. The driver
> leaves the IRP for that ioctl pending, so that the pages stay
> pinned.
>
> I have a cleanup routine that I want to use to detect that the
> application is killed, or closes the handle with the pending
> ioctl. I have no cancel routine for the IRP, I need to do some
> work in PASSIVE mode and hoped that the cleanup routine will
> allow me to cleanup pending IRPs.
>
> But it’s not happening that way. I do not get a cleanup irp
> (and close of the handle hangs) if my map ioctl is pending.
> What is the cleanup irp waiting for?
> –
> Steve Williams “The woods are lovely, dark and deep.
> steve at icarus.com But I have promises to keep,
> http://www.icarus.com and lines to code before I sleep,
> http://www.picturel.com And lines to code before I sleep.”
>

3

The cleanup irp is not waiting, it is the process cleanup code in
general that is waiting. I bet if you look at the pended irp w/out the
cancel routine, Irp->Cancel is TRUE. The process cleanup code is trying
to complete all pending i/o by canceling it. If you need to process the
irp’s completion at passive level, allocate a work item before you pend
the request and in the cancellation routine queue to a work item to
finish the cancellation processing.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Stephen Williams
Sent: Wednesday, June 29, 2005 3:04 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] IRP_MJ_CLEANUP is not called

I must be doing something stupid, but I’m not getting a call
to my IRP_MJ_CLEANUP dispatch routine.

The setup is that the application issues an ioctl to pin down
and map some memory for the device to scribble on. The driver
leaves the IRP for that ioctl pending, so that the pages stay
pinned.

I have a cleanup routine that I want to use to detect that the
application is killed, or closes the handle with the pending
ioctl. I have no cancel routine for the IRP, I need to do some
work in PASSIVE mode and hoped that the cleanup routine will
allow me to cleanup pending IRPs.

But it’s not happening that way. I do not get a cleanup irp
(and close of the handle hangs) if my map ioctl is pending.
What is the cleanup irp waiting for?

Steve Williams “The woods are lovely, dark and deep.
steve at icarus.com But I have promises to keep,
http://www.icarus.com and lines to code before I sleep,
http://www.picturel.com And lines to code before I sleep.”

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

4

So you are saying that the IRP_MJ_CLEANUP will not run until all
pending IRPs for the file are already completed (one way or another)?
Doesn’t that imply that by the time the dispatch function is called
there can’t possibly be any more pending IRPs? And if that’s the
case, why all the talk in the documentation about cleaning up pending
IRPs?

Color me confused. What’s the point of the IRP_MJ_CLEANUP?

Doron Holan wrote:

The cleanup irp is not waiting, it is the process cleanup code in
general that is waiting. I bet if you look at the pended irp w/out the
cancel routine, Irp->Cancel is TRUE. The process cleanup code is trying
to complete all pending i/o by canceling it. If you need to process the
irp’s completion at passive level, allocate a work item before you pend
the request and in the cancellation routine queue to a work item to
finish the cancellation processing.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Stephen Williams
Sent: Wednesday, June 29, 2005 3:04 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] IRP_MJ_CLEANUP is not called

I must be doing something stupid, but I’m not getting a call
to my IRP_MJ_CLEANUP dispatch routine.

The setup is that the application issues an ioctl to pin down
and map some memory for the device to scribble on. The driver
leaves the IRP for that ioctl pending, so that the pages stay
pinned.

I have a cleanup routine that I want to use to detect that the
application is killed, or closes the handle with the pending
ioctl. I have no cancel routine for the IRP, I need to do some
work in PASSIVE mode and hoped that the cleanup routine will
allow me to cleanup pending IRPs.

But it’s not happening that way. I do not get a cleanup irp
(and close of the handle hangs) if my map ioctl is pending.
What is the cleanup irp waiting for?

–
Steve Williams “The woods are lovely, dark and deep.
steve at icarus.com But I have promises to keep,
http://www.icarus.com and lines to code before I sleep,
http://www.picturel.com And lines to code before I sleep.”

5

The IRP_MJ_CLEANUP documentation says:

“Receipt of this request indicates that the last handle for a
file object that is associated with the target device object has
been closed (but, due to outstanding I/O requests, might not have
been released).”

What do the “outstanding I/O requests” refer to? A little further
along, it says:

“…, the driver must cancel and complete only the currently queued
IRPs that are associated with the file object handle that is being
released.”

How can this function “cancel and complete” any IRPs if the IRPs
must be cancelled and completed before this function will run? I
simply don’t get it. And under the title “Writing Dispatch Routines”
and “DispatchCleanup Routines” it says a driver’s DispatchCleanup
routine should:"

* Cancel all IRPs that are currently queued to the target device
object, if they are associated iwth the file object that is
specified in the FileObject member of the driver’s I/O stack
location.

But again, how can there *be* any queued IRPs, in *any* queue, if
they are cancelled by the O/S before the cleanup is called?

Color me brightly confused.

Doron Holan wrote:

The cleanup irp is not waiting, it is the process cleanup code in
general that is waiting. I bet if you look at the pended irp w/out the
cancel routine, Irp->Cancel is TRUE. The process cleanup code is trying
to complete all pending i/o by canceling it. If you need to process the
irp’s completion at passive level, allocate a work item before you pend
the request and in the cancellation routine queue to a work item to
finish the cancellation processing.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Stephen Williams
Sent: Wednesday, June 29, 2005 3:04 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] IRP_MJ_CLEANUP is not called

I must be doing something stupid, but I’m not getting a call
to my IRP_MJ_CLEANUP dispatch routine.

The setup is that the application issues an ioctl to pin down
and map some memory for the device to scribble on. The driver
leaves the IRP for that ioctl pending, so that the pages stay
pinned.

I have a cleanup routine that I want to use to detect that the
application is killed, or closes the handle with the pending
ioctl. I have no cancel routine for the IRP, I need to do some
work in PASSIVE mode and hoped that the cleanup routine will
allow me to cleanup pending IRPs.

But it’s not happening that way. I do not get a cleanup irp
(and close of the handle hangs) if my map ioctl is pending.
What is the cleanup irp waiting for?

–
Steve Williams “The woods are lovely, dark and deep.
steve at icarus.com But I have promises to keep,
http://www.icarus.com and lines to code before I sleep,
http://www.picturel.com And lines to code before I sleep.”

6

Sigh.

When process teardown proceeds, it is because the LAST thread in the
process exits (threads always self-terminate, last thread in the process
tears down the process).

What is more likely is that a thread in the process is blocked and not
terminating because it has waited with APCs disabled.

When the last thread terminates (that is, the last thread calls
PspExitThread) it will walk the handle table for the process and close
all of the handles in the table. Hence, my point - if you don’t see the
IRP_MJ_CLEANUP it is because the last thread isn’t terminating. Use the
debugger to check the process and the thread; if you have one thread
remaining, check to see if it has pending APCs (“!apc” in the debugger)
and if it does, then look at the wait IRQL and the APC disable bits ("dt
nt!_KTHREAD

"). If it really IS blocking the
APC delivery, you'll need to figure out why - either a raise to IRQL
APC_LEVEL or the use of a guarded mutex (W2K3 only) will cause this.

I've used the IRP_MJ_CLEANUP technique numerous times - and it is VASTLY
easier than using the cancellation technique in IRPs.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Stephen Williams
Sent: Wednesday, June 29, 2005 8:01 PM
To: ntdev redirect
Subject: Re:[ntdev] IRP_MJ_CLEANUP is not called

The IRP_MJ_CLEANUP documentation says:

"Receipt of this request indicates that the last handle for a
file object that is associated with the target device object has
been closed (but, due to outstanding I/O requests, might not have
been released)."

What do the "outstanding I/O requests" refer to? A little further
along, it says:

"..., the driver must cancel and complete only the currently queued
IRPs that are associated with the file object handle that is being
released."

How can this function "cancel and complete" any IRPs if the IRPs
must be cancelled and completed before this function will run? I
simply don't get it. And under the title "Writing Dispatch Routines"
and "DispatchCleanup Routines" it says a driver's DispatchCleanup
routine should:"

* Cancel all IRPs that are currently queued to the target device
object, if they are associated iwth the file object that is
specified in the FileObject member of the driver's I/O stack
location.

But again, how can there *be* any queued IRPs, in *any* queue, if
they are cancelled by the O/S before the cleanup is called?

Color me brightly confused.

Doron Holan wrote:
> The cleanup irp is not waiting, it is the process cleanup code in
> general that is waiting. I bet if you look at the pended irp w/out
the
> cancel routine, Irp->Cancel is TRUE. The process cleanup code is
trying
> to complete all pending i/o by canceling it. If you need to process
the
> irp's completion at passive level, allocate a work item before you
pend
> the request and in the cancellation routine queue to a work item to
> finish the cancellation processing.
>
> d
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Stephen
Williams
> Sent: Wednesday, June 29, 2005 3:04 PM
> To: Windows System Software Devs Interest List
> Subject: [ntdev] IRP_MJ_CLEANUP is not called
>
>
> I must be doing something stupid, but I'm not getting a call
> to my IRP_MJ_CLEANUP dispatch routine.
>
> The setup is that the application issues an ioctl to pin down
> and map some memory for the device to scribble on. The driver
> leaves the IRP for that ioctl pending, so that the pages stay
> pinned.
>
> I have a cleanup routine that I want to use to detect that the
> application is killed, or closes the handle with the pending
> ioctl. I have no cancel routine for the IRP, I need to do some
> work in PASSIVE mode and hoped that the cleanup routine will
> allow me to cleanup pending IRPs.
>
> But it's not happening that way. I do not get a cleanup irp
> (and close of the handle hangs) if my map ioctl is pending.
> What is the cleanup irp waiting for?

--
Steve Williams "The woods are lovely, dark and deep.
steve at icarus.com But I have promises to keep,
http://www.icarus.com and lines to code before I sleep,
http://www.picturel.com And lines to code before I sleep."

---
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
7

OK. I am brightly confused now, too.

On the one hand, I believe what Tony says. No question.

On the other hand, I have seen exactly what Steve is talking about.
However, if I waited about 5 or 6 minutes, I would eventually get the
IRP_MJ_CLEANUP, and could clean up after the application crashed. That was
my clue there was a problem with IRPs being cancelled (can’t find the docs
at the moment, but I remember reading something about a ~5 minute timer on
IRP cancellation (cancel rundown error?)). The only way I could find to fix
it was to make all the IRPs I pended cancellable. I was trying (at least I
thought I was trying) to do what Tony described, and avoid the whole mess
with IRP_MJ_CLEANUP. As far as I can recall, I didn’t have any threads in a
blocked state. There shouldn’t have been any threads waiting for anything.
But I always wondered if some process exit code in the C library or OS was
doing this behind the scenes.

-Dan

----- Original Message -----

Subject: RE: IRP_MJ_CLEANUP is not called
From: “Tony Mason”
> Date: Wed, 29 Jun 2005 20:27:10 -0400
> X-Message-Number: 30
>
> Sigh.
>
> When process teardown proceeds, it is because the LAST thread in the
> process exits (threads always self-terminate, last thread in the process
> tears down the process).
>
> What is more likely is that a thread in the process is blocked and not
> terminating because it has waited with APCs disabled.
>
> When the last thread terminates (that is, the last thread calls
> PspExitThread) it will walk the handle table for the process and close
> all of the handles in the table. Hence, my point - if you don’t see the
> IRP_MJ_CLEANUP it is because the last thread isn’t terminating. Use the
> debugger to check the process and the thread; if you have one thread
> remaining, check to see if it has pending APCs (“!apc” in the debugger)
> and if it does, then look at the wait IRQL and the APC disable bits (“dt
> nt!_KTHREAD ”). If it really IS blocking the
> APC delivery, you’ll need to figure out why - either a raise to IRQL
> APC_LEVEL or the use of a guarded mutex (W2K3 only) will cause this.
>
> I’ve used the IRP_MJ_CLEANUP technique numerous times - and it is VASTLY
> easier than using the cancellation technique in IRPs.
>
> Regards,
>
> Tony
>
> Tony Mason
> Consulting Partner
> OSR Open Systems Resources, Inc.
> http://www.osr.com
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Stephen Williams
> Sent: Wednesday, June 29, 2005 8:01 PM
> To: ntdev redirect
> Subject: Re:[ntdev] IRP_MJ_CLEANUP is not called
>
>
> The IRP_MJ_CLEANUP documentation says:
>
> “Receipt of this request indicates that the last handle for a
> file object that is associated with the target device object has
> been closed (but, due to outstanding I/O requests, might not have
> been released).”
>
> What do the “outstanding I/O requests” refer to? A little further
> along, it says:
>
> “…, the driver must cancel and complete only the currently queued
> IRPs that are associated with the file object handle that is being
> released.”
>
> How can this function “cancel and complete” any IRPs if the IRPs
> must be cancelled and completed before this function will run? I
> simply don’t get it. And under the title “Writing Dispatch Routines”
> and “DispatchCleanup Routines” it says a driver’s DispatchCleanup
> routine should:"
>
> * Cancel all IRPs that are currently queued to the target device
> object, if they are associated iwth the file object that is
> specified in the FileObject member of the driver’s I/O stack
> location.
>
> But again, how can there be any queued IRPs, in any queue, if
> they are cancelled by the O/S before the cleanup is called?
>
> Color me brightly confused.
>
>
> Doron Holan wrote:
>> The cleanup irp is not waiting, it is the process cleanup code in
>> general that is waiting. I bet if you look at the pended irp w/out
> the
>> cancel routine, Irp->Cancel is TRUE. The process cleanup code is
> trying
>> to complete all pending i/o by canceling it. If you need to process
> the
>> irp’s completion at passive level, allocate a work item before you
> pend
>> the request and in the cancellation routine queue to a work item to
>> finish the cancellation processing.
>>=20
>> d
>>=20
>> -----Original Message-----
>> From: xxxxx@lists.osr.com
>> [mailto:xxxxx@lists.osr.com] On Behalf Of Stephen
> Williams
>> Sent: Wednesday, June 29, 2005 3:04 PM
>> To: Windows System Software Devs Interest List
>> Subject: [ntdev] IRP_MJ_CLEANUP is not called
>>=20
>>=20
>> I must be doing something stupid, but I’m not getting a call
>> to my IRP_MJ_CLEANUP dispatch routine.
>>=20
>> The setup is that the application issues an ioctl to pin down
>> and map some memory for the device to scribble on. The driver
>> leaves the IRP for that ioctl pending, so that the pages stay
>> pinned.
>>=20
>> I have a cleanup routine that I want to use to detect that the
>> application is killed, or closes the handle with the pending
>> ioctl. I have no cancel routine for the IRP, I need to do some
>> work in PASSIVE mode and hoped that the cleanup routine will
>> allow me to cleanup pending IRPs.
>>=20
>> But it’s not happening that way. I do not get a cleanup irp
>> (and close of the handle hangs) if my map ioctl is pending.
>> What is the cleanup irp waiting for?
>
>
> --=20
> Steve Williams “The woods are lovely, dark and deep.
> steve at icarus.com But I have promises to keep,
> http://www.icarus.com and lines to code before I sleep,
> http://www.picturel.com And lines to code before I sleep.”

8

Daniel E. Germann wrote:

OK. I am brightly confused now, too.

On the one hand, I believe what Tony says. No question.

On the other hand, I have seen exactly what Steve is talking about.
However, if I waited about 5 or 6 minutes, I would eventually get the
IRP_MJ_CLEANUP, and could clean up after the application crashed.

I’m not creating any threads in my test application. In order to
make the ioctl in question pendable without hanging the program,
I opened it for OVERLAPPED I/O. Sure 'nuff, the application will
proceed, but hangs on exit. I see no sign that by CLEANUP is called,
but I don’t recall waiting for ~5 minutes.

–
Steve Williams “The woods are lovely, dark and deep.
steve at icarus.com But I have promises to keep,
http://www.icarus.com and lines to code before I sleep,
http://www.picturel.com And lines to code before I sleep.”

9

Daniel E. Germann wrote:

OK. I am brightly confused now, too.

On the one hand, I believe what Tony says. No question.

On the other hand, I have seen exactly what Steve is talking about.
However, if I waited about 5 or 6 minutes, I would eventually get the
IRP_MJ_CLEANUP, and could clean up after the application crashed.

Well, the 5-minute timeout is (as you alluded to) the I/O rundown
failure timeout. When a thread terminates with outstanding I/O, Windows
sends cancels for each of the IRPs. It will wait up to 5 minutes for
those IRPs to complete (normally or with cancel status, it doesn’t
matter). After 5 minutes, Windows “gives up” and lets the thread exit.

The rules:

IRP_MJ_CLEANUP is called when the last HANDLE is closed on a file object.

IRP_MJ_CLOSE is called when the last REFERENCE to the file object goes away.

At OSR we use the “teardown the in-process mapping from the cleanup
routine (to avoid the otherwise inevitable BSOD)” technique all the
time. That’s what cleanup is for. We first described this several
years ago in an article in The NT Insider about mapping memory back to
user apps.

If you’re not getting the cleanup call, the handle’s not being closed.
But as you’re seeing IRPs in the cancelled state, that sounds unlikely.

Sorry, aside from the above background information, I can’t shed any
further light on what you’re seeing. I suggest, as did Tony, a good
session with the debugger to help sort this out.

If you find anything out, please do post what you find. I’m curious to
know…

Peter
OSR

10

PeterGV (OSR) wrote:

Daniel E. Germann wrote:

> OK. I am brightly confused now, too.
>
> On the one hand, I believe what Tony says. No question.
>
> On the other hand, I have seen exactly what Steve is talking about.
> However, if I waited about 5 or 6 minutes, I would eventually get the
> IRP_MJ_CLEANUP, and could clean up after the application crashed.

Well, the 5-minute timeout is (as you alluded to) the I/O rundown
failure timeout. When a thread terminates with outstanding I/O, Windows
sends cancels for each of the IRPs. It will wait up to 5 minutes for
those IRPs to complete (normally or with cancel status, it doesn’t
matter). After 5 minutes, Windows “gives up” and lets the thread exit.

At OSR we use the “teardown the in-process mapping from the cleanup
routine (to avoid the otherwise inevitable BSOD)” technique all the
time. That’s what cleanup is for. We first described this several
years ago in an article in The NT Insider about mapping memory back to
user apps.

I think I found the culprit, and I think this needs to be documented
somewhere. To review, I’m “pending” an ioctl IRP with METHOD_OUT_DIRECT
in order to lock some user-mode memory into physical memory so that the
device may scribble on it randomly. My problem is that when I close the
application, I’m not getting an IRP_MJ_CLEANUP on the handle for the
“pended” ioctl.

The gotcha, apparently, is that the pended irp needs a Cancel function
set by IoSetCancelFunction. Without out, Windows apparently assumes that
it is pointless to make an IRP_MJ_CLEANUP until the pended IRP is complete.

*But* if the pended IRP has a Cancel function, the IRP_MJ_CLEANUP will
be sent *before* it tries to Cancel the specific IRP. Thus I can do my
PASSIVE-mode cancelling of the pended IRPs and remove the Cancel routines
before they are actually called.

So the trick, and the apparently unpublished requirement, is that all
the pending IRPs must complete on their own, or have a Cancel function
as a signal to Windows that the IRP is cancellable before it will try
to call IRP_MJ_CLEANUP. But the CLEANUP is (apparently) allowed to cancel
those pending IRPs and remove the Cancel function so that they are not
cancelled twice.

Does this make sense?

–
Steve Williams “The woods are lovely, dark and deep.
steve at icarus.com But I have promises to keep,
http://www.icarus.com and lines to code before I sleep,
http://www.picturel.com And lines to code before I sleep.”

11

Stephen Williams wrote:

I think I found the culprit, and I think this needs to be documented
somewhere. To review, I’m “pending” an ioctl IRP with METHOD_OUT_DIRECT
in order to lock some user-mode memory into physical memory so that the
device may scribble on it randomly. My problem is that when I close the
application, I’m not getting an IRP_MJ_CLEANUP on the handle for the
“pended” ioctl.

It seems I got it wrong. I only get the IRP_MJ_CLEANUP if I explicitly
close the handle. If the application is killed or runs off the end, I
still do *not* get get an IRP_MJ_CLEANUP, but I *do* see that the Cancel
function is called.

It does appears that the IRP_MJ_CLEANUP is called *before* the Cancel
function on a normal CloseHandle, and it appears that the IRP_MJ_CLEANUP
is *not* called before the Cancel function when the app dies.

Either way, though, the Cancel function appears to be key.

*sigh*

–
Steve Williams “The woods are lovely, dark and deep.
steve at icarus.com But I have promises to keep,
http://www.icarus.com and lines to code before I sleep,
http://www.picturel.com And lines to code before I sleep.”

12

Stephen Williams wrote:

Stephen Williams wrote:

> I think I found the culprit, and I think this needs to be documented
> somewhere. To review, I’m “pending” an ioctl IRP with METHOD_OUT_DIRECT
> in order to lock some user-mode memory into physical memory so that the
> device may scribble on it randomly. My problem is that when I close the
> application, I’m not getting an IRP_MJ_CLEANUP on the handle for the
> “pended” ioctl.

It seems I got it wrong. I only get the IRP_MJ_CLEANUP if I explicitly
close the handle. If the application is killed or runs off the end, I
still do *not* get get an IRP_MJ_CLEANUP, but I *do* see that the Cancel
function is called.

It does appears that the IRP_MJ_CLEANUP is called *before* the Cancel
function on a normal CloseHandle, and it appears that the IRP_MJ_CLEANUP
is *not* called before the Cancel function when the app dies.

Either way, though, the Cancel function appears to be key.

I have no doubt that you’re seeing what you’re seeing, but I can’t
believe that the act of setting a cancel function into an IRP is the
reason you’re seeing a change in the fundamental behavior of CLEANUP.
It is entirely possible that setting the cancel function causes
something ELSE to happen (like, other open instances get closed) that
then causes the cleanup… But if the cancel function itself is the
key, you have found a major and serious operating system bug – and I
doubt this is the case.

Can you give us more information to debug this problem? For example,
what are the File Object reference counts: (a) when the IOCTL is sitting
in your queue and before the app does the close, (b) after the APP does
the close (if there’s no cancel routine in the IRP?

When you do not have a cancel routine, and the app closes the handle,
does the app return from the close or does it hang in close?

How many open instances of your device are there? What’s the device type?

How is the device being opened (with what open options)??

You said that when your app exits without doing the close specifically,
your cleanup routine isn’t called. But the app DOES completely exit?
So, what’s happening to the file object, then? Does it stay around?

The OS code for close is relatively simple. When a close is issued the
object-specific close procedure is invoked. What the File Object’s close
procedure does, if this is the close of the final handle to the File
Object, is build an IRP with major function code IRP_MJ_CLEANUP and send
it to the driver. How many IRPs are pending against the file object, or
whether they have cancel routines in them or not really isn’t a factor.

I suspect that we’re missing a key piece of the puzzle…

Peter
OSR

13

PeterGV (OSR) wrote:

Stephen Williams wrote:

> Stephen Williams wrote:
>
>> I think I found the culprit, and I think this needs to be documented
>> somewhere. To review, I’m “pending” an ioctl IRP with METHOD_OUT_DIRECT
>> in order to lock some user-mode memory into physical memory so that the
>> device may scribble on it randomly. My problem is that when I close the
>> application, I’m not getting an IRP_MJ_CLEANUP on the handle for the
>> “pended” ioctl.
>
>
>
> It seems I got it wrong. I only get the IRP_MJ_CLEANUP if I explicitly
> close the handle. If the application is killed or runs off the end, I
> still do *not* get get an IRP_MJ_CLEANUP, but I *do* see that the Cancel
> function is called.
>
> It does appears that the IRP_MJ_CLEANUP is called *before* the Cancel
> function on a normal CloseHandle, and it appears that the IRP_MJ_CLEANUP
> is *not* called before the Cancel function when the app dies.
>
> Either way, though, the Cancel function appears to be key.
>

I have no doubt that you’re seeing what you’re seeing, but I can’t
believe that the act of setting a cancel function into an IRP is the
reason you’re seeing a change in the fundamental behavior of CLEANUP. It
is entirely possible that setting the cancel function causes something
ELSE to happen (like, other open instances get closed) that then causes
the cleanup… But if the cancel function itself is the key, you have
found a major and serious operating system bug – and I doubt this is
the case.

When you do not have a cancel routine, and the app closes the handle,
does the app return from the close or does it hang in close?

It appears to hang in the close. Or did until I added the Cancel function.
Either that or I confused myself with multiple tests that I lost track of:-(
In fact, in light of your question, I’ll retest that specific case.

How many open instances of your device are there? What’s the device type?

One open instance at the moment, but there was another instance opened
and closed again before I tried to close the handle of the pended IRP.
I tried manipulating the device with the mapped frame while the frame
is still mapped. But at the time of the close, I’m confident there is
only exactly 1 HANDLE open on the device.

How is the device being opened (with what open options)??

The CreateFile is:
fdx = CreateFile(“\\.\isex0”, GENERIC_READ|GENERIC_WRITE, 0, 0,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL|FILE_FLAG_OVERLAPPED, 0);

Right after that, I try to create a “frame” of memory and map it
like this:

fprintf(stderr, “creating 4096K frame 0\n”);
frame_size = 4096*1024;
frame = VirtualAlloc(0, frame_size, MEM_COMMIT, PAGE_READWRITE);
frame_arg.frame_id = 0;

map_over.hEvent = 0;
rc = DeviceIoControl(fdx, ISEX_MAKE_MAP_FRAME,
&frame_arg, sizeof frame_arg,
frame, frame_size, 0, &map_over);

The DeviceIoControl returns FALSE and GetLastError() returns PENDING,
exactly like I would expect. My driver is taking the ISEX_MAKE_MAP_FRAME
IRP and “pending” it properly

You said that when your app exits without doing the close specifically,
your cleanup routine isn’t called. But the app DOES completely exit?
So, what’s happening to the file object, then? Does it stay around?

The app does *not* completely exit. It hangs exiting. The current
theory is it is waiting for the Cancel to complete the pended IRP.

The OS code for close is relatively simple. When a close is issued the
object-specific close procedure is invoked. What the File Object’s close
procedure does, if this is the close of the final handle to the File
Object, is build an IRP with major function code IRP_MJ_CLEANUP and send
it to the driver. How many IRPs are pending against the file object, or
whether they have cancel routines in them or not really isn’t a factor.

I suspect that we’re missing a key piece of the puzzle…

As likely as not, *I’m* missing an important key to this puzzle.
Since I posted the previous, I’ve found that the Cancel function
is being called when the application is killed, but the CLEANUP
is not being called before the Cancel. In fact, the O/S seems to
be waiting for the IRPs to complete before issuing the CLEANUP.
That is what xxxxx@windows.microsoft.com said in a previous post.
That’s an annoyance as I need to do some work (allocating a small
common buffer to replace some tables, for example) in PASSIVE mode.

The surprise is that the CloseHandle seems to cause the CLEANUP
to be called *before* the Cancel routine. So I’ve got my driver
handling the explicit close cleanly. I’m now only stumped for the
case where the application does not call CloseHandle. But I’m
getting a Cancel, in that case, so I may be able to schedule some
sort of work unit to get things moving.

–
Steve Williams “The woods are lovely, dark and deep.
steve at icarus.com But I have promises to keep,
http://www.icarus.com and lines to code before I sleep,
http://www.picturel.com And lines to code before I sleep.”

14

1.) When you close a handle explicitly (via a call to CloseHandle()) an
IRP_MJ_CLEANUP is generated if the handle count goes to zero (i.e. there
are no duplicate handles outstanding). This is regardless if there are
other outstanding IRPs on the file object. This has nothing at all to do
with whether there is a cancel routine outstanding on an IRP. You do not
have to necessarily flush out all i/o out on a cleanup - for instance
paging i/o can occur on a file object after IRP_MJ_CLEANUP was issued.
There are no hard & fast rules for an FS as to what exactly gets
‘cleaned up’ - for instance outstanding byte range locks are all
released on cleanup.

2.) When you terminate a thread, a couple of things happen:

2.1) First all outstanding IRPs are cancelled and i/o manager
takes a 5 minute time out. This means that if you pended i/o but haven’t
implemented a cancel routine, you will see a full long time out on the
IRP before the thread can really terminate. That is why you must
implement a cancel routine for i/os that you pend.

2.2) After the above, the handle table is rundown. Note that
the app has not closed the handle - this is the process manager forcing
the close to restore sanity after the thread goes away. During the
rundown, the handles are forcibly closed, thereby generating an
IRP_MJ_CLEANUP for the file object.

So if you wait long enough you should see the cleanup. What you really
ought to do for long-standing IRPs is implement a cancel routine.
Cancel routines for an IOCTL IRP and IRP_MJ_CLEANUP for a file object
are completely different things, I don’t see why you have dependencies
on which gets called first. You shouldn’t.

Ravi

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Stephen Williams
Sent: Tuesday, July 19, 2005 3:09 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] IRP_MJ_CLEANUP is not called

PeterGV (OSR) wrote:

Stephen Williams wrote:

> Stephen Williams wrote:
>
>> I think I found the culprit, and I think this needs to be documented

>> somewhere. To review, I’m “pending” an ioctl IRP with
>> METHOD_OUT_DIRECT in order to lock some user-mode memory into
>> physical memory so that the device may scribble on it randomly. My
>> problem is that when I close the application, I’m not getting an
>> IRP_MJ_CLEANUP on the handle for the “pended” ioctl.
>
>
>
> It seems I got it wrong. I only get the IRP_MJ_CLEANUP if I
> explicitly close the handle. If the application is killed or runs off

> the end, I still do *not* get get an IRP_MJ_CLEANUP, but I *do* see
> that the Cancel function is called.
>
> It does appears that the IRP_MJ_CLEANUP is called *before* the Cancel

> function on a normal CloseHandle, and it appears that the
> IRP_MJ_CLEANUP is *not* called before the Cancel function when the
app dies.
>
> Either way, though, the Cancel function appears to be key.
>

I have no doubt that you’re seeing what you’re seeing, but I can’t
believe that the act of setting a cancel function into an IRP is the
reason you’re seeing a change in the fundamental behavior of CLEANUP.
It is entirely possible that setting the cancel function causes
something ELSE to happen (like, other open instances get closed) that
then causes the cleanup… But if the cancel function itself is the
key, you have found a major and serious operating system bug – and I
doubt this is the case.

When you do not have a cancel routine, and the app closes the handle,
does the app return from the close or does it hang in close?

It appears to hang in the close. Or did until I added the Cancel
function.
Either that or I confused myself with multiple tests that I lost track
of:-( In fact, in light of your question, I’ll retest that specific
case.

How many open instances of your device are there? What’s the device
type?

One open instance at the moment, but there was another instance opened
and closed again before I tried to close the handle of the pended IRP.
I tried manipulating the device with the mapped frame while the frame is
still mapped. But at the time of the close, I’m confident there is only
exactly 1 HANDLE open on the device.

How is the device being opened (with what open options)??

The CreateFile is:
fdx = CreateFile(“\\.\isex0”, GENERIC_READ|GENERIC_WRITE, 0,
0,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL|FILE_FLAG_OVERLAPPED, 0);

Right after that, I try to create a “frame” of memory and map it like
this:

fprintf(stderr, “creating 4096K frame 0\n”);
frame_size = 4096*1024;
frame = VirtualAlloc(0, frame_size, MEM_COMMIT, PAGE_READWRITE);
frame_arg.frame_id = 0;

map_over.hEvent = 0;
rc = DeviceIoControl(fdx, ISEX_MAKE_MAP_FRAME,
&frame_arg, sizeof frame_arg,
frame, frame_size, 0, &map_over);

The DeviceIoControl returns FALSE and GetLastError() returns PENDING,
exactly like I would expect. My driver is taking the ISEX_MAKE_MAP_FRAME
IRP and “pending” it properly

You said that when your app exits without doing the close
specifically, your cleanup routine isn’t called. But the app DOES
completely exit?
So, what’s happening to the file object, then? Does it stay around?

The app does *not* completely exit. It hangs exiting. The current theory
is it is waiting for the Cancel to complete the pended IRP.

The OS code for close is relatively simple. When a close is issued
the object-specific close procedure is invoked. What the File Object’s

close procedure does, if this is the close of the final handle to the
File Object, is build an IRP with major function code IRP_MJ_CLEANUP
and send it to the driver. How many IRPs are pending against the file

object, or whether they have cancel routines in them or not really
isn’t a factor.

I suspect that we’re missing a key piece of the puzzle…

As likely as not, *I’m* missing an important key to this puzzle.
Since I posted the previous, I’ve found that the Cancel function is
being called when the application is killed, but the CLEANUP is not
being called before the Cancel. In fact, the O/S seems to be waiting for
the IRPs to complete before issuing the CLEANUP.
That is what xxxxx@windows.microsoft.com said in a previous post.
That’s an annoyance as I need to do some work (allocating a small common
buffer to replace some tables, for example) in PASSIVE mode.

The surprise is that the CloseHandle seems to cause the CLEANUP to be
called *before* the Cancel routine. So I’ve got my driver handling the
explicit close cleanly. I’m now only stumped for the case where the
application does not call CloseHandle. But I’m getting a Cancel, in that
case, so I may be able to schedule some sort of work unit to get things
moving.

–
Steve Williams “The woods are lovely, dark and deep.
steve at icarus.com But I have promises to keep,
http://www.icarus.com and lines to code before I sleep,
http://www.picturel.com And lines to code before I sleep.”

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

15

> So the trick, and the apparently unpublished requirement, is that all

the pending IRPs must complete on their own, or have a Cancel function
as a signal to Windows that the IRP is cancellable before it will try
to call IRP_MJ_CLEANUP. But the CLEANUP is (apparently) allowed to
cancel
those pending IRPs and remove the Cancel function so that they are not
cancelled twice.

Correct. The “kosher” way is to have both cancel routines (this is absolutely a
must) and cancelling in MJ_CLEANUP - with the usual check of not cancelling
twice.

The CLEANUP handler should consume the IRPs from Csq in a usual Csq way and
complete them with STATUS_CANCELLED.

Anyway you can omit the CLEANUP handler, not a must - just a good thing to do.
But the cancel routine is mandatory. Either use MS’s Csq (better) or implement
your own (if you’re feeling really good in parallel programming where the data
can be accessed in parallel from the several CPUs and contexts).

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

16

Yes. CloseHandle will do no IRP cancellation by itself. It only calls
CLEANUP handler, that is all.
App exit means - thread exit, and the thread rundown does the IRP
cancellation internally by IoCancelrp routine for each IRP linked to this
thread.
The app’s handle table is (IIRC) destroyed later in the rundown process.
So, not having the cancel routines will cause the app to hang on exit in the
IRP cancellation on thread rundown.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Stephen Williams”
Newsgroups: ntdev
To: “Windows System Software Devs Interest List”
Sent: Tuesday, July 19, 2005 4:03 AM
Subject: Re:[ntdev] IRP_MJ_CLEANUP is not called

> Stephen Williams wrote:
> > I think I found the culprit, and I think this needs to be documented
> > somewhere. To review, I’m “pending” an ioctl IRP with METHOD_OUT_DIRECT
> > in order to lock some user-mode memory into physical memory so that the
> > device may scribble on it randomly. My problem is that when I close the
> > application, I’m not getting an IRP_MJ_CLEANUP on the handle for the
> > “pended” ioctl.
>
> It seems I got it wrong. I only get the IRP_MJ_CLEANUP if I explicitly
> close the handle. If the application is killed or runs off the end, I
> still do not get get an IRP_MJ_CLEANUP, but I do see that the Cancel
> function is called.
>
> It does appears that the IRP_MJ_CLEANUP is called before the Cancel
> function on a normal CloseHandle, and it appears that the IRP_MJ_CLEANUP
> is not called before the Cancel function when the app dies.
>
> Either way, though, the Cancel function appears to be key.
>
> sigh
>
> –
> Steve Williams “The woods are lovely, dark and deep.
> steve at icarus.com But I have promises to keep,
> http://www.icarus.com and lines to code before I sleep,
> http://www.picturel.com And lines to code before I sleep.”
>
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com

17

Stephen Williams wrote:

>>
In fact, the O/S seems to
be waiting for the IRPs to complete before issuing the CLEANUP.
That is what xxxxx@windows.microsoft.com said in a previous post.
>>>

I don’t understand what Doron is alluding to here.

Doron! I’m looking at your posting from 6/29… What are you referring to?

>>
The surprise is that the CloseHandle seems to cause the CLEANUP
to be called *before* the Cancel routine.

>>>>

No. This is precisely how it’s SUPPOSED to work. To re-iterate:

IRP_MJ_CLEANUP: Last handle to the file object goes away.

IRP_MJ_CLOSE: Last reference to the file object goes away.

If you look back at Tony’s post from 6/29, he said:

>>> (Tony Mason)

What is more likely is that a thread in the process is blocked and not
terminating because it has waited with APCs disabled.

When the last thread terminates (that is, the last thread calls
PspExitThread) it will walk the handle table for the process and close
all of the handles in the table. Hence, my point - if you don’t see the
IRP_MJ_CLEANUP it is because the last thread isn’t terminating. Use the
debugger to check the process and the thread; if you have one thread
remaining, check to see if it has pending APCs (“!apc” in the debugger)
and if it does, then look at the wait IRQL and the APC disable bits ("dt
nt!_KTHREAD

"). If it really IS blocking the
APC delivery, you'll need to figure out why - either a raise to IRQL
APC_LEVEL or the use of a guarded mutex (W2K3 only) will cause this.

>>>>

Did you try what Tony advised? What HE said makes sense.

Sorry, I just don't have the bandwidth to whip together a driver to
demonstrate that this works the way it should...

P
18

Ravisankar Pudipeddi wrote:

1.) When you close a handle explicitly (via a call to CloseHandle()) an
…

2.) When you terminate a thread, a couple of things happen:

2.1) First all outstanding IRPs are cancelled and i/o manager
…

2.2) After the above, the handle table is rundown. Note that
…

VERY nicely written, Ravi. Thanks for taking the time.

Peter
OSR

19

Maxim S. Shatskih wrote:

cancel
>those pending IRPs and remove the Cancel function so that they are not
>cancelled twice.

Correct. The “kosher” way is to have both cancel routines (this is absolutely a
must) and cancelling in MJ_CLEANUP - with the usual check of not cancelling
twice.

The CLEANUP handler should consume the IRPs from Csq in a usual Csq way and
complete them with STATUS_CANCELLED.

Anyway you can omit the CLEANUP handler, not a must - just a good thing to do.
But the cancel routine is mandatory. Either use MS’s Csq (better) or implement
your own (if you’re feeling really good in parallel programming where the data
can be accessed in parallel from the several CPUs and contexts).

…well, now (for the archives) we have to be complete.

If you decide to do as Max suggests and “consume the IRPs for the CSQ”
in your MJ_CLEANUP routine (personally, I don’t recommend that people do
this… why bother?), you need to be aware that new IRPs can arrive
after your CLEANUP routine has been called.

Irrespective of whether or not you do IRP cancellation in your CLEANUP
routine, if you do something like tear-down state in the CLEANUP routine
(which is what the OP is doing, IRRC) you need to make sure you don’t do
the “wrong thing” when an IRP arrives after CLEANUP… when you’ve
already torn down the state.

Set a flag in the File Object to track that you’ve gotten a cleanup, and
reject any newly arriving IRPs for that File Object if the flag is set.

More critical trivia, brought to you through your use WDM,

Peter
OSR

20

Thanks but I have a typo, and I apologize for that (really need to have
a doc writer backing me up when I send long emails):
you don’t get handle rundown on termination of a thread, you get handle
rundown on termination of a *process*.
So if you just terminate the thread you just see 2.1 (i.e. cancellation
of the IRPs & the timeout), but you don’t see handle rundown.
Handle rundown occurs after the last thread terminates. So
TerminateThread() will not necessarily cause an IRP_MJ_CLEANUP on the
file object.
Ravi

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of PeterGV (OSR)
Sent: Wednesday, July 20, 2005 2:22 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] IRP_MJ_CLEANUP is not called

Ravisankar Pudipeddi wrote:

1.) When you close a handle explicitly (via a call to CloseHandle())
an
…

2.) When you terminate a thread, a couple of things happen:

2.1) First all outstanding IRPs are cancelled and i/o manager
…

2.2) After the above, the handle table is rundown. Note that
…

VERY nicely written, Ravi. Thanks for taking the time.

Peter
OSR

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com