Queuing IRP's from completion routine

I need to do passive or APC-level processing on IRP’s from my completion
routine. My thought is to return STATUS_MORE_PROCESSING_REQUIRED and
queue the IRP. But what about pending the IRP? If I understand right, my
dispatch routine needs to mark the IRP pending and return STATUS_PENDING
if I’m going to ever queue the IRP, whether at dispatch or completion.
The problem is that my only criterea for queuing the IRP in the completion
routine is the IRQL level it’s called at; if I’m running IRQL >=
DISPATCH_LEVEL, I want to queue the IRP. I certainly don’t want to mark
every IRP pending in the dispatch and incur the overhead of an APC to
complete every IRP when this may not be necessary.

Is there a way to queue IRP’s from my completion routine depending on the
IRQL it is called at without having to mark every IRP pending and return
STATUS_PENDING in my dispatch routine? I could just use a work item and
an event and wait from the completion routine for the work item to do the
processing, but I’d like to see if there is a way I can queue it.

Instead of queuing the Irp in the completion, I would return
STATUS_MORE_PROCESSING_REQUIRED and fire a work item to do whatever needs to
be done at PASSIVE_LEVEL and finally complete the IRP.

Make sure to topmost driver in the stack returned STATUS_PENDING from his
DispatchRoutine.

Calvin Guan, Software Developer xxxxx@nospam.ati.com
SW2D-Radeon NT Core Drivers
ATI Technologies Inc.
1 Commerce Valley Drive East
Markham, Ontario, Canada L3T 7X6
Tel: (905) 882-2600 Ext. 8654
Find a driver: http://www.ati.com/support/driver.html

-----Original Message-----
From: Jonathan [mailto:xxxxx@xmission.com]
Sent: Wednesday, November 05, 2003 3:35 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Queuing IRP’s from completion routine

I need to do passive or APC-level processing on IRP’s from my
completion
routine. My thought is to return STATUS_MORE_PROCESSING_REQUIRED and
queue the IRP. But what about pending the IRP? If I
understand right, my
dispatch routine needs to mark the IRP pending and return
STATUS_PENDING
if I’m going to ever queue the IRP, whether at dispatch or
completion.
The problem is that my only criterea for queuing the IRP in
the completion
routine is the IRQL level it’s called at; if I’m running IRQL >=
DISPATCH_LEVEL, I want to queue the IRP. I certainly don’t
want to mark
every IRP pending in the dispatch and incur the overhead of an APC to
complete every IRP when this may not be necessary.

Is there a way to queue IRP’s from my completion routine
depending on the
IRQL it is called at without having to mark every IRP pending
and return
STATUS_PENDING in my dispatch routine? I could just use a
work item and
an event and wait from the completion routine for the work
item to do the
processing, but I’d like to see if there is a way I can queue it.


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@ati.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

If you weren’t to return STATUS_PENDING from your dispatch routine in
all cases, you are implying that your dispatch routine must wait for the
completion routine to fire before proceeding (otherwise, how would it
know to choose between STATUS_PENDING or the lower driver’s return
value?). This is forcing synchronous behavior on all IRPs, which for
most categories of I/O is a much bigger performance hit than the
overhead of forcing an APC for second-stage I/O completion.

Jonathan wrote:

I need to do passive or APC-level processing on IRP’s from my completion
routine. My thought is to return STATUS_MORE_PROCESSING_REQUIRED and
queue the IRP. But what about pending the IRP? If I understand right, my
dispatch routine needs to mark the IRP pending and return STATUS_PENDING
if I’m going to ever queue the IRP, whether at dispatch or completion.
The problem is that my only criterea for queuing the IRP in the completion
routine is the IRQL level it’s called at; if I’m running IRQL >=
DISPATCH_LEVEL, I want to queue the IRP. I certainly don’t want to mark
every IRP pending in the dispatch and incur the overhead of an APC to
complete every IRP when this may not be necessary.

Is there a way to queue IRP’s from my completion routine depending on the
IRQL it is called at without having to mark every IRP pending and return
STATUS_PENDING in my dispatch routine? I could just use a work item and
an event and wait from the completion routine for the work item to do the
processing, but I’d like to see if there is a way I can queue it.


Nick Ryan (MVP for DDK)

Is the reason for returning STATUS_PENDING that the I/O manager may try to
free the IRP twice if not?

I think you can return STATUS_MORE_PROCESSING_REQUIRED
and queue the IRP for any further processing. I mean
in your completion routine. If the
Irp->PendingReturned flag was set, you need to make
the Irp as pending.

In the completion routine if you return any other
status besides STATUS_MORE_PROCESSING_REQUIRED, the
completion process will continue. Since the IRP has
been completed, after that, you couldn’t do any
further processing to that IRP.

Michael

— Jonathan wrote: > I need
to do passive or APC-level processing on
> IRP’s from my completion
> routine. My thought is to return
> STATUS_MORE_PROCESSING_REQUIRED and
> queue the IRP. But what about pending the IRP? If
> I understand right, my
> dispatch routine needs to mark the IRP pending and
> return STATUS_PENDING
> if I’m going to ever queue the IRP, whether at
> dispatch or completion.
> The problem is that my only criterea for queuing the
> IRP in the completion
> routine is the IRQL level it’s called at; if I’m
> running IRQL >=
> DISPATCH_LEVEL, I want to queue the IRP. I
> certainly don’t want to mark
> every IRP pending in the dispatch and incur the
> overhead of an APC to
> complete every IRP when this may not be necessary.
>
> Is there a way to queue IRP’s from my completion
> routine depending on the
> IRQL it is called at without having to mark every
> IRP pending and return
> STATUS_PENDING in my dispatch routine? I could just
> use a work item and
> an event and wait from the completion routine for
> the work item to do the
> processing, but I’d like to see if there is a way I
> can queue it.
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as:
> xxxxx@yahoo.ca
> To unsubscribe send a blank email to
xxxxx@lists.osr.com

______________________________________________________________________
Post your free ad now! http://personals.yahoo.ca

> I need to do passive or APC-level processing on IRP’s from my completion

routine. My thought is to return STATUS_MORE_PROCESSING_REQUIRED and
queue the IRP.

This is OK.

But what about pending the IRP? If I understand right, my
dispatch routine needs to mark the IRP pending and return STATUS_PENDING
if I’m going to ever queue the IRP, whether at dispatch or completion.

No, your dispatch routine needs to call IoCallDriver in order for the IRP to go
to the completion routine :slight_smile:
Completion routines are called by lower drivers.

So, just:

IoSetCompletionRoutine()
return IoCallDriver()

in the dispatch routine, and queue in the completion. No marking as pending.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

Yes.

IRPs sent by the IO manager are freed in IopCompleteRequest routine.

IoCompleteRequest does (gross simplification):

UnwindAllStackLocations
if( IsMarkedAsPending(Irp) )
KeInsertQueueApc(&(Irp->Tail.Overlay.Apc), IopCompleteRequest);

There is also a function of IopSynchronousServiceTail, which is called by
NtxxxFile syscalls to do IoCallDriver and postprocess the IRP. It does:

Status = IoCallDriver()
if( Status !=STATUS_PENDING )
IopCompleteRequest().

Note that, if the status is not pending, then IopCompleteRequest is called
directly and not as an APC, which is faster. Surely this implies that
IoCompleteRequest was already called inline in the driver’s dispatch routine.

Then - IoMarkIrpPending without returning pending = calling
IopCompleteRequest twice = MULTIPLE_IRP_COMPLETE_REQUESTS BSOD.
Then - returning pending without IoMarkIrpPending = never calling
IopCompleteRequest = a hang.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Jonathan”
To: “Windows System Software Devs Interest List”
Sent: Thursday, November 06, 2003 12:26 AM
Subject: [ntdev] RE: Queuing IRP’s from completion routine

> Is the reason for returning STATUS_PENDING that the I/O manager may try to
> free the IRP twice if not?
>
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com

If he’s queueing the IRP in his completion routine and returning
STATUS_MORE_PROCESSING_REQUIRED, he definitely must mark the IRP pending
and return STATUS_PENDING from his dispatch routine (unless he ensures
that the IRP is post-processed and completed in his worker thread or in
the dispatch thread before returning).

Maxim S. Shatskih wrote:

>I need to do passive or APC-level processing on IRP’s from my completion
>routine. My thought is to return STATUS_MORE_PROCESSING_REQUIRED and
>queue the IRP.

This is OK.

>But what about pending the IRP? If I understand right, my
>dispatch routine needs to mark the IRP pending and return STATUS_PENDING
>if I’m going to ever queue the IRP, whether at dispatch or completion.

No, your dispatch routine needs to call IoCallDriver in order for the IRP to go
to the completion routine :slight_smile:
Completion routines are called by lower drivers.

So, just:

IoSetCompletionRoutine()
return IoCallDriver()

in the dispatch routine, and queue in the completion. No marking as pending.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com


Nick Ryan (MVP for DDK)

Why?

The following:

IoSetCompletionRoutine()
return IoCallDriver()

will also work.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Nick Ryan”
Newsgroups: ntdev
To: “Windows System Software Devs Interest List”
Sent: Thursday, November 06, 2003 5:12 AM
Subject: [ntdev] Re: Queuing IRP’s from completion routine

> If he’s queueing the IRP in his completion routine and returning
> STATUS_MORE_PROCESSING_REQUIRED, he definitely must mark the IRP pending
> and return STATUS_PENDING from his dispatch routine (unless he ensures
> that the IRP is post-processed and completed in his worker thread or in
> the dispatch thread before returning).
>
> Maxim S. Shatskih wrote:
>
> >>I need to do passive or APC-level processing on IRP’s from my completion
> >>routine. My thought is to return STATUS_MORE_PROCESSING_REQUIRED and
> >>queue the IRP.
> >
> >
> > This is OK.
> >
> >
> >>But what about pending the IRP? If I understand right, my
> >>dispatch routine needs to mark the IRP pending and return STATUS_PENDING
> >>if I’m going to ever queue the IRP, whether at dispatch or completion.
> >
> >
> > No, your dispatch routine needs to call IoCallDriver in order for the IRP
to go
> > to the completion routine :slight_smile:
> > Completion routines are called by lower drivers.
> >
> > So, just:
> >
> > IoSetCompletionRoutine()
> > return IoCallDriver()
> >
> > in the dispatch routine, and queue in the completion. No marking as
pending.
> >
> > Maxim Shatskih, Windows DDK MVP
> > StorageCraft Corporation
> > xxxxx@storagecraft.com
> > http://www.storagecraft.com
> >
> >
> >
>
> –
> Nick Ryan (MVP for DDK)
>
>
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com

Are you not realizing that this is an IRP sent by the I/O Manager not one
of my own IRP’s?

From Max’s prior comment, it sounds to me like I shouldn’t be able to
queue the IRP at all if my dispatch hasn’t returned STATUS_PENDING.

I’m confused about something.

Why?

The following:

IoSetCompletionRoutine()
return IoCallDriver()

will also work.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Nick Ryan”
> Newsgroups: ntdev
> To: “Windows System Software Devs Interest List”
> Sent: Thursday, November 06, 2003 5:12 AM
> Subject: [ntdev] Re: Queuing IRP’s from completion routine
>
>
> > If he’s queueing the IRP in his completion routine and returning
> > STATUS_MORE_PROCESSING_REQUIRED, he definitely must mark the IRP pending
> > and return STATUS_PENDING from his dispatch routine (unless he ensures
> > that the IRP is post-processed and completed in his worker thread or in
> > the dispatch thread before returning).
> >
> > Maxim S. Shatskih wrote:
> >
> > >>I need to do passive or APC-level processing on IRP’s from my completion
> > >>routine. My thought is to return STATUS_MORE_PROCESSING_REQUIRED and
> > >>queue the IRP.
> > >
> > >
> > > This is OK.
> > >
> > >
> > >>But what about pending the IRP? If I understand right, my
> > >>dispatch routine needs to mark the IRP pending and return STATUS_PENDING
> > >>if I’m going to ever queue the IRP, whether at dispatch or completion.
> > >
> > >
> > > No, your dispatch routine needs to call IoCallDriver in order for the IRP
> to go
> > > to the completion routine :slight_smile:
> > > Completion routines are called by lower drivers.
> > >
> > > So, just:
> > >
> > > IoSetCompletionRoutine()
> > > return IoCallDriver()
> > >
> > > in the dispatch routine, and queue in the completion. No marking as
> pending.
> > >
> > > Maxim Shatskih, Windows DDK MVP
> > > StorageCraft Corporation
> > > xxxxx@storagecraft.com
> > > http://www.storagecraft.com
> > >
> > >
> > >
> >
> > –
> > Nick Ryan (MVP for DDK)
> >
> >
> > —
> > Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
> >
> > You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> > To unsubscribe send a blank email to xxxxx@lists.osr.com

Let’s say that your completion routine aborts completion processing by
returning STATUS_MORE_PROCESSING_REQUIRED, also queueing the IRP for
later completion in a worker thread. If your dispatch routine then
returns IoCallDriver() instead of STATUS_PENDING, and if the lower
driver does it work immediately and returns STATUS_SUCCESS from its
dispatch routine, your caller will see STATUS_SUCCESS and will
incorrectly assume that the IRP has been fully completed. His next step
may be to free the IRP, and BSOD ensues.

Maxim S. Shatskih wrote:

Why?

The following:

IoSetCompletionRoutine()
return IoCallDriver()

will also work.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Nick Ryan”
> Newsgroups: ntdev
> To: “Windows System Software Devs Interest List”
> Sent: Thursday, November 06, 2003 5:12 AM
> Subject: [ntdev] Re: Queuing IRP’s from completion routine
>
>
>
>>If he’s queueing the IRP in his completion routine and returning
>>STATUS_MORE_PROCESSING_REQUIRED, he definitely must mark the IRP pending
>>and return STATUS_PENDING from his dispatch routine (unless he ensures
>>that the IRP is post-processed and completed in his worker thread or in
>>the dispatch thread before returning).
>>
>>Maxim S. Shatskih wrote:
>>
>>
>>>>I need to do passive or APC-level processing on IRP’s from my completion
>>>>routine. My thought is to return STATUS_MORE_PROCESSING_REQUIRED and
>>>>queue the IRP.
>>>
>>>
>>>This is OK.
>>>
>>>
>>>
>>>>But what about pending the IRP? If I understand right, my
>>>>dispatch routine needs to mark the IRP pending and return STATUS_PENDING
>>>>if I’m going to ever queue the IRP, whether at dispatch or completion.
>>>
>>>
>>>No, your dispatch routine needs to call IoCallDriver in order for the IRP
>
> to go
>
>>>to the completion routine :slight_smile:
>>>Completion routines are called by lower drivers.
>>>
>>>So, just:
>>>
>>> IoSetCompletionRoutine()
>>> return IoCallDriver()
>>>
>>>in the dispatch routine, and queue in the completion. No marking as
>
> pending.
>
>>>Maxim Shatskih, Windows DDK MVP
>>>StorageCraft Corporation
>>>xxxxx@storagecraft.com
>>>http://www.storagecraft.com
>>>
>>>
>>>
>>
>>–
>>Nick Ryan (MVP for DDK)
>>
>>
>>—
>>Questions? First check the Kernel Driver FAQ at
>
> http://www.osronline.com/article.cfm?id=256
>
>>You are currently subscribed to ntdev as: xxxxx@storagecraft.com
>>To unsubscribe send a blank email to xxxxx@lists.osr.com
>
>
>
>


Nick Ryan (MVP for DDK)

Yes, thanks for correcting me.

The rule must be: - always do if( Irp->PendingReturned )
IoMarkIrpPending(Irp); in a completion routine which is logically coupled with
“return IoCallDriver” sequence, even if this completion returns SMPR.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Nick Ryan”
Newsgroups: ntdev
To: “Windows System Software Devs Interest List”
Sent: Thursday, November 06, 2003 9:35 PM
Subject: [ntdev] Re: Queuing IRP’s from completion routine

> Let’s say that your completion routine aborts completion processing by
> returning STATUS_MORE_PROCESSING_REQUIRED, also queueing the IRP for
> later completion in a worker thread. If your dispatch routine then
> returns IoCallDriver() instead of STATUS_PENDING, and if the lower
> driver does it work immediately and returns STATUS_SUCCESS from its
> dispatch routine, your caller will see STATUS_SUCCESS and will
> incorrectly assume that the IRP has been fully completed. His next step
> may be to free the IRP, and BSOD ensues.
>
> Maxim S. Shatskih wrote:
>
> > Why?
> >
> > The following:
> >
> > IoSetCompletionRoutine()
> > return IoCallDriver()
> >
> > will also work.
> >
> > Maxim Shatskih, Windows DDK MVP
> > StorageCraft Corporation
> > xxxxx@storagecraft.com
> > http://www.storagecraft.com
> >
> >
> > ----- Original Message -----
> > From: “Nick Ryan”
> > Newsgroups: ntdev
> > To: “Windows System Software Devs Interest List”
> > Sent: Thursday, November 06, 2003 5:12 AM
> > Subject: [ntdev] Re: Queuing IRP’s from completion routine
> >
> >
> >
> >>If he’s queueing the IRP in his completion routine and returning
> >>STATUS_MORE_PROCESSING_REQUIRED, he definitely must mark the IRP pending
> >>and return STATUS_PENDING from his dispatch routine (unless he ensures
> >>that the IRP is post-processed and completed in his worker thread or in
> >>the dispatch thread before returning).
> >>
> >>Maxim S. Shatskih wrote:
> >>
> >>
> >>>>I need to do passive or APC-level processing on IRP’s from my completion
> >>>>routine. My thought is to return STATUS_MORE_PROCESSING_REQUIRED and
> >>>>queue the IRP.
> >>>
> >>>
> >>>This is OK.
> >>>
> >>>
> >>>
> >>>>But what about pending the IRP? If I understand right, my
> >>>>dispatch routine needs to mark the IRP pending and return STATUS_PENDING
> >>>>if I’m going to ever queue the IRP, whether at dispatch or completion.
> >>>
> >>>
> >>>No, your dispatch routine needs to call IoCallDriver in order for the IRP
> >
> > to go
> >
> >>>to the completion routine :slight_smile:
> >>>Completion routines are called by lower drivers.
> >>>
> >>>So, just:
> >>>
> >>> IoSetCompletionRoutine()
> >>> return IoCallDriver()
> >>>
> >>>in the dispatch routine, and queue in the completion. No marking as
> >
> > pending.
> >
> >>>Maxim Shatskih, Windows DDK MVP
> >>>StorageCraft Corporation
> >>>xxxxx@storagecraft.com
> >>>http://www.storagecraft.com
> >>>
> >>>
> >>>
> >>
> >>–
> >>Nick Ryan (MVP for DDK)
> >>
> >>
> >>—
> >>Questions? First check the Kernel Driver FAQ at
> >
> > http://www.osronline.com/article.cfm?id=256
> >
> >>You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> >>To unsubscribe send a blank email to xxxxx@lists.osr.com
> >
> >
> >
> >
>
> –
> Nick Ryan (MVP for DDK)
>
>
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com

It’s still not right in this specific case. The problem is that the lower
driver can return something other than STATUS_PENDING, so just doing
return IoCallDriver() could return a non-pending status, regardless of
what your completion handler does.

Meanwhile the completion handler might or might not flob the IRP off to a
worker thread. The rule is that if your completion routine can complete
an IO request on a different thread from that on which it received the
request, then its dispatch routine or IoCompletion routine must call
IoMarkIrpPending, and its dispatch routine must return STATUS_PENDING.

Consider the case where the driver above the OP’s does the following:

IoSetCompletionHandler(…); // sets event on completion

Status = IoCallDriver(…); // to the OP’s driver

if (Status == STATUS_PENDING) {
// we don’t get here if status != STATUS_PENDING, right?
waitForSomeEventToFire();
}
// correct but very bad for the OP’s driver:
SendIrpBackToWhereverItWasAllocatedFrom();
// ouch the OP is off in a worker thread processing this IRP.

In his case he must take the brute force approach in his dispatch routine:

IoMarkIrpPending(…);
IoSetCompletionHandler(…);
(void) IoCallDriver(…);

return STATUS_PENDING;

Anything else will eventually catch the window where his worker thread
runs after the Irp is recycled.

===========================
Mark Roddy
Consultant, Microsoft DDK MVP
Hollis Technology Solutions
xxxxx@hollistech.com
www.hollistech.com
603-321-1032

-----Original Message-----
From: “Maxim S. Shatskih”
To: “Windows System Software Devs Interest List”
Date: Fri, 7 Nov 2003 01:57:41 +0300
Subject: [ntdev] Re: Queuing IRP’s from completion routine

> Yes, thanks for correcting me.
>
> The rule must be: - always do if( Irp->PendingReturned )
> IoMarkIrpPending(Irp); in a completion routine which is logically
> coupled with
> “return IoCallDriver” sequence, even if this completion returns SMPR.
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> ----- Original Message -----
> From: “Nick Ryan”
> Newsgroups: ntdev
> To: “Windows System Software Devs Interest List”
> Sent: Thursday, November 06, 2003 9:35 PM
> Subject: [ntdev] Re: Queuing IRP’s from completion routine
>
>
> > Let’s say that your completion routine aborts completion processing
> by
> > returning STATUS_MORE_PROCESSING_REQUIRED, also queueing the IRP for
> > later completion in a worker thread. If your dispatch routine then
> > returns IoCallDriver() instead of STATUS_PENDING, and if the lower
> > driver does it work immediately and returns STATUS_SUCCESS from its
> > dispatch routine, your caller will see STATUS_SUCCESS and will
> > incorrectly assume that the IRP has been fully completed. His next
> step
> > may be to free the IRP, and BSOD ensues.
> >
> > Maxim S. Shatskih wrote:
> >
> > > Why?
> > >
> > > The following:
> > >
> > > IoSetCompletionRoutine()
> > > return IoCallDriver()
> > >
> > > will also work.
> > >
> > > Maxim Shatskih, Windows DDK MVP
> > > StorageCraft Corporation
> > > xxxxx@storagecraft.com
> > > http://www.storagecraft.com
> > >
> > >
> > > ----- Original Message -----
> > > From: “Nick Ryan”
> > > Newsgroups: ntdev
> > > To: “Windows System Software Devs Interest List”
>
> > > Sent: Thursday, November 06, 2003 5:12 AM
> > > Subject: [ntdev] Re: Queuing IRP’s from completion routine
> > >
> > >
> > >
> > >>If he’s queueing the IRP in his completion routine and returning
> > >>STATUS_MORE_PROCESSING_REQUIRED, he definitely must mark the IRP
> pending
> > >>and return STATUS_PENDING from his dispatch routine (unless he
> ensures
> > >>that the IRP is post-processed and completed in his worker thread
> or in
> > >>the dispatch thread before returning).
> > >>
> > >>Maxim S. Shatskih wrote:
> > >>
> > >>
> > >>>>I need to do passive or APC-level processing on IRP’s from my
> completion
> > >>>>routine. My thought is to return STATUS_MORE_PROCESSING_REQUIRED
> and
> > >>>>queue the IRP.
> > >>>
> > >>>
> > >>>This is OK.
> > >>>
> > >>>
> > >>>
> > >>>>But what about pending the IRP? If I understand right, my
> > >>>>dispatch routine needs to mark the IRP pending and return
> STATUS_PENDING
> > >>>>if I’m going to ever queue the IRP, whether at dispatch or
> completion.
> > >>>
> > >>>
> > >>>No, your dispatch routine needs to call IoCallDriver in order for
> the IRP
> > >
> > > to go
> > >
> > >>>to the completion routine :slight_smile:
> > >>>Completion routines are called by lower drivers.
> > >>>
> > >>>So, just:
> > >>>
> > >>> IoSetCompletionRoutine()
> > >>> return IoCallDriver()
> > >>>
> > >>>in the dispatch routine, and queue in the completion. No marking
> as
> > >
> > > pending.
> > >
> > >>>Maxim Shatskih, Windows DDK MVP
> > >>>StorageCraft Corporation
> > >>>xxxxx@storagecraft.com
> > >>>http://www.storagecraft.com
> > >>>
> > >>>
> > >>>
> > >>
> > >>–
> > >>Nick Ryan (MVP for DDK)
> > >>
> > >>
> > >>—
> > >>Questions? First check the Kernel Driver FAQ at
> > >
> > > http://www.osronline.com/article.cfm?id=256
> > >
> > >>You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> > >>To unsubscribe send a blank email to
> xxxxx@lists.osr.com
> > >
> > >
> > >
> > >
> >
> > –
> > Nick Ryan (MVP for DDK)
> >
> >
> > —
> > Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
> >
> > You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> > To unsubscribe send a blank email to xxxxx@lists.osr.com
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@hollistech.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com

Hate to hijack the thread, but if I could jump in here for clarification on
this…

The rule must be: - always do if( Irp->PendingReturned )
IoMarkIrpPending(Irp); in a completion routine which is logically coupled
with
“return IoCallDriver” sequence, even if this completion returns SMPR.

This means that Walter Oney’s “easy” rule of IoMarkIrpPending on pg. 247 of
his 2nd edition might change from:

“Execute [if (I->PR) IoMPR()] in any completion routine that does not
return STATUS_MORE_PROCESSING_REQUIRED”

to

“Execute [if (I->PR) IoMPR()] in any completion routine that is not
associated with a dispatch routine that synchronizes its execution with a
completion routine that returns STATUS_MORE_PROCESSING_REQUIRED.”

In other words, if a completion routine conditionally returns either SUCCESS
or SMPR, and its associated dispatch routine returns the value of
IoCallDriver(), then the completion routine should always execute “if(I->PR)
IoMIP().”

To get back to the OP question (or, close to it): Is there a standard
technique for conditionally queueing an IRP for asynchronous completion from
a completion routine that doesn’t force the dispatch routine to always
return STATUS_PENDING? It seems like always returning STATUS_PENDING would
be unnecessarily expensive if it’s just infrequently that the completion
routine needs to do some asynchronous post-processing before allowing the
IRP to complete.

Myk

> IoSetCompletionHandler(…); // sets event on completion

Status = IoCallDriver(…); // to the OP’s driver

if (Status == STATUS_PENDING) {
// we don’t get here if status != STATUS_PENDING, right?
waitForSomeEventToFire();
}

This piece of code usually has a very simple completion handler - KeSetEvent
and quit, without the pending flag propagation. So, no problems with it, since
it will not queue the IRP to worker threads :slight_smile:

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

> To get back to the OP question (or, close to it): Is there a standard

technique for conditionally queueing an IRP for asynchronous completion from
a completion routine that doesn’t force the dispatch routine to always
return STATUS_PENDING? It seems like always returning STATUS_PENDING would

One more issue. Driver A:

IoSetCompletionRoutine(CompletionRoutine);
return IoCallDriver();

CompletionRoutine()
{
put to queue
return STATUS_MORE_PROCESSING_REQUIRED;
}

Now let’s imagine the lower driver returns STATUS_SUCCESS. This will cause IO
to call IopCompleteRequest inline, while the IRP is in the queue. A crash.

Propagating the pending flag in the completion cannot solve this. The only
solution is:

IoMarkIrpPending(Irp);
(VOID)IoCallDriver();
return STATUS_PENDING;

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

No Max, it is the lower driver (the OP’s driver) that sometimes queues the
IRP to a worker thread in its completion handler - that was the OP’s
original problem: what to do in his dispatch handler when his completion
handler can sometimes complete the IRP in a different thread. The driver
above the OP’s driver can behave correctly just as I outlined, and in doing
so the system can crash if, as you suggested, the OP’s driver simply returns
the status from IoCallDriver.

=====================
Mark Roddy

-----Original Message-----
From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
Sent: Friday, November 07, 2003 8:46 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Re: Queuing IRP’s from completion routine

> IoSetCompletionHandler(…); // sets event on completion
>
> Status = IoCallDriver(…); // to the OP’s driver
>
> if (Status == STATUS_PENDING) {
> // we don’t get here if status != STATUS_PENDING, right?
> waitForSomeEventToFire();
> }

This piece of code usually has a very simple completion
handler - KeSetEvent and quit, without the pending flag
propagation. So, no problems with it, since it will not queue
the IRP to worker threads :slight_smile:

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as:
xxxxx@stratus.com To unsubscribe send a blank email to
xxxxx@lists.osr.com

> IRP to a worker thread in its completion handler - that was the OP’s

original problem: what to do in his dispatch handler when his completion
handler can sometimes complete the IRP in a different thread. The driver

Then looks like this:

IoMarkIrpPending(Irp);
IoSetCompletionRoutine(Irp, CompletionWhichWillPutToQueue…);
(VOID)IoCallDriver(…);
return STATUS_PENDING;

is the only solution.

Anyway - is it a much speed-up to ever returning non-pending status from the
dispatch routine?

Well, KeInsertQueueApc will be called for the IRP, but inline to the same
thread we are running on, if this is on < DISPATCH_LEVEL as usual for such
routines - then KeInsertQueueApc will degrade to a function call framed by
KeRaise/Lower to APC_LEVEL.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

Maxim S. Shatskih wrote:

Anyway - is it a much speed-up to ever returning non-pending status
from the
> dispatch routine?

I think the performance hit comes when (practically inevitably) the
driver above you waits on the completion event when it sees your
STATUS_PENDING, thus switching their thread out. The dispatcher lock is
pretty hot in the OS (and it’s hit both by the wait and the SetEvent).
Probably good to avoid if it’s not necessary (and assuming it’s
possible… time to dig up Adrian’s presentation from WinHEC :-).

Speaking of which, the answer to most questions of this nature
(including this one) can usually be found at:

http://support.microsoft.com/default.aspx?scid=kb;en-us;320275

Adrian’s (wonderfully informative) WinHEC presentation, which explains
the reasons for most of the above scenarios, can be found at:

http://download.microsoft.com/download/c/f/1/cf1806ad-5a4f-4f7d-a5b2-07fdb59a7adb/WH03_DDT22.exe

Now, in the OP’s particular situation (which admittedly is somewhat
vague), he would have to return STATUS_PENDING because the IRP will be
completed in another thread. If he doesn’t return STATUS_PENDING, the
IRP could be freed before his thread is ever scheduled (as has been
mentioned here before).

…/ray..

> Then looks like this:

IoMarkIrpPending(Irp);
IoSetCompletionRoutine(Irp, CompletionWhichWillPutToQueue…);
(VOID)IoCallDriver(…);
return STATUS_PENDING;

is the only solution.

Anyway - is it a much speed-up to ever returning non-pending status from the
dispatch routine?

Well, KeInsertQueueApc will be called for the IRP, but inline to the same
thread we are running on, if this is on < DISPATCH_LEVEL as usual for such
routines - then KeInsertQueueApc will degrade to a function call framed by
KeRaise/Lower to APC_LEVEL.

I tend to agree with Maxim Shatskih that the pain of avoiding the
unnecessary STATUS_PENDING return (unnecessary APC) is not worth the
gain.

However,

If someone is determined to do it than it seems to me that the issues
are as follows:

  1. If the completion routine queues the IRP it must return
    STATUS_MORE_PRO… to indicate that it is withholding completion. As
    always, it is not required to call
    IoMarkIrpPending() but it may. Ultimately, this depends on whether or
    not it eventually completes the request from the callers thread
    (thread of the dispatch routine) or some other thread. The usual
    technique here is to signal
    an event to tell the dispatch routine to get on with the job of
    completing the request. However, in this case, we have queued a worker
    thread to finish the job.

  2. When a worker thread is scheduled, the dispatch routine must do 1
    of 2 things. It must either return STATUS_PENDING or wait for the
    worker thread to finish. The easy way would be to have the dispatch
    routine waiting on an event and have either the completion routine (if
    it finishes the task) or the worker thread signal the event. The
    dispatch routine then completes the request. In this case
    STATUS_PENDING is never returned.

  3. It sounds like OP would really like to not have the dispatch
    routine wait unnecessarily for completion before returning. To do this
    it seems that OP should take the following steps:

a) Dispatch calls IoCallDriver()

b) if IoCallDriver() returns STATUS_PENDING than dispatch returns
STATUS_PENDING. It makes no difference whether or not the completion
routine will eventually complete or schedule a worker thread to
complete.

c) if IoCallDriver() does not return STATUS_PENDING than we know
that our completion routine has already been called and we
merely need to check a flag (set in the completion routine) to know
whether or not a worker thread was scheduled. If there is no worker
thread than dispatch returns the result of IoCallDriver(),
otherwise return STATUS_PENDING.

d) If a worker thread is scheduled either the completion routine or
the worker thread must take responsibility for propagating any
pending return from lower drivers with IoMarkIrpPending(). The
worker thread, if used, always calls IoCompleteRequest() when it is
done.