APC not being invoked for ZwReadFile

NTFSD
1

I have a loop that invokes ZwReadFile multiple times in a loop, each time
specifying an ApcRoutine for completion notification. After submitting all
of the reads, I wait on an event that is signalled by the last APC to
execute (the ApcContext contains a pointer to a shared variable used to
track outstanding requests).

What I’ve found is that my APC is sometimes not invoked when a read
completes, even though ZwReadFile returns STATUS_SUCCESS. In these cases,
the IO_STATUS_BLOCK will have been filled out correctly
(Status=STATUS_SUCCESS, Information=expected read size), but the APC has not
been (and never will be, apparently) invoked.

Are there situations in which ZwReadFile will not invoke an APC upon
completion? Maybe there’s some path through FastIo that skips the queueing
of the APC?

Or maybe I’m doing something else wrong. A chopped down version of my code
is below.

Thanks, as always, for any suggestions.

Myk

XxxReadFileApc( )
{
context->Invoked = TRUE;
if ( InterlockedDecrement( context->ReadsOutstanding ) == 0 )
{
KeSetEvent( context->Event, 0, FALSE );
}
return STATUS_SUCCESS;
}

DoMultipleReadsAtPassiveLevel()
{
LONG readsOutstanding = 1;
KEVENT event; KeInitializeEvent( &event, NotificationEvent,
FALSE );

for ( each range to read )
{
ApcContext = ExAllocatePoolWithTag( … );
ApcContext->ReadsOutstanding = &readsOutstanding;
ApcContext->Invoked = FALSE;
ApcContext->Event = &event;

InterlockedIncrement( &readsOutstanding );

status = ZwReadFile(
FileHandle,
NULL, // event
XxxReadFileApc,
ApcContext, // context
&ApcContext->IoStatusBlock, …
);
if ( !NT_SUCCESS(status) )
{
InterlockedDecrement( &readsOutstanding );
}

} // for (each section)

if ( InterlockedDecrement( &readsOutstanding ) != 0 )
{
// at this point, we will (sometimes) hang forever with
readsOutstanding stuck
// at a non-zero value. Inspection of the APC context structures
shows that
// sometimes one has not been invoked (Invoked==FALSE), even though
its
// associated IO_STATUS_BLOCK looks like it completed successfully.
KeWaitForSingleObject( &event, UserRequest, KernelMode, FALSE,
NULL );
}
// …
}

2

APC can’t be delivered to the thread is you specified WaitMode == KernelMode
and Alterable == FALSE.
Try to set WaitMode == UserMode and Alterable == TRUE in
KeWaitForSingleObject.

Alexei.

“Myk Willis” wrote in message news:xxxxx@ntfsd…
>
> I have a loop that invokes ZwReadFile multiple times in a loop, each time
> specifying an ApcRoutine for completion notification. After submitting
all
> of the reads, I wait on an event that is signalled by the last APC to
> execute (the ApcContext contains a pointer to a shared variable used to
> track outstanding requests).
>
> What I’ve found is that my APC is sometimes not invoked when a read
> completes, even though ZwReadFile returns STATUS_SUCCESS. In these cases,
> the IO_STATUS_BLOCK will have been filled out correctly
> (Status=STATUS_SUCCESS, Information=expected read size), but the APC has
not
> been (and never will be, apparently) invoked.
>
> Are there situations in which ZwReadFile will not invoke an APC upon
> completion? Maybe there’s some path through FastIo that skips the
queueing
> of the APC?
>
> Or maybe I’m doing something else wrong. A chopped down version of my
code
> is below.
>
> Thanks, as always, for any suggestions.
>
> Myk
>
>
> XxxReadFileApc( )
> {
> context->Invoked = TRUE;
> if ( InterlockedDecrement( context->ReadsOutstanding ) == 0 )
> {
> KeSetEvent( context->Event, 0, FALSE );
> }
> return STATUS_SUCCESS;
> }
>
> DoMultipleReadsAtPassiveLevel()
> {
> LONG readsOutstanding = 1;
> KEVENT event; KeInitializeEvent( &event, NotificationEvent,
> FALSE );
>
> for ( each range to read )
> {
> ApcContext = ExAllocatePoolWithTag( … );
> ApcContext->ReadsOutstanding = &readsOutstanding;
> ApcContext->Invoked = FALSE;
> ApcContext->Event = &event;
>
> InterlockedIncrement( &readsOutstanding );
>
> status = ZwReadFile(
> FileHandle,
> NULL, // event
> XxxReadFileApc,
> ApcContext, // context
> &ApcContext->IoStatusBlock, …
> );
> if ( !NT_SUCCESS(status) )
> {
> InterlockedDecrement( &readsOutstanding );
> }
>
> } // for (each section)
>
> if ( InterlockedDecrement( &readsOutstanding ) != 0 )
> {
> // at this point, we will (sometimes) hang forever with
> readsOutstanding stuck
> // at a non-zero value. Inspection of the APC context structures
> shows that
> // sometimes one has not been invoked (Invoked==FALSE), even
though
> its
> // associated IO_STATUS_BLOCK looks like it completed
successfully.
> KeWaitForSingleObject( &event, UserRequest, KernelMode, FALSE,
> NULL );
> }
> // …
> }
>
>
>
>

3

Actually, according to the article “Do Waiting Threads Receive Alerts
and APCs?” in the IFSKIT, those two parameters don’t affect whether a
kernel-mode APC is delivered and executed. The article also specifies
three conditions necessary for a normal kernel-mode APC to fire:

“IRQL < APC_LEVEL, thread not already in an APC, thread not in a
critical section”.

Alexei Jelvis wrote:

APC can’t be delivered to the thread is you specified WaitMode == KernelMode
and Alterable == FALSE.
Try to set WaitMode == UserMode and Alterable == TRUE in
KeWaitForSingleObject.

Alexei.

“Myk Willis” wrote in message news:xxxxx@ntfsd…
>
>>I have a loop that invokes ZwReadFile multiple times in a loop, each time
>>specifying an ApcRoutine for completion notification. After submitting
>
> all
>
>>of the reads, I wait on an event that is signalled by the last APC to
>>execute (the ApcContext contains a pointer to a shared variable used to
>>track outstanding requests).
>>
>>What I’ve found is that my APC is sometimes not invoked when a read
>>completes, even though ZwReadFile returns STATUS_SUCCESS. In these cases,
>>the IO_STATUS_BLOCK will have been filled out correctly
>>(Status=STATUS_SUCCESS, Information=expected read size), but the APC has
>
> not
>
>>been (and never will be, apparently) invoked.
>>
>>Are there situations in which ZwReadFile will not invoke an APC upon
>>completion? Maybe there’s some path through FastIo that skips the
>
> queueing
>
>>of the APC?
>>
>>Or maybe I’m doing something else wrong. A chopped down version of my
>
> code
>
>>is below.
>>
>>Thanks, as always, for any suggestions.
>>
>>Myk
>>
>>
>>XxxReadFileApc( )
>>{
>> context->Invoked = TRUE;
>> if ( InterlockedDecrement( context->ReadsOutstanding ) == 0 )
>> {
>> KeSetEvent( context->Event, 0, FALSE );
>> }
>> return STATUS_SUCCESS;
>>}
>>
>>DoMultipleReadsAtPassiveLevel()
>>{
>> LONG readsOutstanding = 1;
>> KEVENT event; KeInitializeEvent( &event, NotificationEvent,
>>FALSE );
>>
>> for ( each range to read )
>> {
>> ApcContext = ExAllocatePoolWithTag( … );
>> ApcContext->ReadsOutstanding = &readsOutstanding;
>> ApcContext->Invoked = FALSE;
>> ApcContext->Event = &event;
>>
>> InterlockedIncrement( &readsOutstanding );
>>
>> status = ZwReadFile(
>> FileHandle,
>> NULL, // event
>> XxxReadFileApc,
>> ApcContext, // context
>> &ApcContext->IoStatusBlock, …
>> );
>> if ( !NT_SUCCESS(status) )
>> {
>> InterlockedDecrement( &readsOutstanding );
>> }
>>
>> } // for (each section)
>>
>> if ( InterlockedDecrement( &readsOutstanding ) != 0 )
>> {
>> // at this point, we will (sometimes) hang forever with
>>readsOutstanding stuck
>> // at a non-zero value. Inspection of the APC context structures
>>shows that
>> // sometimes one has not been invoked (Invoked==FALSE), even
>
> though
>
>>its
>> // associated IO_STATUS_BLOCK looks like it completed
>
> successfully.
>
>> KeWaitForSingleObject( &event, UserRequest, KernelMode, FALSE,
>>NULL );
>> }
>>// …
>>}
>>
>>
>>
>>
>
>
>
>
>

–
Nick Ryan (MVP for DDK)

4

Alexei -

APC can’t be delivered to the thread is you specified WaitMode ==
KernelMode
and Alterable == FALSE.
Try to set WaitMode == UserMode and Alterable == TRUE in
KeWaitForSingleObject.

I’m invoking ZwReadFile from a system work item that was queued by a
completion routine in my file system filter driver. It was my understanding
that the Alertable parameter had no effect on the delivery of kernel-mode
APCs, and since I’m a kernel-mode caller, I assumed that my routine was
going to be called as a kernel-mode APC. From the docs:
“The delivery of kernel-mode APCs to a waiting thread does not depend on
whether the thread can be alerted. If the kernel-mode APC is a special
kernel-mode APC, then the APC is delivered provided that IRQL < APC_LEVEL.
If the kernel-mode APC is a normal kernel-mode APC, then the APC is
delivered provided that the following three conditions hold: (1) IRQL <
APC_LEVEL, (2) no kernel APC is in progress, and (3) the thread is not in a
critical section.”

Maybe I’m wrong in believing my APC will be invoked as a kernel-mode APC?

And as for UserMode/KernelMode: the system thread has no user mode context
(as far as I know), so what would it do to specify UserMode wait state aside
from allow the stack to be pagable?

It sure could be the case that this works “most of the time” because
normally the reads complete before I call KeWaitForSingleObject. So maybe
the APCs aren’t being delivered once I enter the wait. But I still don’t
see why that would be the case given the documentation.

Myk

“Alexei Jelvis” wrote in message news:xxxxx@ntfsd…

>
> APC can’t be delivered to the thread is you specified WaitMode ==
KernelMode
> and Alterable == FALSE.
> Try to set WaitMode == UserMode and Alterable == TRUE in
> KeWaitForSingleObject.
>
> Alexei.
>
> “Myk Willis” wrote in message news:xxxxx@ntfsd…
> >
> > I have a loop that invokes ZwReadFile multiple times in a loop, each
time
> > specifying an ApcRoutine for completion notification. After submitting
> all
> > of the reads, I wait on an event that is signalled by the last APC to
> > execute (the ApcContext contains a pointer to a shared variable used to
> > track outstanding requests).
> >
> > What I’ve found is that my APC is sometimes not invoked when a read
> > completes, even though ZwReadFile returns STATUS_SUCCESS. In these
cases,
> > the IO_STATUS_BLOCK will have been filled out correctly
> > (Status=STATUS_SUCCESS, Information=expected read size), but the APC has
> not
> > been (and never will be, apparently) invoked.
> >
> > Are there situations in which ZwReadFile will not invoke an APC upon
> > completion? Maybe there’s some path through FastIo that skips the
> queueing
> > of the APC?
> >
> > Or maybe I’m doing something else wrong. A chopped down version of my
> code
> > is below.
> >
> > Thanks, as always, for any suggestions.
> >
> > Myk
> >
> >
> > XxxReadFileApc( )
> > {
> > context->Invoked = TRUE;
> > if ( InterlockedDecrement( context->ReadsOutstanding ) == 0 )
> > {
> > KeSetEvent( context->Event, 0, FALSE );
> > }
> > return STATUS_SUCCESS;
> > }
> >
> > DoMultipleReadsAtPassiveLevel()
> > {
> > LONG readsOutstanding = 1;
> > KEVENT event; KeInitializeEvent( &event, NotificationEvent,
> > FALSE );
> >
> > for ( each range to read )
> > {
> > ApcContext = ExAllocatePoolWithTag( … );
> > ApcContext->ReadsOutstanding = &readsOutstanding;
> > ApcContext->Invoked = FALSE;
> > ApcContext->Event = &event;
> >
> > InterlockedIncrement( &readsOutstanding );
> >
> > status = ZwReadFile(
> > FileHandle,
> > NULL, // event
> > XxxReadFileApc,
> > ApcContext, // context
> > &ApcContext->IoStatusBlock, …
> > );
> > if ( !NT_SUCCESS(status) )
> > {
> > InterlockedDecrement( &readsOutstanding );
> > }
> >
> > } // for (each section)
> >
> > if ( InterlockedDecrement( &readsOutstanding ) != 0 )
> > {
> > // at this point, we will (sometimes) hang forever with
> > readsOutstanding stuck
> > // at a non-zero value. Inspection of the APC context
structures
> > shows that
> > // sometimes one has not been invoked (Invoked==FALSE), even
> though
> > its
> > // associated IO_STATUS_BLOCK looks like it completed
> successfully.
> > KeWaitForSingleObject( &event, UserRequest, KernelMode, FALSE,
> > NULL );
> > }
> > // …
> > }
> >
> >
> >
> >
>
>
>
>