KeWaitForSingleObject and Apc

Ravi_Agrawal · December 14, 2015, 11:16am

I know many similar question is being asked in the forum and various MSDN doc referred. Could not conclude

In one of the legacy WDM driver I find following function is DeviceIoControl(custom user code)
KeWaitForSingleObject()
Object= Kernel Mutex
WaitReason=Executive
WaitMode,=KERNEL
BOOLEAN=FALSE
Timeout=NULL

* My understanding Normal kernel APC and Special Kernel APC can be executed on this thread
* Does post Kewait. Can this thread be suspended and later raise to APC?
Is there a way to reproduce this in Kernel Mode?
Dump question can a User program take over such thread?

Specifies the reason for the wait. A driver should set this value to Executive, unless it is doing work on behalf of a user and is running in the context of a user thread, in which case it should set this value to UserRequest.
What is risk in setting to Executive in user context?

I am going through some documents to understand it better. Above question crossed my mind as I was walking through some legacy code

anton_bassov · December 14, 2015, 11:33am

There was just a wonderful document available on MSFT site some years ago - this doc provided answers to all your questions in the form of a table, with parameters to KeWaitXXXX() and APC types at Passive and APC IRQLs presented as respectively columns and rows. There is a good chance that one is still able to find it somewhere on MSFT site…

Anton Bassov

Ravi_Agrawal · December 14, 2015, 11:59am

From Locks Doc:-
Alertable = FALSE
WaitMode = KernelMode No If (A), then Yes No If (B), then Yes No No
*A: IRQL < APC_LEVEL.
**B: IRQL < APC_LEVEL, thread not already in an APC, and thread not in a critical region.

Later in
15 Security Issues
Drivers that use locks at IRQL PASSIVE_LEVEL outside a critical region are open to denial of service attacks if the thread that holds the lock is suspended. This problem occurs because Windows queues a normal kernel-mode APC to suspend the thread. Even if the driver specifies a KernelMode wait, normal kernel-mode APCs are delivered whenever all of the following are true:
? The target thread is running at IRQL < APC_LEVEL.
? The target thread is not already running an APC.
? The target thread is not in a critical region; that is, it did not call KeEnterCriticalRegion before calling KeWaitXxx.

Note
Kernel mutexes and fast mutexes do not have this problem. The operating system enters a critical region before it acquires a fast mutex or kernel mutex on behalf of a thread.

Ravi:- From the table it seems Normal and Special APC are enabled. Later part indicates the Normal APC is disabled for specifically kernel mutex.
Which gives the impression that for Kernel Mutex only special kernel APC is enabled. Please correct me

anton_bassov · December 14, 2015, 12:16pm

> Ravi:- From the table it seems Normal and Special APC are enabled. Later part

indicates the Normal APC is disabled for specifically kernel mutex. Which gives
the impression that for Kernel Mutex only special kernel APC is enabled. Please correct me

Just read it again:

Once "the operating system enters a critical region before it acquires a fast mutex or kernel mutex "
it raises IRQL to APC_LEVEL which disables APC delivery to the target thread …

Anton Bassov

Ravi_Agrawal · December 14, 2015, 12:53pm

Thanks Anton for quick reply

KeWait:- KMUTEX

There I have a contradiction with the experiment. Post KeWait I find !irql on a given thread is PASSIVE_LEVEL. OS:- 2k8R2

kd> !irql
Debugger saved IRQL for processor 0x0 – 0 (LOW_LEVEL)

Ravi_Agrawal · December 14, 2015, 1:01pm

From Locks:-
Special kernel-mode APCs, such as the special kernel-mode APC for I/O completion, can still be delivered. Internally, acquiring a kernel mutex calls KeEnterCriticalRegion. If the thread is running at PASSIVE_LEVEL when it acquires the mutex, this call disables the delivery of normal kernel-mode APCs until the thread releases the mutex. If the thread is running at APC_LEVEL when it acquires the mutex, entering a critical region has no effect because normal kernel-mode APC delivery is already disabled.

Ravi:- It explains why !irql is giving PASSIVE
Is there a way to queue Special Kernel Mode APC to a given thread? Will it raise the IRQL to APC post Wait?

anton_bassov · December 14, 2015, 1:04pm

>There I have a contradiction with the experiment.Post KeWait I find !irql on a given

thread is PASSIVE_LEVEL. OS:- 2k8R2

Well, I am not particularly surprised about it, especially taking into consideration that the quotation speaks about the mutex objects described by FAST_MUTEX and KGUARDED_MUTEX structures
that have to be acquired by respectively KeAcquireFastMutex() and KeAcquireGuardedMutex() functions…

Anton Bassov

Ravi_Agrawal · December 14, 2015, 1:27pm

From document Special Kernel Mode APC interrupt are enabled.

When this interrupt is served? Immediately post wait or any time

Post completion of Special Kernel Function…the IRQL is lowered to original i.e. PASSIVE_LEVEL

Scott_Noone_OSR · December 14, 2015, 1:47pm

Special Kernel Mode APCs can be disabled either by raising to APC_LEVEL or
by entering a *Guarded* Region (which simply modifies a field of the
KTHREAD). Guarded Regions were added to Server 2003 and are now more
commonly used than APC_LEVEL.

Normal and Special KAPCs can be delivered while the thread is running, they
do not require that the thread be in a wait state (hence the potential
denial of service)

https://msdn.microsoft.com/en-us/library/windows/hardware/ff564853(v=vs.85).aspx

The APIs to queue any type of KAPC are undocumented.

(While the original question about thread suspension has already been
answered, I’ll also again refer to our recent article about Critical
Regions: https://www.osr.com/nt-insider/2015-issue3/critical-regions/)

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntdev…

From document Special Kernel Mode APC interrupt are enabled.

When this interrupt is served? Immediately post wait or any time

Post completion of Special Kernel Function…the IRQL is lowered to original
i.e. PASSIVE_LEVEL

Ravi_Agrawal · December 14, 2015, 2:24pm

Thanks for the link.

I wanted to confirm in case of Kernel Mutex
* Is there a threat of denial of service by another kernel thread because special kernel APC is enabled?
* Post the interrupt operation in special kernel APC, Does it restore the original irql?
There is a legacy code which assume it is always at passive level as it is from DeviceIOCTL

I agree fix is to disable Special APC by entering Guarded Region

Ravi_Agrawal · December 14, 2015, 3:30pm

Thanks Every I got the answer

Post the interrupt operation in special kernel APC, Does it restore the
original irql?
Yes

Scott_Noone_OSR · December 14, 2015, 3:31pm

Well, yes and no…Your thinking is correct that a SKAPC could run while the
thread has a kernel mutex is acquired. However, SKAPCs can only be sent from
kernel mode. Because there is no security boundary between two kernel mode
components, it’s not a security issue.

The IRQL is restored after the SKAPC.

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntdev…

Thanks for the link.

I wanted to confirm in case of Kernel Mutex
* Is there a threat of denial of service by another kernel thread because
special kernel APC is enabled?
* Post the interrupt operation in special kernel APC, Does it restore the
original irql?
There is a legacy code which assume it is always at passive level as
it is from DeviceIOCTL

I agree fix is to disable Special APC by entering Guarded Region

Maxim_S_Shatskih · December 15, 2015, 7:46am

> WaitMode,=KERNEL

BOOLEAN=FALSE

No, such wait cannot be interrupted by any APC, including the TerminateThread/Process path (which works by sending a specially crafted APC to a thread).

Specifies the reason for the wait. A driver should set this value to Executive,

I think this is just auxiliary value only purposed for debugging tools printout, and not to influence any behavior.

What is risk in setting to Executive in user context?

No risk.

–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

Maxim_S_Shatskih · December 15, 2015, 7:49am

> There was just a wonderful document available on MSFT site some years ago - this doc provided

answers to all your questions in the form of a table, with parameters to KeWaitXXXX() and APC
types at Passive and APC IRQLs presented as respectively columns and rows.

Yes, and IIRC one of the combinations only allows the termination APC to interrupt the wait, in which case you will get STATUS_USER_APC retval from KeWait

Anyway: if you pass something other then (WaitMode=KernelMode, Alertable=FALSE) to KeWait, you must be prepared for getting STATUS_ALERTED or STATUS_USER_APC from KeWait, in which case your code must return from all your paths ASAP, to get to Io and then the syscall epilog path ASAP.

It is the syscall epilog which will actually call the APC.

–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

Maxim_S_Shatskih · December 15, 2015, 7:52am

> Is there a way to queue Special Kernel Mode APC to a given thread?

I think IopCompleteRequest is the only Special Kernel APC ever used.

Why do you need APCs to your thread?

–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

Maxim_S_Shatskih · December 15, 2015, 7:55am

> Normal and Special KAPCs can be delivered while the thread is running, they

do not require that the thread be in a wait state (hence the potential
denial of service)

I think any APC in kmode can be delivered (i.e. the APC function being called) only in a) wait OR b) syscall epilog.

The normal execution flow of the thread cannot be interrupted by the APC, neither in kmode nor in umode (where you need an alertable wait for APC to be delivered).

This is also true on thread termination APC.

Am I wrong?

–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

anton_bassov · December 15, 2015, 8:44am

> I think any APC in kmode can be delivered (i.e. the APC function being

called) only in a) wait OR b) syscall epilog.

Have you ever asked yourself a question about the very purpose of IRQL APC_LEVEL??? Look - if the things worked the way described above there would be simply no need for it, in the first place.
What you are saying is true for the Windows userland which does not have the concept of
IRQL - unlike UNIX-like systemsWindows does not support the concept of asynch signal delivery which, in order to be functional, requires the possibility of a thread disabling signal delivery to itself (i.e the concept that is implemented as IRQL==APC_LEVEL in Windows kernel). Under Windows a user APC can be delivered to a thread only when the target thread makes a blocking call, so that there is simply no need for giving a thread a possibility of disabling APC delivery to itself…

Anton Bassov

Scott_Noone_OSR · December 15, 2015, 10:28am

As long as the CombinedApcDisable field in the KTHREAD is zero the kernel
APCs can be delivered. I forced it to happen in a VM to get a call stack by
putting the thread into an infinite loop and then suspending it using
Process Explorer. Here are the steps and the resulting call stack:

kd> bp Nothing!NothingRead
kd> g
Nothing!NothingRead:
fffff801`66bf61b0 4889542410 mov qword ptr [rsp+10h],rdx
kd> ??@$thread->Tcb.CombinedApcDisable
unsigned long 0
kd> ew @$ip 0xFEEB
kd> g

Thread is now taking up 100% CPU. Now suspend the thread using Process
Explorer and check the call stack:

kd> k
*** Stack trace for last set context - .thread/.cxr resets it

Child-SP RetAddr Call Site

00 ffffd0018ebc64a0 fffff8033d250d7e nt!KiSwapContext+0x76
01 ffffd0018ebc65e0 fffff8033d2507f9 nt!KiSwapThread+0x14e
02 ffffd0018ebc6680 fffff8033d2788d0 nt!KiCommitThreadWait+0x129
03 ffffd0018ebc6700 fffff8033d24d64c nt!KeWaitForSingleObject+0x2c0
04 ffffd0018ebc6790 fffff8033d24e279 nt!KiSchedulerApc+0x78
05 ffffd0018ebc67f0 fffff8033d374723 nt!KiDeliverApc+0x209
06 ffffd0018ebc6870 fffff80166bf61b0 nt!KiApcInterrupt+0xc3
07 ffffd0018ebc6a08 fffff8033d605788 Nothing!NothingRead
08 ffffd0018ebc6a10 fffff8033d603336 nt!IopSynchronousServiceTail+0x170
09 ffffd0018ebc6ae0 fffff8033d37b1b3 nt!NtReadFile+0x656
0a ffffd0018ebc6bd0 00007ffc38420caa nt!KiSystemServiceCopyEnd+0x13
0b 0000009dfd31fa28 00007ffc358f83a8 ntdll!NtReadFile+0xa

-scott
OSR
@OSRDrivers

“Maxim S. Shatskih” wrote in message news:xxxxx@ntdev…

Normal and Special KAPCs can be delivered while the thread is running,
they
do not require that the thread be in a wait state (hence the potential
denial of service)

I think any APC in kmode can be delivered (i.e. the APC function being
called) only in a) wait OR b) syscall epilog.

The normal execution flow of the thread cannot be interrupted by the APC,
neither in kmode nor in umode (where you need an alertable wait for APC to
be delivered).

This is also true on thread termination APC.

Am I wrong?

–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com