Unexpected RESOURCE_NOT_OWNED on hibernation request

Hi,

i was developing a helper application in .NET that is using the System.Diagnostic-Namespaces Process-Class for terminating a specific process if it had to be terminated forcibly and no other “soft” way did work. The Process-Class does have a member named Kill() that internally expects a process id and internally forwards this to the Kernel32::TerminateProcess by using the processes handle and a “-1” as second parameter. I accidentally managed to include the SYSTEM process into the list of processes (thats what i found out later by analysing my code) to be terminated but the termination of the SYSTEM process failed for sure, but lead to a unexpected BSOD on systems hibernation request. At first it wasnt clear from where i got the BSOD since all worked fine yet and i never got the RESOURCE_NOT_OWNED before on that system. As said i found out that this was in direct relation to a failing TerminateProcess() request on the SYSTEM process on my Vista 32bit system. While the system is working nothing happens and all runs fine, but if you request the session written to disk with hibernation, the BSOD appears right after the display turns dark for hibernation attempt. Thats what the Bugcheck in detail shows, The terminator application is a win32 app that is elevated and has debug privilege (for other tasks it can carry out on demand).

Microsoft (R) Windows Debugger Version 6.12.0002.633 X86
Copyright (c) Microsoft Corporation. All rights reserved.

Loading Dump File [C:\Windows\MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available

Symbol search path is: srv*c:\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows Server 2008/Windows Vista Kernel Version 6002 (Service Pack 2) MP (2 procs) Free x86 compatible
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 6002.18533.x86fre.vistasp2_gdr.111025-0338
Machine Name:
Kernel base = 0x82202000 PsLoadedModuleList = 0x82319c70
Debug session time: Thu Mar 15 07:06:36.670 2012 (UTC + 1:00)
System Uptime: 0 days 0:15:40.700
Loading Kernel Symbols



Loading User Symbols

Loading unloaded module list

*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck E3, {823043a0, 84fb4ad0, 0, 2}

Probably caused by : ntkrpamp.exe ( nt!ExpReleaseResourceForThreadLite+d8 )

Followup: MachineOwner

0: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

RESOURCE_NOT_OWNED (e3)
A thread tried to release a resource it did not own.
Arguments:
Arg1: 823043a0, Address of resource
Arg2: 84fb4ad0, Address of thread
Arg3: 00000000, Address of owner table if there is one
Arg4: 00000002

Debugging Details:

DEFAULT_BUCKET_ID: VISTA_DRIVER_FAULT

BUGCHECK_STR: 0xE3

PROCESS_NAME: System

CURRENT_IRQL: 2

LAST_CONTROL_TRANSFER: from 822ad792 to 822cfb3f

STACK_TEXT:
8b957a04 822ad792 000000e3 823043a0 84fb4ad0 nt!KeBugCheckEx+0x1e
8b957a34 822ad6b3 00000001 8b957ab8 823b34c8 nt!ExpReleaseResourceForThreadLite+0xd8
8b957a40 823b34c8 00000000 824db7bc 8b957ad8 nt!ExReleaseResourceLite+0xf
8b957a48 824db7bc 8b957ad8 00000001 00000000 nt!ExReleaseTimeRefreshLock+0xd
8b957aa4 824db413 8b957ab8 00000003 00000001 nt!PoBroadcastSystemState+0x398
8b957ad8 824dc771 3f958e03 8b957c10 8b957c94 nt!PopSetDevicesSystemState+0x7b
8b957bfc 8224cc7a 00000002 c000010a 01000000 nt!NtSetSystemPowerState+0x6c5
8b957bfc 8224bb9d 00000002 c000010a 01000000 nt!KiFastCallEntry+0x12a
8b957c80 824903eb 00000002 00000005 80000000 nt!ZwSetSystemPowerState+0x11
8b957ccc 8248fd7e 00000000 00000002 00000005 nt!PopIssueActionRequest+0x352
8b957d08 822328ce 3f9588bb 8230413c 84fb4ad0 nt!PopPolicyWorkerAction+0x45
8b957d44 822a7e22 00000004 00000000 84fb4ad0 nt!PopPolicyWorkerThread+0x6e
8b957d7c 823d7fe2 80000000 3f95883f 00000000 nt!ExpWorkerThread+0xfd
8b957dc0 82240efe 822a7d25 00000001 00000000 nt!PspSystemThreadStartup+0x9d
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

STACK_COMMAND: kb

FOLLOWUP_IP:
nt!ExpReleaseResourceForThreadLite+d8
822ad792 cc int 3

SYMBOL_STACK_INDEX: 1

SYMBOL_NAME: nt!ExpReleaseResourceForThreadLite+d8

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: nt

IMAGE_NAME: ntkrpamp.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 4ea6b87e

FAILURE_BUCKET_ID: 0xE3_nt!ExpReleaseResourceForThreadLite+d8

BUCKET_ID: 0xE3_nt!ExpReleaseResourceForThreadLite+d8

Followup: MachineOwner

So my question here is: If i dont have termination access to the system process, why does the TerminateProcess() “affect” the SYSTEM process in some way or why does the TerminateProcess() lead to some kind of change in either the SYSTEM process or some Kernel data structures?

K.

Do you have SE_DEBUG_PRIVILEGE?

It doesn’t really matter. He put the system into an unsupported, unconfigured state. Random crashes as a result are not surprising.

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@broadcom.com
Sent: Thursday, March 15, 2012 10:06 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Unexpected RESOURCE_NOT_OWNED on hibernation request

Do you have SE_DEBUG_PRIVILEGE?


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

On 15-Mar-2012 08:40, xxxxx@arcor.de wrote:

As said i found out that this was in direct
relation to a failing TerminateProcess() request on the SYSTEM
process on my Vista 32bit system.

Can you repro this on Win7 SP1?
Vista is pretty dead IMHO; even if this is a genuine bug, unlikely it
will be addressed. Try to repro on a clean system. If you have something
suspicious installed (especially, of your own brew) it could cause this.
– pa

>Do you have SE_DEBUG_PRIVILEGE?

Yes, its part of the applications privileges.

He put the system into an unsupported, unconfigured state. Random crashes as a result are not
surprising.

I guess this is what happens, but if i dont have terminate access to the SYSTEM process, so WHY does this leave “something” changed?

Can you repro this on Win7 SP1?

Didnt try yet, but i will,…

Vista is pretty dead IMHO; even if this is a genuine bug, unlikely it will be addressed.

IMHO there are still many many Vista systems running, but i dont think that this will be fixed any more either.

Try to repro on a clean system.
If you have something suspicious installed (especially, of your own brew) it could cause this.

I dont have any self made kernel code running on that system and mostly all drivers are either signed and WHQL passed and/or MS supplied.

Anyway i just was wondering why such an (accidental) attempt to terminate the SYSTEM process (wich is not allowed!) leaves the system in such a state. It should fully deny that kind of request instead leaving something half and fail on some point and leave the core system destabilized.

K.

SE_DEBUG_PRIVILEGE gives you full access to processes and threads even running under another account. System process threads are normally not designed to deal with termination request (which is likely to affect the threads waiting with UserMode, for example, worker threads for DelayedWorkqueue type).

On 15-Mar-2012 21:16, xxxxx@broadcom.com wrote:

SE_DEBUG_PRIVILEGE gives you full access to processes and threads even running under another account.

This is not affected / complicated by “integrity levels” in NT6?

–pa

>SE_DEBUG_PRIVILEGE gives you full access to processes and threads even

running under another account.

Yes, i am aware of this and the security mechanism that SE_DEBUG_NAME privilege grants. I use this by time to work on remote processes. What made me wonder is that this special SYSTEM(4) process on Windows Vista “seems” not to be properly protected against such level of access like a TerminateProcess/NtxZwTerminateProcess request wich for sure is completely sensless like terminatig things like one of the the client/server runtime subsystem processes or a process/thread/object etc. that is marked critical and can cause a BSOD on premature termination/close. Remembering the RtlSetProcessIsCritical() here,…

K.

The protection is that only admin-equivalent users are granted SeDebugPrivilege.

An admin-equivalent user has countless ways of wrecking the system if they want to.

  • S (Msft)

From: xxxxx@lists.osr.com [xxxxx@lists.osr.com] on behalf of xxxxx@arcor.de [xxxxx@arcor.de]
Sent: Thursday, March 15, 2012 12:59 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Unexpected RESOURCE_NOT_OWNED on hibernation request

SE_DEBUG_PRIVILEGE gives you full access to processes and threads even
running under another account.

Yes, i am aware of this and the security mechanism that SE_DEBUG_NAME privilege grants. I use this by time to work on remote processes. What made me wonder is that this special SYSTEM(4) process on Windows Vista “seems” not to be properly protected against such level of access like a TerminateProcess/NtxZwTerminateProcess request wich for sure is completely sensless like terminatig things like one of the the client/server runtime subsystem processes or a process/thread/object etc. that is marked critical and can cause a BSOD on premature termination/close. Remembering the RtlSetProcessIsCritical() here,…

K.


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

On 3/15/2012 8:00 PM, xxxxx@arcor.de wrote:

Anyway i just was wondering why such an (accidental) attempt to
terminate the SYSTEM process (wich is not allowed!) leaves the system
in such a state. It should fully deny that kind of request instead
leaving something half and fail on some point and leave the core
system destabilized. K.

Yes. Sounds like a bug in an extremely rare and unsupported corner case,
under very specific circumstances.

“When I kick my PC, sometimes it reboots.”

>Yes. Sounds like a bug in an extremely rare and

unsupported corner case, under very specific
circumstances.

Yes, i encountered this unexpectedly and under very special situations due to a simple typo and fortunately my helper application has a transactional verbose log and also uses systems logs to write down any action before it commits any changes. Typo means here something special: The helper application is expecting an array of strings delimited by a space character to probe and signal given process names to carry out actions. If they do not respond to the signals or return “busy” state in a given timespan, they will be terminated. In this case i was passing a few test-hardcoded names to my application and one changed from “DBCKV12System” to “DBCKV12 System” and then the application tried first to signal the SYSTEM process and then to terminate it with TerminateProcess. After that the helper app should hibernate the system with SetSuspendState(). Luckily the logfile facillity is recording ech step made, including system events and actions carried out by the app and i was able to backtrack that isse. Thats the background.

I tested this on a Windows7 64 bit and the result is still not as expected but at least the system is not going BSOD and “recovering” somehow. On Windows7 the system appears to hibernate, but after fading the screen to black and a little disk spinning it comes right back with a logon screen expecting me to Ctrl-Alt-Del for a logon as it were booted for the first time. Logfiles from the system do not show any error or warning stating any unexpected exception or situation. Sure, this is nothing common and very rare and unusual, but this seems to be a real bug on Vista, Seven and possibly on Eight (not tested yet) too.

K.

What happens is that some of the SYSTEM process threads will be knocked out, but whether the OS will BSOD depends on what state the threads were, what drivers owned them, etc.

>What happens is that some of the SYSTEM process

threads will be knocked out, but whether the OS will
BSOD depends on what state the threads were, what
drivers owned them, etc.

So this means that some (kernel mode) thread tries to access a already freed and/or possibly to another thread reassigned object/handle and the windows kernel intercepts here due to much worse corruption that could be done by accessing foreign or invalid objects/handles,…the fundamental intention behind the systems STOP for sure.

The BSOD only appears on Vista when the session is about to be send to disk, so interessting would be here to know what threads and objects exactly had been taken out. Working with the system is perfectly fine, even loading drivers, unloading them, attaching devices, removing them, etc,…

TerminateProcess is a very large and dangerous tool to invoke. It can
lead to all kinds of failures. Yes, you can open a jar of pickles with a
sledgehammer, but you will eat fewer glass slivers if you use a bottle
opener.

Just because an API exists, you should not assume that using it represents
Best Practice, or even Acceptable Practice. Or, in this case, you need to
recognize that TerminateProcess generally represents Bad Practice.

When you are using Task Manager to kill a process, CAREFULLY READ the
“fine print” in the MessageBox, which warns that terminating a process can
cause data loss, system instability, hair loss, ED, etc. There’s a reason
for that warning.

I tell my students that TerminateProcess sticks a charge of C4 in the
Process Control Block, and blows it to smithereens. Bits go flying every
which way. And the collateral damage can be substantial. I then
illustrate this with several examples.

When used inexpertly, it is a lot less like C4 (which is stable until
triggered) and a lot more like nitroglycerin (likely to do serious damage
spontaneously).
joe

>Yes. Sounds like a bug in an extremely rare and
>unsupported corner case, under very specific
>circumstances.

Yes, i encountered this unexpectedly and under very special situations due
to a simple typo and fortunately my helper application has a transactional
verbose log and also uses systems logs to write down any action before it
commits any changes. Typo means here something special: The helper
application is expecting an array of strings delimited by a space
character to probe and signal given process names to carry out actions. If
they do not respond to the signals or return “busy” state in a given
timespan, they will be terminated. In this case i was passing a few
test-hardcoded names to my application and one changed from
“DBCKV12System” to “DBCKV12 System” and then the application tried first
to signal the SYSTEM process and then to terminate it with
TerminateProcess. After that the helper app should hibernate the system
with SetSuspendState(). Luckily the logfile facillity is recording ech
step made, including system events and actions carried out by the app and
i was able to backtrack that isse. Thats the background.

I tested this on a Windows7 64 bit and the result is still not as expected
but at least the system is not going BSOD and “recovering” somehow. On
Windows7 the system appears to hibernate, but after fading the screen to
black and a little disk spinning it comes right back with a logon screen
expecting me to Ctrl-Alt-Del for a logon as it were booted for the first
time. Logfiles from the system do not show any error or warning stating
any unexpected exception or situation. Sure, this is nothing common and
very rare and unusual, but this seems to be a real bug on Vista, Seven and
possibly on Eight (not tested yet) too.

K.


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Hi Joseph,

i always carefully read all your post of high quality and i am aware of the fact that “forcing” a process to terminate by using TerminateProcess and others from that family can lead to lots of unexpected situations, stuff like leaking memory, leaking handles, hanging threads, stuff waiting forever, locking and even in worst case leading to a complete system fail ranging from blue screens to allover freezez even making the cpu reset! Terminate Process is like TerminateThread, which is pure evil and can lead to holes into your system by leacking memory, zombie handles, invalid data structures left in kernel space, etc, pp. Fortunately the windows team made a lot of good improvements on the never systems to make things more stable and less leaky, especially on TerminateThread looking at Vista and upper that frees the stack in kernel. I do have a mechanism i am using on the application. Let me quote myself from one of my posts before “The helper application … to probe and signal given process names to carry out actions. If they do not respond to the signals or return “busy” state in a given timespan, they will be terminated.”. I know this is not good, but sometimes the only way to get rid of something that stuck and simply wont go down. Its bad, but there is a reason, why these APIs exist. One should even use levels of termination requests to a e.g. windowed application before using a crowbar to break the process. Levels can be WM_CLOSE; WM_QUIT, WM_* and if nothing works, then i guess an TerminateProcess attempt is ok. There are many many other ways to make a process go down (debug process->close debug object, write garbage to the process memory regions, closing all its handles, modifying the threads contexts), but all has its price as you said. So what should one do if there is a unexpected situation where no implementation esits to handle it. Sometimes there is no other way,…

K.

So, I think we agree that “you get what you get” in this situation.

However… rare, corner-case, unsupported, admin priv’ed or otherwise… it STILL reads like a code bug to me: Releasing a lock I don’t hold down an error path. If it was MY code, I’d appreciate somebody filing the bug report on it.

Now, it may NOT be a bug… and even if it is it’s certainly not a high priority bug. But if I was the dev owner, I’d STILL like to hear about it. Even if I just get to close it “As Designed”.

Peter
OSR

>terminating a process can cause data loss, system instability, hair loss, ED

Darn it, why has nobody warned me before? Looks like my wild days of Task Managing need to end. Luckily, there’s a pill for that.

When a process terminates, all its handles are released, all the memory it
is using is released. It is a common misconception (I hear it all the
time from my students) that killing a process can “leak memory”, although
I know of no mechanism that would make this possible. Nor am I aware of
any way killing a process can “leak handles”. This was true since Windows
NT 3.1.

Key here is that you have to try all the accepted “good” ways of asking a
process to close; if they all fail, then you can consider
TerminateProcess.

There is at least one and perhaps two articles in MSDN on proper process
shutdown protocols. You need to try those first. Note that you should
never SendMessage to another process (use SenMessageTimeout or
PostMessage) and putting a WM_QUIT message in the queue is about on par
with TerminateProcess: it is deadly dangerous to the data the process is
managing.

Note that anything that causes a long delay in message processing (such as
DNS queries, complex database queries, blocks on potentially unbounded I/O
devices, etc.) should never be in the main message loop, but should be
handled by secondary threads. Otherwise, you either get “false positives”
to your timeout intervals, or the main GUI thread really is wedged, and
the correct solution is to fix that program.

A couple of your ideas are isomorphic to TerminateProcess, but far less
elegant. For example, writing garbage to memory. This is about as
sensible as using a military-surplus flame thrower to remove wallpaper.
It gets the job done, but has other serious consequences. (You can also
use napalm on your dog to remove the stench of skunk. When you’re done,
your dog no longer smells of skunk).

In a case where you are creating an app that monitors another app you
created, you should provide for a Registered Window Message requsting
emergency shutdown, which is semi-isomorphic to WM_CLOSE, except it knows
there is no user to respond to confirmation messages so takes the
minimally-destructive path (such as saving important state for later
recovery). I’ve built these systems, and it takes a lot of care. But my
co-programs can be restarted at any time and will recover state from each
other so one can contnue to function without the other. In debug mode,
there is no auto-restart of the co-program because I might be recompiling
it, but in release mode each can restart the other.

And there is not a TerminateProcess in sight. They require the user take
explicit action in the Task Manager, and we do not guarantee data
integrity is maintained. Each co-program has menu items to shut down and
to restart its partner, but each will auto-detect te other has been shut
down, and will also detect when its co-program has come back up. One
executes as a “tray” app and uses balloon notifications to notify the
operator if there is a problem, or that a problem has been resolved.

If you take as a design goal “there shall be no TerminateProcess call”
then you can build coopertaing suites of programs that are robust and can
make guarantees about correctness. If you allow TerminateProcess, you
have to assue on every startup that all persistent data is subject to
arbitrary corruption. I’ve built these systems, too, and they are vastly
more difficult to build.
joe

Hi Joseph,

i always carefully read all your post of high quality and i am aware of
the fact that “forcing” a process to terminate by using TerminateProcess
and others from that family can lead to lots of unexpected situations,
stuff like leaking memory, leaking handles, hanging threads, stuff waiting
forever, locking and even in worst case leading to a complete system fail
ranging from blue screens to allover freezez even making the cpu reset!
Terminate Process is like TerminateThread, which is pure evil and can lead
to holes into your system by leacking memory, zombie handles, invalid data
structures left in kernel space, etc, pp. Fortunately the windows team
made a lot of good improvements on the never systems to make things more
stable and less leaky, especially on TerminateThread looking at Vista and
upper that frees the stack in kernel. I do have a mechanism i am using on
the application. Let me quote myself from one of my posts before “The
helper application … to probe and signal given process names to carry
out actions. If they do not respond to the signals or return “busy” state
in a given timespan, they will be terminated.”. I know this is not good,
but sometimes the only way to get rid of something that stuck and simply
wont go down. Its bad, but there is a reason, why these APIs exist. One
should even use levels of termination requests to a e.g. windowed
application before using a crowbar to break the process. Levels can be
WM_CLOSE; WM_QUIT, WM_* and if nothing works, then i guess an
TerminateProcess attempt is ok. There are many many other ways to make a
process go down (debug process->close debug object, write garbage to the
process memory regions, closing all its handles, modifying the threads
contexts), but all has its price as you said. So what should one do if
there is a unexpected situation where no implementation esits to handle
it. Sometimes there is no other way,…

K.


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Hi Jospeh,

Key here is that you have to try all the accepted “good”
ways of asking a process to close; if they all fail, then you
can consider TerminateProcess.

Never said something different,…

and putting a WM_QUIT message in the queue

Putting this on a foreign process is for sure deadly, but not if you are prepared to handle this in your own app!

Note that anything that causes a long delay in message processing

Thats why i am using threads, synchronisation and a signaling mechanism to make sure not to get false positives as less as possible.

Registered Window Message requsting emergency shutdown, which
is semi-isomorphic to WM_CLOSE, except it knows there is no user to
respond to confirmation

If your application has a own mechanism to handle this its fine, but the application must be still responding or there will be no arrival of anykind of message once its message pump, threads or polling queque for commands is frozen! Here the Application Recovery and Restart (ARR) comes quite handy or another watchdog who can be of help e.g. a service, but there you have the session barries, etc,…

suites of programs that are robust and can make guarantees about correctness.

There is simply no :frowning:

i think we can infinitely debate on this “best practises” and “design concepts” topic and still find no 100% failsafe solution. Its impossible to handle all kind of failures in a application, because there are things that can be out of the scope of your app, e.g. the runtime, external situations, operating system failure, powerloss and and and. IMHO there is NO 100% safe way to make a application work 100% safe, even by using transational processes, there is always something that can fail. I am sure you know who Edward A. Murphy is :slight_smile:

I would like to ask something on DuplicateHandle and handle leaks: What if i do have a Process A and a Process B and Process A accesses a object handle in Process B with DuplicateHandle and DUPLICATE_SAME_ACCESS but Process A dies before Process B can close the handle to the Process A object handle. Am i right if i say that the kernel object is not freed unless the Process B releases the handle to object in dead Process A. Isnt that a “sort” of handle leak? As long as there is a reference count >0 the object and its memory is alive but the object does not belong to Process B. Or am i wrong and the handle to the object goes down with Process A and the duplicate handle on Process B is invalid?

On 18-Mar-2012 13:46, xxxxx@arcor.de wrote:

I would like to ask something on DuplicateHandle and handle leaks:
What if i do have a Process A and a Process B and Process A accesses a object handle in Process B
with DuplicateHandle and DUPLICATE_SAME_ACCESS but Process A dies before Process B
can close the handle to the Process A object handle.
Am i right if i say that the kernel object is not freed unless the Process B releases
the handle to object in dead Process A. Isnt that a “sort” of handle leak?
As long as there is a reference count>0 the object and its memory is alive but the
object does not belong to Process B. Or am i wrong and the handle to the object goes
down with Process A and the duplicate handle on Process B is invalid?

Process B gets its own handle that refers to the same kernel object.
So no, there is no handle leak. It is a form of “anonymous” sharing
when the name of the shared object is unknown to the recipient.
But otherwise it is a normal sharing.
When an object gets shared, it does not “belong” to any one of the parties.

– pa