Crash in PsTerminateSystemThread

Hi,

I’m stumped by an odd crash (see below) that occurs when
PsTerminateSystemThread is called from within a thread I created within my
filter driver. The thread terminates when a flag is set from within my
IRP_MJ_CLOSE processing. The thread creation code called during
IRP_MJ_CREATE looks like:

// :
// Init abort handling
pDevExt->bAbortThread = FALSE;
KeInitializeEvent(&pDevExt->eventAbort, NotificationEvent, FALSE);

// Startup the kernel thread
ntStatus = PsCreateSystemThread(&hThread, (ACCESS_MASK)0,
NULL, (HANDLE)0, NULL, JCSerialThreadMain, pDevExt);
if (!NT_SUCCESS(ntStatus))
goto ZB_OPEN_EXIT;

// Convert the Thread object handle into a pointer to the
// Thread object itself. Then close the handle.
ObReferenceObjectByHandle(hThread, THREAD_ALL_ACCESS, NULL,
KernelMode, &pDevExt->threadWorker, NULL);
ZwClose(hThread);
// :

The thread itself is structured like this:

VOID JCSerialThreadMain(PVOID Context)
{
// :
// :
while (TRUE) // Loop until thread suicide is requested
{
if (pDevExt->bAbortThread)
break; // Abort the thread
//:
//:
}
JCSerialDump(JCSTRACECALLS, (“JCSerial: JCSerialThreadMain - End\n”));
PsTerminateSystemThread(STATUS_SUCCESS);
}

Finally the IRP_MJ_CLOSE code looks like:

// Set the abort flag and trigger the sync event used
// by the worker thread.
pDevExt->bAbortThread = TRUE;
KeSetEvent(&pDevExt->eventAbort, IO_NO_INCREMENT, TRUE);

// Wait for the thread to die
KeWaitForSingleObject(pDevExt->threadWorker, Executive, KernelMode, FALSE,
NULL);

// The thread is dead
ObDereferenceObject(pDevExt->threadWorker); // Release hold on object

From looking around this technique is pretty standard stuff. The crash is
somewhat random in occurence. Has anyone seen anything like this? Can
someone shed some light what PsUnlockProcess does?

Dale


Access violation - code c0000005 (!!! second chance !!!)
nt!PsUnlockProcess+47:
806647d3 ff8180000000 inc dword ptr [ecx+0x80]
kd> !analyze -v
*******************************************************************************

*
*
* Bugcheck Analysis
*
*
*
*******************************************************************************

Unknown bugcheck code (0)
Unknown bugcheck description
Arguments:
Arg1: 00000000
Arg2: 00000000
Arg3: 00000000
Arg4: 00000000

Debugging Details:

DEFAULT_BUCKET_ID: DRIVER_FAULT
BUGCHECK_STR: 0x0
LAST_CONTROL_TRANSFER: from 806657fc to 806647d3

STACK_TEXT:
f3247cc0 806657fc 00000001 00000000 00000000 nt!PsUnlockProcess+0x47
f3247d74 80581359 00000000 f3247da8 eb333e58 nt!PspExitThread+0x4a2
f3247d80 eb333e58 00000000 20000000 20000000
nt!PsTerminateSystemThread+0x31
f3247da8 805811e6 81eda648 00000000 00000000
jcserial!JCSerialThreadMain+0x32a [d:\daleprojs\jcserial\jcserial.c @
1019]
f3247ddc 8059bd4a eb333b2e 81eda648 00000000
nt!PspSystemThreadStartup+0x54
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

FOLLOWUP_IP:
jcserial!JCSerialThreadMain+32a
eb333e58 8be5 mov esp,ebp

FOLLOWUP_NAME: MachineOwner
SYMBOL_NAME: jcserial!JCSerialThreadMain+32a
MODULE_NAME: jcserial
IMAGE_NAME: jcserial.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 3d778508
STACK_COMMAND: kb
BUCKET_ID: 0x0_jcserial!JCSerialThreadMain+32a

Hi,

I think you call ZwClose() without waiting for exit event from the thread
after PsCreateSystemThread() in the following code. so when you call
ZwClose(), the thread handle has been already invalid.
you need to KeWaitForSingleObject(), don’t you?

Thanks,
Futoshi

-----Original Message-----
From: Dale Larson [mailto:d-larson@wi.rr.com]
Sent: Friday, September 06, 2002 5:11 AM
To: NT Developers Interest List
Subject: [ntdev] Crash in PsTerminateSystemThread

Hi,

I’m stumped by an odd crash (see below) that occurs when
PsTerminateSystemThread is called from within a thread I created within my
filter driver. The thread terminates when a flag is set from within my
IRP_MJ_CLOSE processing. The thread creation code called during
IRP_MJ_CREATE looks like:

// :
// Init abort handling
pDevExt->bAbortThread = FALSE;
KeInitializeEvent(&pDevExt->eventAbort, NotificationEvent, FALSE);

// Startup the kernel thread
ntStatus = PsCreateSystemThread(&hThread, (ACCESS_MASK)0,
NULL, (HANDLE)0, NULL, JCSerialThreadMain, pDevExt);
if (!NT_SUCCESS(ntStatus))
goto ZB_OPEN_EXIT;

// Convert the Thread object handle into a pointer to the
// Thread object itself. Then close the handle.
ObReferenceObjectByHandle(hThread, THREAD_ALL_ACCESS, NULL,
KernelMode, &pDevExt->threadWorker, NULL);
ZwClose(hThread);
// :

The thread itself is structured like this:

VOID JCSerialThreadMain(PVOID Context)
{
// :
// :
while (TRUE) // Loop until thread suicide is requested
{
if (pDevExt->bAbortThread)
break; // Abort the thread
//:
//:
}
JCSerialDump(JCSTRACECALLS, (“JCSerial: JCSerialThreadMain - End\n”));
PsTerminateSystemThread(STATUS_SUCCESS);
}

Finally the IRP_MJ_CLOSE code looks like:

// Set the abort flag and trigger the sync event used
// by the worker thread.
pDevExt->bAbortThread = TRUE;
KeSetEvent(&pDevExt->eventAbort, IO_NO_INCREMENT, TRUE);

// Wait for the thread to die
KeWaitForSingleObject(pDevExt->threadWorker, Executive, KernelMode, FALSE,
NULL);

// The thread is dead
ObDereferenceObject(pDevExt->threadWorker); // Release hold on object

From looking around this technique is pretty standard stuff. The crash is
somewhat random in occurence. Has anyone seen anything like this? Can
someone shed some light what PsUnlockProcess does?

Dale


Access violation - code c0000005 (!!! second chance !!!)
nt!PsUnlockProcess+47:
806647d3 ff8180000000 inc dword ptr [ecx+0x80]
kd> !analyze -v
****************************************************************************
***

*
*
* Bugcheck Analysis
*
*
*
****************************************************************************
***

Unknown bugcheck code (0)
Unknown bugcheck description
Arguments:
Arg1: 00000000
Arg2: 00000000
Arg3: 00000000
Arg4: 00000000

Debugging Details:

DEFAULT_BUCKET_ID: DRIVER_FAULT
BUGCHECK_STR: 0x0
LAST_CONTROL_TRANSFER: from 806657fc to 806647d3

STACK_TEXT:
f3247cc0 806657fc 00000001 00000000 00000000 nt!PsUnlockProcess+0x47
f3247d74 80581359 00000000 f3247da8 eb333e58 nt!PspExitThread+0x4a2
f3247d80 eb333e58 00000000 20000000 20000000
nt!PsTerminateSystemThread+0x31
f3247da8 805811e6 81eda648 00000000 00000000
jcserial!JCSerialThreadMain+0x32a [d:\daleprojs\jcserial\jcserial.c @
1019]
f3247ddc 8059bd4a eb333b2e 81eda648 00000000
nt!PspSystemThreadStartup+0x54
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

FOLLOWUP_IP:
jcserial!JCSerialThreadMain+32a
eb333e58 8be5 mov esp,ebp

FOLLOWUP_NAME: MachineOwner
SYMBOL_NAME: jcserial!JCSerialThreadMain+32a
MODULE_NAME: jcserial
IMAGE_NAME: jcserial.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 3d778508
STACK_COMMAND: kb
BUCKET_ID: 0x0_jcserial!JCSerialThreadMain+32a


You are currently subscribed to ntdev as: xxxxx@citrix.co.jp
To unsubscribe send a blank email to %%email.unsub%%

> Hi,

I think you call ZwClose() without waiting for exit event from the thread
after PsCreateSystemThread() in the following code. so when you call
ZwClose(), the thread handle has been already invalid.
you need to KeWaitForSingleObject(), don’t you?

Are you saying that I shouldn’t call “ZwClose” after I call
“ObReferenceObjectByHandle”? My impression was that an increased reference
count was good enough. I do wait for the thread object to be signaled
during IRP_MJ_CLOSE processing before decreasing the thread object’s ref
count.

As a test I moved the ZwClose to after to the thread shutdown to see what
would happen and it crashed in the same manner. When I have my call
tracing code enabled I get the following:

JCSerial: Enter JCSerialDispatch
JCSerial: Dispatch IRP: 81eb7a68
JCSerial: IRP 81eb7a68 details: Maj = 0, Min = 0, Flag = 0, Comp = 0
JCSerial: Enter JCSerialFilterDispatch
// At this time I’ve sent the IRP_MJ_CREATE call down to the
// serial.sys driver so it can complete its processing before
// I create a thread to drive the port with a proprietory protocol.
// I intercept the I/O completion and, if there is no error, I continue
// on with the thread set up. The following power IRP is triggered
// by serial.sys processing the IRP_MJ_CREATE.
JCSerial: Enter JCSerialDispatch
JCSerial: Dispatch IRP: 81ee3008
JCSerial: IRP 81ee3008 details: Maj = 16, Min = 2, Flag = 0, Comp =
8057c42e
JCSerial: Enter JCSerialFilterDispatch
// IRP is passed down with call to PoCallDriver thusly:
// PoStartNextPowerIrp(pIrp);
// IoSkipCurrentIrpStackLocation(pIrp);
// JCSerialDump(JCSIRPPATH, (“JCSerial: Pass down power IRP: %x\n”,
pIrp));
// ntStatus = PoCallDriver(pDevExt->pSerialDevice, pIrp);
// JCSerialDump(JCSTRACECALLS, (“JCSerial: Leave JCSerial … (5)\n”));
// return ntStatus;
JCSerial: Pass down power IRP: 81ee3008
JCSerial: Leave JCSerialFilterDispatch (5)
JCSerial: Leave JCSerialDispatch
JCSerial: Enter JCSerialOpenCompletion
// Caught completion of IRP_MJ_CREATE and
// signaled event so my create priocess can proceed.
JCSerial: Claim ownership for IRP: 81eb7a68
JCSerial: Leave JCSerialOpenCompletion
JCSerial: Enter JCSerialZoneBusOpen
JCSerial: Leave JCSerialZoneBusOpen
// Thread is running right here…
JCSerial: JCSerialThreadMain - Start
JCSerial: Enter JCSerialInitZoneBusSerialPort
// Some IRP_MJ_DEVICE_CONTROL are sent here to
// configure the serial port
JCSerial: Leave JCSerialInitZoneBusSerialPort (7)
// thread is now waiting for a CTS or Abort event.
//
// The dispatch processing resumes and completes
// the IRP_MJ_CREATE command.
JCSerial: Complete IRP: 81eb7a68
JCSerial: Leave JCSerialFilterDispatch (1)
JCSerial: Leave JCSerialDispatch
JCSerial: Enter JCSerialDispatch
// Here is the IRP_MJ_CLEANUP command
JCSerial: Dispatch IRP: 81eb7a68
JCSerial: IRP 81eb7a68 details: Maj = 12, Min = 0, Flag = 0, Comp =
eb37a919
JCSerial: Enter JCSerialFilterDispatch
JCSerial: Pass down IRP: 81eb7a68
JCSerial: Leave JCSerialFilterDispatch (6)
JCSerial: Leave JCSerialDispatch
JCSerial: Enter JCSerialDispatch
// Here is the IRP_MJ_CLOSE command
JCSerial: Dispatch IRP: 81eb7a68
JCSerial: IRP 81eb7a68 details: Maj = 2, Min = 0, Flag = 0, Comp =
eb37a919
JCSerial: Enter JCSerialFilterDispatch
JCSerial: Enter JCSerialZoneBusClose
// Abort event is signalled and the thread exits
JCSerial: JCSerialThreadMain - End
// KA-BLAM!
Access violation - code c0000005 (!!! second chance !!!)
nt!PsUnlockProcess+47:
806647d3 ff8180000000 inc dword ptr [ecx+0x80]

Only four IRP’s are being processed:
IRP_MJ_CREATE, IRP_MJ_POWER, IRP_MJ_CLEANUP and IRP_CLOSE. I don’t process
the IRP_MJ_CLEANUP and only pass it down the serial.sys driver. It looks
like it completes since the IRP is being reused by the caller (PORTMON).

What’s really strange is I have a test program that opens the driver and
appears to do all the same things but it functions fine. The sequence of
IRPs are the same.

Dale

Thanks,
Futoshi

-----Original Message-----
pDevExt->bAbortThread = FALSE;
KeInitializeEvent(&pDevExt->eventAbort, NotificationEvent, FALSE);

// Startup the kernel thread
ntStatus = PsCreateSystemThread(&hThread, (ACCESS_MASK)0,
NULL, (HANDLE)0, NULL, JCSerialThreadMain, pDevExt);
if (!NT_SUCCESS(ntStatus))
goto ZB_OPEN_EXIT;

// Convert the Thread object handle into a pointer to the
// Thread object itself. Then close the handle.
ObReferenceObjectByHandle(hThread, THREAD_ALL_ACCESS, NULL,
KernelMode, &pDevExt->threadWorker, NULL);
ZwClose(hThread);

Finally the IRP_MJ_CLOSE code looks like:

// Set the abort flag and trigger the sync event used
// by the worker thread.
pDevExt->bAbortThread = TRUE;
KeSetEvent(&pDevExt->eventAbort, IO_NO_INCREMENT, TRUE);

// Wait for the thread to die
KeWaitForSingleObject(pDevExt->threadWorker, Executive, KernelMode, FALSE,
NULL);

// The thread is dead
ObDereferenceObject(pDevExt->threadWorker); // Release hold on object

From looking around this technique is pretty standard stuff. The crash is
somewhat random in occurence. Has anyone seen anything like this? Can
someone shed some light what PsUnlockProcess does?

Dale


Access violation - code c0000005 (!!! second chance !!!)
nt!PsUnlockProcess+47:
806647d3 ff8180000000 inc dword ptr [ecx+0x80]
kd> !analyze -v
****************************************************************************
***

*
*
* Bugcheck Analysis
*
*
*
****************************************************************************
***

Unknown bugcheck code (0)
Unknown bugcheck description
Arguments:
Arg1: 00000000
Arg2: 00000000
Arg3: 00000000
Arg4: 00000000

Debugging Details:

DEFAULT_BUCKET_ID: DRIVER_FAULT
BUGCHECK_STR: 0x0
LAST_CONTROL_TRANSFER: from 806657fc to 806647d3

STACK_TEXT:
f3247cc0 806657fc 00000001 00000000 00000000 nt!PsUnlockProcess+0x47
f3247d74 80581359 00000000 f3247da8 eb333e58 nt!PspExitThread+0x4a2
f3247d80 eb333e58 00000000 20000000 20000000
nt!PsTerminateSystemThread+0x31
f3247da8 805811e6 81eda648 00000000 00000000
jcserial!JCSerialThreadMain+0x32a [d:\daleprojs\jcserial\jcserial.c @
1019]
f3247ddc 8059bd4a eb333b2e 81eda648 00000000
nt!PspSystemThreadStartup+0x54
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

FOLLOWUP_IP:
jcserial!JCSerialThreadMain+32a
eb333e58 8be5 mov esp,ebp

FOLLOWUP_NAME: MachineOwner
SYMBOL_NAME: jcserial!JCSerialThreadMain+32a
MODULE_NAME: jcserial
IMAGE_NAME: jcserial.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 3d778508
STACK_COMMAND: kb
BUCKET_ID: 0x0_jcserial!JCSerialThreadMain+32a


You are currently subscribed to ntdev as: xxxxx@citrix.co.jp
To unsubscribe send a blank email to %%email.unsub%%



This had the feel of me doing something dumb and the cause did not
disappoint! The problem was I used a pointer to my own device in some
internal calls for that created IRPs I would send down to the serial
driver. :frowning:

Thanks to anyone who took the time read and/or respond to my original
post.

Dale

> > Hi,
> >
> > I think you call ZwClose() without waiting for exit event from the thread
> > after PsCreateSystemThread() in the following code. so when you call
> > ZwClose(), the thread handle has been already invalid.
> > you need to KeWaitForSingleObject(), don’t you?
>
> Are you saying that I shouldn’t call “ZwClose” after I call
> “ObReferenceObjectByHandle”? My impression was that an increased reference
> count was good enough. I do wait for the thread object to be signaled
> during IRP_MJ_CLOSE processing before decreasing the thread object’s ref
> count.
>
> As a test I moved the ZwClose to after to the thread shutdown to see what
> would happen and it crashed in the same manner. When I have my call
> tracing code enabled I get the following:

Dang I spoke too soon!

The fault is back and I’m nearly certain I’m setting my IRPs up properly.
Does
anyone have any idea what PsUnlockProcess does? The call to
PsTerminateSystemThread doesn’t have any parameters which point to data
so I don’t see how it gets a bad memory address. !irpfind doesn’t find
any IRPs associated with the thread.


*** This is the crashed worker thread
Access violation - code c0000005 (!!! second chance !!!)
nt!PsUnlockProcess+47:
806647d3 ff8180000000 inc dword ptr [ecx+0x80]

kd> !thread
THREAD 81f1e020 Cid 8.130 Teb: 00000000 Win32Thread: 00000000 RUNNING
Not impersonating
Owning Process 827676e0
WaitTime (seconds) 8366
Context Switch Count 4
UserTime 0:00:00.0000
KernelTime 0:00:00.0000
Start Address jcserial!JCSerialThreadMain (0xeb33bb3e)
Stack Init f347a000 Current f3479c50 Base f347a000 Limit f3477000 Call 0
Priority 16 BasePriority 8 PriorityDecrement 0 DecrementCount 0

ChildEBP RetAddr Args to Child
f3479cc0 806657fc 00000001 00000000 00000000 nt!PsUnlockProcess+0x47
f3479d74 80581359 00000000 f3479da8 eb33bebc nt!PspExitThread+0x4a2
f3479d80 eb33bebc 00000000 20000000 20000000
nt!PsTerminateSystemThread+0x31
f3479da8 805811e6 81eb1b88 00000000 00000000
jcserial!JCSerialThreadMain+0x37e
f3479ddc 8059bd4a eb33bb3e 81eb1b88 00000000
nt!PspSystemThreadStartup+0x54
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

**** Disassembly of PsUnlockProcess:
nt!PsUnlockProcess:
8066478c 55 push ebp
8066478d 8bec mov ebp,esp
8066478f 56 push esi
80664790 8b350c864d80 mov esi,[nt!_imp__KeGetCurrentIrql
(804d860c)]
80664796 ffd6 call esi
80664798 3c01 cmp al,0x1
8066479a 7628 jbe nt!PsUnlockProcess+0x38 (806647c4)
8066479c ffd6 call esi
8066479e 0fb6c0 movzx eax,al
806647a1 50 push eax
806647a2 683e476680 push 0x8066473e
806647a7 e824d4f1ff call nt!DbgPrint (80581bd0)
806647ac 59 pop ecx
806647ad 59 pop ecx
806647ae 6a00 push 0x0
806647b0 68e2080000 push 0x8e2
806647b5 6866476680 push 0x80664766
806647ba 6886476680 push 0x80664786
806647bf e8872af2ff call nt!RtlAssert (8058724b)
806647c4 be40695c80 mov esi,0x805c6940
806647c9 8bce mov ecx,esi
806647cb e87005f3ff call nt!ExAcquireFastMutexUnsafe (80594d40)
806647d0 8b4d08 mov ecx,[ebp+0x8]
*** The fault is on the following line
806647d3 ff8180000000 inc dword ptr [ecx+0x80]
ds:0023:00000081=???
806647d9 83a19800000000 and dword ptr [ecx+0x98],0x0
806647e0 8d8180000000 lea eax,[ecx+0x80]
806647e6 8b8180000000 mov eax,[ecx+0x80]
806647ec 83f801 cmp eax,0x1
806647ef 740d jz nt!PsUnlockProcess+0x72 (806647fe)


**** This is the thread waiting for my worker thread to
**** terminate
kd> !thread 81f0cda0
THREAD 81f0cda0 Cid 400.1ac Teb: 7ffdb000 Win32Thread: e1ccb788 WAIT:
(Executive) KernelMode Non-Alertable
81f1e020 Thread
IRP List:
81ec8a68: (0006,0190) Flags: 00000404 Mdl: 00000000
Not impersonating
Owning Process 81eac020
WaitTime (seconds) 8359
Context Switch Count 111 LargeStack
UserTime 0:00:00.0015
KernelTime 0:00:00.0031

Start Address kernel32!BaseThreadStartThunk (0x77e87532)
Win32 Start Address ntvdm (0x0f0056de)
Stack Init f2f2a000 Current f2f2997c Base f2f2a000 Limit f2f25000 Call 0
Priority 9 BasePriority 8 PriorityDecrement 0 DecrementCount 0

ChildEBP RetAddr Args to Child
f2f29994 8051312e 81eb1ad0 81ec8a68 00000000 nt!KiSwapThread+0x1b1
f2f299bc eb33b3c4 81f1e020 00000000 00000000
nt!KeWaitForSingleObject+0x1dd
f2f299e0 eb33a941 81eb1b88 81eb1ad0 81ec8a68
jcserial!JCSerialZoneBusClose+0x8e
f2f29a4c eb339e92 81eb1ad0 81ec8a68 eb37be40
jcserial!JCSerialFilterDispatch+0x29f
f2f29a78 804fc968 81eb1ad0 81ec8a68 81e9a000
jcserial!JCSerialDispatch+0x15c
f2f29a90 eb37b0e6 81e48020 81ec8a68 00000000 nt!IopfCallDriver+0x4f
f2f29bf8 eb37b1be 00f29ac4 81ec8a68 804fc968 PORTMSYS+0x30e6
f2f29c50 8065280e 81f30d00 8277f8c8 81f30db0 PORTMSYS+0x31be
f2f29c74 805772e8 81f30dc8 80785e2c 81f30db0
nt!ObpRemoveObjectRoutine+0x1ce
f2f29ca0 80577787 f2f29d58 0252fa04 805774ea nt!ObfDereferenceObject+0x26a
f2f29d4c 8059660a 00000114 e1cbb1f8 00000000 nt!NtClose+0x29d
f2f29d4c 77f82811 00000114 e1cbb1f8 00000000 nt!KiSystemService+0x10a
0252fa04 00000000 00000000 00000000 00000000 ntdll!NtClose+0xb