Ensuring all threads are terminated before unloading.

Chris_Schade · August 26, 2009, 7:41am

Hello all,

My Driver Unload routine is creating threads( PsCreateSystemThread ) and after creating the thread it waits for an event to be fired by the newly created thread before it proceeds with unloading. So in Pseudo code:

VOID myUnload( )
{
PsCreateSystemThread();
KeWaitForSingleObject( );
}

There is more to it than just but that’s the general idea. The thread routine that I ‘execute’ with the created system thread will fire the event that we wait for and terminate its own thread with PsTerminateSystemThread.

VOID myThreadFunc( )
{
// Do something
KeSetEvent ( );
PsTerminateSystemThread ( 0 );
}

Now this normally runs fine, but every now and then I get a bugcheck telling me that the driver didn’t terminate all of it’s threads etc.

I assume that this happens because KeSetEvent() is called before PsTerminateSystemThread, and as soon as this is called the ‘main’ thread continues unloading, which may be faster than the PsTerminateSystemThread.
Is that idea about right? If so, should I include some sort of sleep after my KeWaitForSingleObject? Or are more correct solutions to ensure that all threads are terminated before unloading.

Thank you for your help.

James_Harper · August 26, 2009, 7:54am

>

Hello all,

My Driver Unload routine is creating threads( PsCreateSystemThread )
and after
creating the thread it waits for an event to be fired by the newly
created
thread before it proceeds with unloading. So in Pseudo code:

VOID myUnload( )
{
PsCreateSystemThread();
KeWaitForSingleObject( );
}

There is more to it than just but that’s the general idea. The thread
routine
that I ‘execute’ with the created system thread will fire the event
that we
wait for and terminate its own thread with PsTerminateSystemThread.

VOID myThreadFunc( )
{
// Do something
KeSetEvent ( );
PsTerminateSystemThread ( 0 );
}

Now this normally runs fine, but every now and then I get a bugcheck
telling
me that the driver didn’t terminate all of it’s threads etc.

I assume that this happens because KeSetEvent() is called before
PsTerminateSystemThread, and as soon as this is called the ‘main’
thread
continues unloading, which may be faster than the
PsTerminateSystemThread.
Is that idea about right? If so, should I include some sort of sleep
after my
KeWaitForSingleObject? Or are more correct solutions to ensure that
all
threads are terminated before unloading.

I puzzled over this a while back. You actually need to get the thread
object when you create the thread, and then wait on that. The thread
object becomes ‘set’ when the thread terminates.

The docs on “Device-Dedicated Threads” say:

" Because thread objects are themselves a type of dispatcher object, a
thread can wait for another thread to complete. To obtain the thread
object pointer associated with a thread, a driver can call
ObReferenceObjectByHandle, passing in the thread handle received from
PsCreateSystemThread."

James

OSR_Community_User · August 26, 2009, 8:05am

Whenever I use worker threads, I always use 2 events. The first event
is the kill event, which my driver sets when it wants to kill a thread
or threads, and the second event is the dead event that the thread sets
just before it exits. I prefer not to call PsTerminateSystemThread…

–Mark Cariddi
OSR, Open Systems Resources, Inc.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@gmail.com
Sent: Wednesday, August 26, 2009 7:43 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Ensuring all threads are terminated before unloading.

Hello all,

My Driver Unload routine is creating threads( PsCreateSystemThread ) and
after creating the thread it waits for an event to be fired by the newly
created thread before it proceeds with unloading. So in Pseudo code:

VOID myUnload( )
{
PsCreateSystemThread();
KeWaitForSingleObject( );
}

There is more to it than just but that’s the general idea. The thread
routine that I ‘execute’ with the created system thread will fire the
event that we wait for and terminate its own thread with
PsTerminateSystemThread.

VOID myThreadFunc( )
{
// Do something
KeSetEvent ( );
PsTerminateSystemThread ( 0 );
}

Now this normally runs fine, but every now and then I get a bugcheck
telling me that the driver didn’t terminate all of it’s threads etc.

I assume that this happens because KeSetEvent() is called before
PsTerminateSystemThread, and as soon as this is called the ‘main’ thread
continues unloading, which may be faster than the
PsTerminateSystemThread.
Is that idea about right? If so, should I include some sort of sleep
after my KeWaitForSingleObject? Or are more correct solutions to ensure
that all threads are terminated before unloading.

Thank you for your help.

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Paul_Sanders-1 · August 26, 2009, 8:08am

Mark,

I’m curious. Why not call PsTerminateSystemThread?

Thanks,
Paul
----- Original Message -----
From: “Mark Cariddi”

I prefer not to call PsTerminateSystemThread…

–Mark Cariddi
OSR, Open Systems Resources, Inc.

James_Harper · August 26, 2009, 8:19am

>

Whenever I use worker threads, I always use 2 events. The first event
is the kill event, which my driver sets when it wants to kill a thread
or threads, and the second event is the dead event that the thread
sets
just before it exits. I prefer not to call
PsTerminateSystemThread…

That’s just asking for a crash. Maybe not on your system but on a super
fast system of tomorrow it could happen that in the tiny amount of time
between the ‘dead event’ and the thread exiting (either because it fell
of the end of the function or the call to PsTerminate…) the device
could finish up and leave the thread still running.

Much better to wait on the thread object itself.

James

Gabriel_Bercea · August 26, 2009, 8:42am

The documentation specifies that a thread created with PsCreateSystemThread
should be terminated with PsTerminateSystemThread, but isn’t returning from
the thread equivalent to nightly terminating it ?

With respect,
Gabriel Bercea

GaMiTech Software Development
Mobile contact: ?(+40)0740049634
eMail: xxxxx@gmail.com
Blog: http://gamitech.blogspot.com/
Linkedin: http://www.linkedin.com/in/gamitech
Twitter: http://www.twitter.com/gamitech

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of James Harper
Sent: Wednesday, August 26, 2009 3:19 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ensuring all threads are terminated before unloading.

Whenever I use worker threads, I always use 2 events. The first event
is the kill event, which my driver sets when it wants to kill a thread
or threads, and the second event is the dead event that the thread
sets
just before it exits. I prefer not to call
PsTerminateSystemThread…

That’s just asking for a crash. Maybe not on your system but on a super
fast system of tomorrow it could happen that in the tiny amount of time
between the ‘dead event’ and the thread exiting (either because it fell
of the end of the function or the call to PsTerminate…) the device
could finish up and leave the thread still running.

Much better to wait on the thread object itself.

James

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Chris_Schade · August 26, 2009, 8:59am

I suspect that that is exactly what happened in my case where I set the event just before PsTerminate… and the driver would unload before the thread termination finished.

I have followed James Harper’s advice and so far it hasn’t crashed on me yet when unloading.

I will keep you updated of course in case it decides to crash in a later stage, but so far this approach seems more logic than firing an event just before PsTerminate…

Doron_Holan · August 26, 2009, 10:18am

Can i take a step back here for a second? Why are you creating a thread in driver unload? It is not as if you are making it go any faster, you are waiting for the thread which will take the same amount of time as if you did the work inline. You are also in the same context for (system) for either unload or your thread so that cannot be the reason.

Also, what do you do if the thread creation fails?

d

Sent from my phone with no t9, all spilling mistakes are not intentional.

-----Original Message-----
From: xxxxx@gmail.com
Sent: Wednesday, August 26, 2009 5:59 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Ensuring all threads are terminated before unloading.

I suspect that that is exactly what happened in my case where I set the event just before PsTerminate… and the driver would unload before the thread termination finished.

I have followed James Harper’s advice and so far it hasn’t crashed on me yet when unloading.

I will keep you updated of course in case it decides to crash in a later stage, but so far this approach seems more logic than firing an event just before PsTerminate…

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Ken_Johnson · August 26, 2009, 1:00pm

Unless this is documented to be the case, you shouldn’t rely on it.

S

-----Original Message-----
From: Bercea Gabriel
Sent: Wednesday, August 26, 2009 05:42
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ensuring all threads are terminated before unloading.

The documentation specifies that a thread created with PsCreateSystemThread
should be terminated with PsTerminateSystemThread, but isn’t returning from
the thread equivalent to nightly terminating it ?

With respect,
Gabriel Bercea

GaMiTech Software Development
Mobile contact: (+40)0740049634
eMail: xxxxx@gmail.com
Blog: http://gamitech.blogspot.com/
Linkedin: http://www.linkedin.com/in/gamitech
Twitter: http://www.twitter.com/gamitech

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of James Harper
Sent: Wednesday, August 26, 2009 3:19 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Ensuring all threads are terminated before unloading.

>
> Whenever I use worker threads, I always use 2 events. The first event
> is the kill event, which my driver sets when it wants to kill a thread
> or threads, and the second event is the dead event that the thread
sets
> just before it exits. I prefer not to call
PsTerminateSystemThread…
>

That’s just asking for a crash. Maybe not on your system but on a super
fast system of tomorrow it could happen that in the tiny amount of time
between the ‘dead event’ and the thread exiting (either because it fell
of the end of the function or the call to PsTerminate…) the device
could finish up and leave the thread still running.

Much better to wait on the thread object itself.

James

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Chris_Schade · August 26, 2009, 3:59pm

I was expecting that question already, but I have reasons to create threads in the unload routine that arn’t really relevant to the discussion.

I am currently not coding anything that will result in production code, I am merely trying things out just to get a better understanding of it all - call it fooling around with the purpose of education ;).

For the same reason I am doomed if the thread creation should fail, but as it is only educational code I hope not to encounter this.

Thank you all for your replies, I’ve learned now that the best approach is to wait for the thread object if we want to be sure that the thread has fully terminated.

anton_bassov · August 26, 2009, 4:31pm

> I have reasons to create threads in the unload routine that arn’t really relevant to the discussion.

WEll, although you believe that you have some reasons for it the system thinks otherwise…

In my experience, doing cleanup in a worker that gets started by DrvUnload() normally results in bugcheck with the error msg saying that driver had unloaded without having canceled pending operations. Furthermore, in my experience, waiting on event in DrUnload() is not going to help you in this situation…

The workaround that I used on 32-bit OS (where I could use inline assembly) was referencing DRIVER_OBJECT in DrvUnload before creating a thread, modifying the thread’s stack in such way that execution gets forwarded to ObDerefenceObject() and then to PsTerminateSystemThread() without ever getting back to the driver, and making a return from the thread routine with RET instruction right from inline assembly block

The trick is based upon the assumption that as long as DRIVER_OBJECT 's refcount is non-zero the target module does not get ummapped/removed from RAM even after DrvUnload() returns control.
However, you have to ensure that execution does not return back to your driver after you have decremented refcount, and this is why I had to resort to stack manipulations. I know it is ugly, but it worked…

Anton Bassov

Gabriel_Bercea · August 26, 2009, 4:58pm

Hmmm this is really interesting.
Do you have a sample of that ?

With respect,
Gabriel Bercea

GaMiTech Software Development
Mobile contact: ?(+40)0740049634
eMail: xxxxx@gmail.com
Blog: http://gamitech.blogspot.com/
Linkedin: http://www.linkedin.com/in/gamitech
Twitter: http://www.twitter.com/gamitech

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@hotmail.com
Sent: Wednesday, August 26, 2009 11:32 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Ensuring all threads are terminated before unloading.

I have reasons to create threads in the unload routine that arn’t really
relevant to the discussion.

WEll, although you believe that you have some reasons for it the system
thinks otherwise…

In my experience, doing cleanup in a worker that gets started by DrvUnload()
normally results in bugcheck with the error msg saying that driver had
unloaded without having canceled pending operations. Furthermore, in my
experience, waiting on event in DrUnload() is not going to help you in this
situation…

The workaround that I used on 32-bit OS (where I could use inline assembly)
was referencing DRIVER_OBJECT in DrvUnload before creating a thread,
modifying the thread’s stack in such way that execution gets forwarded to
ObDerefenceObject() and then to PsTerminateSystemThread() without ever
getting back to the driver, and making a return from the thread routine with
RET instruction right from inline assembly block

The trick is based upon the assumption that as long as DRIVER_OBJECT 's
refcount is non-zero the target module does not get ummapped/removed from
RAM even after DrvUnload() returns control.
However, you have to ensure that execution does not return back to your
driver after you have decremented refcount, and this is why I had to resort
to stack manipulations. I know it is ugly, but it worked…

Anton Bassov

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Lyndon_J_Clarke-2 · August 26, 2009, 6:46pm

Don’t wait for the super fast system of next year … I’ve seen this on the
super slow systems of last year … and yeah, wait for the thread object

“James Harper” wrote in message
news:xxxxx@ntdev…
>
> Whenever I use worker threads, I always use 2 events. The first event
> is the kill event, which my driver sets when it wants to kill a thread
> or threads, and the second event is the dead event that the thread
sets
> just before it exits. I prefer not to call
PsTerminateSystemThread…
>

That’s just asking for a crash. Maybe not on your system but on a super
fast system of tomorrow it could happen that in the tiny amount of time
between the ‘dead event’ and the thread exiting (either because it fell
of the end of the function or the call to PsTerminate…) the device
could finish up and leave the thread still running.

Much better to wait on the thread object itself.

James

James_Harper · August 26, 2009, 7:12pm

>

Don’t wait for the super fast system of next year … I’ve seen this
on the
super slow systems of last year … and yeah, wait for the thread
object

You can produce some really bizarre behaviour on a heavily loaded
virtual server. Your VM may have 4 virtual cpu’s assigned to it, but it
could happen that a physical cpu doesn’t get scheduled ‘into’ the
virtual cpu in a timely manner so you can see situations we are talking
about here fairly easily.

James