How to catch ExitProcess?

NTDEV
1

Apologies for a user-mode question; I seem to be spending far too much of my
time debugging UM apps these days.

I’ve got a very large program on my hands that has been (and still is)
developed by many people over the years, some of whom are dead and many
retired. So I can’t “ask the guy that wrote it” – he is dead, retired, or
both. The program uses lots of MS process and threading interfaces, C++
libraries and stuff, and raw calls to C library functions. And a bunch of
DLLs, some of which are 3rd party.

The program runs along just fine until memory gets tight. Then (with the
debugger attached) instead of having 50+ threads doing different things, I
suddenly have one completely arbitrary thread hanging in
WaitForMultipleObjects or the like. Obviously someone called ExitProcess or
some similar function on some thread (not this one). Somewhere. There is no
debug output that could be helpful.

I’ve done the usual text searches for exit-anything and put breakpoints or
debug stuff or disabled them, and I’m still not catching who is exiting. I
suspect someone in a DLL or maybe even in ntdll, but I can’t prove that,
yet.

Anyone know how I can catch who the bad guy is that is blowing out the
program? Unfortunately I CANNOT use kernel debugging on this test system, so
it will have to be some UM trick to catch it.

Thanks, and apologies for talking about UM stuff. You guys know more about
UM than most of the UM experts floating around.

Loren

2

IDA is your friend here - find out the address of ZwTerminateProcess() in ntdll.dll (I really hope I don’t have to explain to you how to do it), and set a breakpoint. Once IDA allows you to change execution context on a fly, you will be able to make the code return straight away without actually making a call, and trace the stack until
you find a culprit. Simple, ugh…

Anton Bassov

3

When a process calls ExitProcess from some thread, all other threads are forced to terminate instantly. Before they are killed completely, all I/O issued from them is canceled and is allowed to complete. Any locks are abandoned (in Vista+). In XP, any held locks will hang. When ExitProcess calls all DllMain, they MUST NOT do anything significant - free memory, wait for threads completion, etc. The only reasonable action for DllMain in DLL_PROCESS_DETACH on process exit is to return.

You can catch ExitProcess by setting a breakpoint in ntdll!ExitProcess. Also set a breakpoint in TerminateProcess.

It could also be a last thread issuing ExitThread.

It’s possible that an unhandled exception happens which invokes a system handler which invokes DrWatson or whatever it’s called these days. In this case, the process is still alive while the dump is taken. If you run it under debugger it should not be invoked, though.

You can force a full crashdump, and then study the dump in kernel debugger. You’ll see which callstack invoked ExitProcess.

4

I’ve found a couple useful things for this kind of user mode debugging. One is to run the app under application verifier. The problem is if it’s never been run under app verifier before it’s likely going to find stuff. Another thing I have found helpful is running an app under the FULL checked OS, which includes integrity checking of the user mode code too. Same warning, the checked OS often finds things wrong with what seem like functioning apps.

Like others have said, setting a break point at ExitProcess may be useful, and see what the call stack looks like.

I know many years ago, like maybe VS6 days, there was a bug in the C run-time that just exited the process if it could not allocate a new heap slab. The behavior was a sudden exit, with no message of any kind, under low memory conditions. MSFT perhaps fixed the DLL version of the run-time, but if they were statically linked into a dll or exe, that old code may still be there.

Jan

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Loren Wilton
Sent: Monday, April 29, 2013 7:22 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] How to catch ExitProcess?

Apologies for a user-mode question; I seem to be spending far too much of my time debugging UM apps these days.

I’ve got a very large program on my hands that has been (and still is) developed by many people over the years, some of whom are dead and many retired. So I can’t “ask the guy that wrote it” – he is dead, retired, or both. The program uses lots of MS process and threading interfaces, C++ libraries and stuff, and raw calls to C library functions. And a bunch of DLLs, some of which are 3rd party.

The program runs along just fine until memory gets tight. Then (with the debugger attached) instead of having 50+ threads doing different things, I suddenly have one completely arbitrary thread hanging in WaitForMultipleObjects or the like. Obviously someone called ExitProcess or some similar function on some thread (not this one). Somewhere. There is no debug output that could be helpful.

I’ve done the usual text searches for exit-anything and put breakpoints or debug stuff or disabled them, and I’m still not catching who is exiting. I suspect someone in a DLL or maybe even in ntdll, but I can’t prove that, yet.

Anyone know how I can catch who the bad guy is that is blowing out the program? Unfortunately I CANNOT use kernel debugging on this test system, so it will have to be some UM trick to catch it.

Thanks, and apologies for talking about UM stuff. You guys know more about UM than most of the UM experts floating around.

Loren

NTDEV is sponsored by OSR

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

5

There is a C library onexit() handler, and you could add a call there.

Is this app MFC or raw Win32? If MFC, there is a virtual method of the
CWinApp-derived class where you could set a breakpoint,
CYourApp::ExitInstance, and if it doesn’t exist, set a breakpoint at
CWinApp::ExitInstance.

Look also for calls to PostQuitMessage. This is often used to shut down
secondary threads that have a message pump, and if accidentally executed
in the main thread, will cause the app to shut down.

If non-MFC, find the message pump, and set a breakpoint after it; this
will tell you if you have exited the main thread. Look for where that
WFMO call is being done, and see what it is waiting on; this can tell you
a lot.

The problem with projects like this, that “just growed”, is that someone
decides to cause a thread exit when function X is called, thinking it is
never called from the main thread. Perhaps at the time that decision was
made, it was true. Then a later programmer spots this function, says
“Hey, this does everything I need!” and calls it from the main thread, not
noticing it also shuts down its calling thread. Oopsie. I’ve seen one
case where a WFMO was waiting on the calling thread’s handle, so it
couldn’t proceed to exit until it already exited.

A breakpoint on the C library onexit() call will reveal function addresses
that are to be called when the app exits. Putting breakpoints on one of
these may allow you to trace back to who might be the guilty party.

If it is MFC, then the secondary threads will also have ExitInstance
methods. Otherwise, you need to find the secondary thread message pump(s)
and breakpoint those after the GetMessage loop.

How are the threads being created? If the CreateThread API is used
dihrectly, all bets are off; the world is screwed up from the start. If
it is pure Win32, it should be calling the C library _beginthreadex call,
otherwise te C library is not properly initialized for the thread. Also,
if CreateThread is called directly it is possible to link without the C
multithreaded library, and you are definitely hosed. If it is MFC, you
/must/ call AfxBeginThread to create threads; otherwise, the per-thread
state for te MFC library is hosed. _beginthread can be dangerous because
it automatically closes the thread handle on thread termination, which
makes WFMO on thread handles impossible.

Which reminds me: that hung call on WFMO–where was it called in the
source? Knowing the path that gets you to the caller’s routine can tell
you a lot.

I spent most of my time in user space, and for 15 years was a C++/MFC MVP,
so I actually know many of these tricks for debugging. My apologies to
the rest of the readers for this lengthy OT answer. We can take this
offline here; email me at xxxxx@flounder.com.
joe

Apologies for a user-mode question; I seem to be spending far too much of
my
time debugging UM apps these days.

I’ve got a very large program on my hands that has been (and still is)
developed by many people over the years, some of whom are dead and many
retired. So I can’t “ask the guy that wrote it” – he is dead, retired, or
both. The program uses lots of MS process and threading interfaces, C++
libraries and stuff, and raw calls to C library functions. And a bunch of
DLLs, some of which are 3rd party.

The program runs along just fine until memory gets tight. Then (with the
debugger attached) instead of having 50+ threads doing different things, I
suddenly have one completely arbitrary thread hanging in
WaitForMultipleObjects or the like. Obviously someone called ExitProcess
or
some similar function on some thread (not this one). Somewhere. There is
no
debug output that could be helpful.

I’ve done the usual text searches for exit-anything and put breakpoints or
debug stuff or disabled them, and I’m still not catching who is exiting. I
suspect someone in a DLL or maybe even in ntdll, but I can’t prove that,
yet.

Anyone know how I can catch who the bad guy is that is blowing out the
program? Unfortunately I CANNOT use kernel debugging on this test system,
so
it will have to be some UM trick to catch it.

Thanks, and apologies for talking about UM stuff. You guys know more about
UM than most of the UM experts floating around.

Loren

NTDEV is sponsored by OSR

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

6

How about register a notify routine with PsSetCreateThreadNotifyRoutine? when a target thread is detroyed, call RtlWalkFrameChain to back through the call stack and make a log.

7

ProcDump can also be configured to generate a user mode dump on process
exit. It’s not clear if this will catch things at the right time in this
case, but easy enough to try:

http://technet.microsoft.com/en-us/sysinternals/dd996900

-scott
OSR

wrote in message news:xxxxx@ntdev…

How about register a notify routine with PsSetCreateThreadNotifyRoutine?
when a target thread is detroyed, call RtlWalkFrameChain to back through the
call stack and make a log.

8

last but not least you can run adplus in crash mode waiting silently
for any unexpected process exits

adplus used to be vbs (visual basic script) when i used it
but i think it has been enhanced in the newer windbg packages and now
has a manager of sorts

it is described in windbg help doc how to use setup and run adplus

though kinda cryptic at first sight iirc it worked rather well in a
hung appp situation

On 4/30/13, Scott Noone wrote:
> ProcDump can also be configured to generate a user mode dump on process
> exit. It’s not clear if this will catch things at the right time in this
> case, but easy enough to try:
>
> http://technet.microsoft.com/en-us/sysinternals/dd996900
>
> -scott
> OSR
>
> wrote in message news:xxxxx@ntdev…
>
> How about register a notify routine with PsSetCreateThreadNotifyRoutine?
> when a target thread is detroyed, call RtlWalkFrameChain to back through the
>
> call stack and make a log.
>
>
>
>
> —
> NTDEV is sponsored by OSR
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

9

If you are running on Windows 7 or later there is GFlags support for this. See “Monitoring Silent Process Exit (Windows Debuggers)”
http://msdn.microsoft.com/en-us/library/windows/hardware/jj602791(v=vs.85).aspx

10

And one more after last… you can just use our old friend, procmon.exe
to log the process in question. It will log all thread and process exit
events, with stack trace.
– pa

On 30-Apr-2013 22:52, raj_r wrote:

last but not least you can run adplus in crash mode waiting silently
for any unexpected process exits

adplus used to be vbs (visual basic script) when i used it
but i think it has been enhanced in the newer windbg packages and now
has a manager of sorts

it is described in windbg help doc how to use setup and run adplus

though kinda cryptic at first sight iirc it worked rather well in a
hung appp situation

On 4/30/13, Scott Noone wrote:
>> ProcDump can also be configured to generate a user mode dump on process
>> exit. It’s not clear if this will catch things at the right time in this
>> case, but easy enough to try:
>>
>> http://technet.microsoft.com/en-us/sysinternals/dd996900
>>
>> -scott
>> OSR
>>

11

xxxxx@jadeworld.com wrote:

If you are running on Windows 7 or later there is GFlags support for this. See “Monitoring Silent Process Exit (Windows Debuggers)”
http://msdn.microsoft.com/en-us/library/windows/hardware/jj602791(v=vs.85).aspx

That is the direction I would head. Not only does it dump the exiting
process when a non-crash abnormal exit occurs, but if a second process
actually called TerminateProcess against your process, it will give
you a dump of /that/ process too.

Alan Adams

12

Which reminded me of somethiing. If the process terminates, it has no
running threads, but if another process did this, and failed to close the
process handle, some vestiges of the process will remain.

I am curious what defies a “non-crash abnormal exit”. Exit codes are
hardly ever used by GUI processes, and in any case, if te process
terminates for any of the reasons I suggested earlier, it will look
entirely like a “normal” exit, even within the process, which will see
what it thinks is a “clean” termination.

joe

xxxxx@jadeworld.com wrote:

> If you are running on Windows 7 or later there is GFlags support for
> this. See “Monitoring Silent Process Exit (Windows Debuggers)”
> http://msdn.microsoft.com/en-us/library/windows/hardware/jj602791(v=vs.85).aspx

That is the direction I would head. Not only does it dump the exiting
process when a non-crash abnormal exit occurs, but if a second process
actually called TerminateProcess against your process, it will give
you a dump of /that/ process too.

Alan Adams

NTDEV is sponsored by OSR

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

13

I actually had an interesting experience along those lies. I got a note
from a client saying “we have run this under app verifier, and it seems
you never tested it, because <> keeps failing”.

To which I responded

“Read the description of <>. It clearly states “If the
operation succeeds, it returns a handle to <>. If it fails, it
returns NULL.” That is the correct behavior. If you read the code, it
clearly says:
HANDLE h = <>(<>);
if(h== NULL)
{ // Set exhausted
…stuff to do at end of iteration
}
else
{ // process set member
…stuff to do on current handle
}

So I am using the API as it is documented, and the NULL is an expected and
valid return. If your checker tells you this is an error in my code, it
is wrong.”

So sometimes, when you see a “failure”, you may need somewhere between a
grain of salt to an 80lb bag of salt to process that information. That
said, it is stlll not a bad idea to run App Verifier, because the answer
may be found in a few minutes. But when you see anomalies, you really
need to read the code to see what was intended.

One person’s “error” may be another person’s “correct use of an API”.
joe

> I’ve found a couple useful things for this kind of user mode debugging.
> One is to run the app under application verifier. The problem is if it’s
> never been run under app verifier before it’s likely going to find stuff.
> Another thing I have found helpful is running an app under the FULL
> checked OS, which includes integrity checking of the user mode code too.
> Same warning, the checked OS often finds things wrong with what seem like
> functioning apps.
>
> Like others have said, setting a break point at ExitProcess may be useful,
> and see what the call stack looks like.
>
> I know many years ago, like maybe VS6 days, there was a bug in the C
> run-time that just exited the process if it could not allocate a new heap
> slab. The behavior was a sudden exit, with no message of any kind, under
> low memory conditions. MSFT perhaps fixed the DLL version of the run-time,
> but if they were statically linked into a dll or exe, that old code may
> still be there.
>
> Jan
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Loren Wilton
> Sent: Monday, April 29, 2013 7:22 PM
> To: Windows System Software Devs Interest List
> Subject: [ntdev] How to catch ExitProcess?
>
> Apologies for a user-mode question; I seem to be spending far too much of
> my time debugging UM apps these days.
>
> I’ve got a very large program on my hands that has been (and still is)
> developed by many people over the years, some of whom are dead and many
> retired. So I can’t “ask the guy that wrote it” – he is dead, retired, or
> both. The program uses lots of MS process and threading interfaces, C++
> libraries and stuff, and raw calls to C library functions. And a bunch of
> DLLs, some of which are 3rd party.
>
> The program runs along just fine until memory gets tight. Then (with the
> debugger attached) instead of having 50+ threads doing different things, I
> suddenly have one completely arbitrary thread hanging in
> WaitForMultipleObjects or the like. Obviously someone called ExitProcess
> or some similar function on some thread (not this one). Somewhere. There
> is no debug output that could be helpful.
>
> I’ve done the usual text searches for exit-anything and put breakpoints or
> debug stuff or disabled them, and I’m still not catching who is exiting. I
> suspect someone in a DLL or maybe even in ntdll, but I can’t prove that,
> yet.
>
> Anyone know how I can catch who the bad guy is that is blowing out the
> program? Unfortunately I CANNOT use kernel debugging on this test system,
> so it will have to be some UM trick to catch it.
>
> Thanks, and apologies for talking about UM stuff. You guys know more about
> UM than most of the UM experts floating around.
>
> Loren
>
>
> —
> NTDEV is sponsored by OSR
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> —
> NTDEV is sponsored by OSR
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

14



The logic behind the above “masterpiece” is just amazing. Basically,what you are saying is that if you, say, pass an invalid hardcoded argument to a function and it always returns, for the obvious reasons, a code it is supposed to return on error, it is not your fault just because this error code is expected - once you deal with it properly, everything is OK. The fact that a call consistently fails due to your error is, in your opinion, of zero significance. OK, you are not going to crash because you, indeed, ensure that there is no run-time error coming coming into a play later, but a program may simply fail to ever exercise its intended functionality because of your bug. In “Dr.Joe”'s opinion, it does not matter…

Anton Bassov

15

>The fact that a call consistently fails due to your error is, in your opinion, of zero significance.

Wow, not so fast, Einstein. What if the API in question is like FindNextFile (Dr. Joe says: “Set exausted”), and “fail” is just another expected result?

16

> Wow, not so fast, Einstein. What if the API in question is like FindNextFile

(Dr. Joe says: “Set exausted”), and “fail” is just another expected result?

Well, in such case you are very unlikely to get a complaint from your customer the way " Dr.Joe" did,
i.e.the one about a certain API call that kept on failing on regular basis, don’t you think…

Please read his post carefully. Basically, what he says is that, as long as you properly deal with the possible failures of your calls to API functions (he calls it “the correct use of API”), your code is OK. Such a pronouncement is a complete nonsense, for understandable reasons. Above mentioned checking of return value happens to be just one of many pre-requisites for writing bug-free code. The only type of error that you can avoid this way is a run-time one. However, besides that, there may be errors of some other type (for example, semantical ones) in your code, and the API call may fail solely due to your error.

Judging from “Dr.Joe” post, he does not seem to care about it, does he…

Anton Bassov

17

>>The fact that a call consistently fails due to your error is, in your

> opinion, of zero significance.

Wow, not so fast, Einstein. What if the API in question is like
FindNextFile (Dr. Joe says: “Set exausted”), and “fail” is just another
expected result?

And actually, it was not FindNextFile, but the details don’t matter. I
was using the API according to its spec, and their verifier (I think it
was from the SoftICE people) thought it was always an error if that call
returned NULL.
joe

NTDEV is sponsored by OSR

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

18

[quote]
So I am using the API as it is documented, and the NULL is an expected and
valid return. If your checker tells you this is an error in my code, it
is wrong."

[quote]

and Anton said…

Anton… in your eagerness to pounce on Dr. Newcomer, I’m sorry to say the only one who’s pronouncing complete nonsense here is you.

Dr. Newcomer’s post – and described use of the API – is infinitely reasonable. The customer was complaining that this API was being called and a dynamic analysis determined it was returning an error. Dr. Newcomer simply said he used this error as a flag (to indicate something like the end of an iteration). There’s nothing at all inherently wrong with that. As Mr. Grig said, this is no different to calling one of the allocation APIs with an impossibly small buffer, just so that it will return you an error and the buffer size required. This is a well established programming pattern.

Precisely. And the point of ANY automated tool, from lint to PreFast to SDV to AppVerifier, is to raise issues to the dev so that they can review them and be sure things are working as anticipated. I’m not saying people should be cavalier about dismissing diagnostic errors… but just like building at /W4, some things you do because you WANT to do them. In this way, dynamic behavior is not necessarily any different to syntax.

Peter
OSR

19

This has so many dependencies and variations that it will make your head
hurt. Much of this is discussed in gory detail by those of us who
practice or did practice software fault injection. For a relatively
simple example from the kernel:

for ( i = 0; ; i++ )
{
status = ZwQueryObject( h, i, … );
…
}

According to Anton this is obviously an error since the second argument
of ZwQueryObject can only be 0 or 2. I would contend it really depends
on what the rest of the code in the block does. For instance you could
have a counter to break after n occurances of STATUS_INVALID_INFO_CLASS
and a check so that if there are more than two successful calls you log
something so that you can change your code to handle this new case.

Microsoft has a number of cases where errors will not be returned if you
pass the correct parameters. One can argue that you should still test
the result, and for a lot of us who are paranoid we will test, but
Microsoft publishes examples where they don’t (and has the PreFast
annotations so they don’t require testing the status). This is as much
a philosophy question as the indentation of braces.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

“xxxxx@hotmail.com” wrote in message
news:xxxxx@ntdev:

> > Wow, not so fast, Einstein. What if the API in question is like FindNextFile
> > (Dr. Joe says: “Set exausted”), and “fail” is just another expected result?
>
>
> Well, in such case you are very unlikely to get a complaint from your customer the way " Dr.Joe" did,
> i.e.the one about a certain API call that kept on failing on regular basis, don’t you think…
>
>
>
> Please read his post carefully. Basically, what he says is that, as long as you properly deal with the possible failures of your calls to API functions (he calls it “the correct use of API”), your code is OK. Such a pronouncement is a complete nonsense, for understandable reasons. Above mentioned checking of return value happens to be just one of many pre-requisites for writing bug-free code. The only type of error that you can avoid this way is a run-time one. However, besides that, there may be errors of some other type (for example, semantical ones) in your code, and the API call may fail solely due to your error.
>
> Judging from “Dr.Joe” post, he does not seem to care about it, does he…
>
>
> Anton Bassov

20

>

[quote]

I got a note from a client saying “we have run this under app verifier,
and it seems you never tested it, because <> keeps
> failing”. To which I responded “Read the description of <>.
> It clearly states “If the operation succeeds, it returns a handle to
> <>. If it fails, it returns NULL.”
>
> …
>
>
> So I am using the API as it is documented, and the NULL is an expected
> and valid return.
> If your checker tells you this is an error in my code, it is wrong.”
>
>
[/quote]

>
>
> The logic behind the above “masterpiece” is just amazing. Basically,what
> you are saying is that if you, say, pass an invalid hardcoded argument to
> a function and it always returns, for the obvious reasons, a code it is
> supposed to return on error, it is not your fault just because this error
> code is expected - once you deal with it properly, everything is OK. The
> fact that a call consistently fails due to your error is, in your
> opinion, of zero significance. OK, you are not going to crash because you,
> indeed, ensure that there is no run-time error coming coming into a play
> later, but a program may simply fail to ever exercise its intended
> functionality because of your bug. In “Dr.Joe”'s opinion, it does not
> matter…
>
Huh? Where did I say I was passing an invalid hard-coded argument? On
the contrary, I was passing in a computed value, asking for the next item
in a set whose members had an enumersble algorithm by which they were
created. There was no way, as in FindNextFile, to get an error “no more
items left in the set”, so the only way to discover set exhaustion was to
get a NULL return when the call gave the computed “next item” value. What
you’re saying is that a read operation that fails on EOF is an example of
deliberately coding an error into the program. I was enunerating a set of
values whose “name” started at “000” and there could never be more than
about 64, but I added the extra “0” so I wouldn’t have to change anything
if they’d lied to me. How would YOU ask for “the next item” in a set of
values, where the only guarantee was the namespace comprised a dense
sequence of integers in the range 0…N where N is unknown because another
program created that set of values? I did it by iterating until I tried
to access the N+1th eelement, and when te call returned NULL I knew I had
reached the end. I do not consider this a “bug”, when it is using the
only known way to get at those values.
joe
>
> Anton Bassov
>
> —
> NTDEV is sponsored by OSR
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>