Unique process id

>> Really not. How can I know whether that thread is in a lengthy OS call

> (that
> may take several minutes) or holding a lock ? I cannot.

Then wait for this. The file calls can be interrupted by closing a file
handle, this cancels the IRPs.

If you have something that is taking several minutes in the kernel holding
a lock (short of an actual LockFile API) then there is something much
deeper wrong.
joe


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

>

“Maxim S. Shatskih” wrote in message
> news:xxxxx@ntdev…
>>> Therefore in my multithreaded software I see no other solution than to
>>> kill
>>> those threads
>
>>Too dirty. It is really, really possible to do the graceful shutdown of
>> all
>>threads.
>
>
> Really not. How can I know whether that thread is in a lengthy OS call
> (that
> may take several minutes) or holding a lock ? I cannot. Because even if I
> would be able to save state in a lock free manner to make it visible to
> the
> thread which gracefully requests the termination, that information can be
> invalidated at any time (race).

Somehow you have become very confused about how to write multithreaded
code. IT IS YOUR RESPONSIBILITY TO CLEANLY TERMINATE ALL THREADS IN YOUR
APPLICATION. Period. Not negotiable. TerminateThread is not the correct
solution, and in fact indicates erroneous programming. If you write a
thread you don’t know how to terminate, your code is wrong. If you have
to wait for the kernel to return from a blocked thread, you wait for the
thread to terminate. That’s it. I usually keep a thread reference count,
and each thread sends a message to my main GUI thread when it terminates
which tells me to decrement the reference count, and I don’t exit the GUI
thread until the active-secondary-thread-count value goes to zero. I used
to teach courses in this.
>
> //Daniel
>
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

>>> Note that Windows has no concept of a “main thread”; in general, the

>> main thread can exit any time,
>> and the process is live as long as there is any potentially runnable
>> thread in the process.

???

The thread executing main() or WinMain() calls ExitProcess after return.

Which will not kill threads that are blocked in the kernel. Or at least,
it used to be that way.
joe


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

> ExitProcess terminates the other threads internally.

In a Win32 program, returning from main eventually leads you to
ExitProcess.

Also, coding an ExitProcess call in an application is an indicator of
erroneous code. The only ExitProcess call that exists should be the
implicit one you just described.

joe

  • S (Msft)

From: xxxxx@resplendence.commailto:xxxxx
> Sent: ý11/ý21/ý2012 23:21
> To: Windows System Software Devs Interest Listmailto:xxxxx
> Subject: Re:[ntdev] Unique process id
>
>>> Note that Windows has no concept of a “main thread”; in general, the
>>> main
>>> thread can exit any time,
>
> Windows does have a concept of primary thread for a process.
>
>
>>> and the process is live as long as there is any potentially runnable
>>> thread in the process.
>
> That is true. If the primary thread terminates, it doesn’t mean the
> process
> or outstanding threads die. The process continues to live until all
> outstanding threads have terminated.
>
> Very often when closing an application it simply takes WAY too much time
> to
> wait for all outstanding threads (which may be involved in a lengthy
> function call) to finish gracefully and terminating the outstanding
> threads
> may cause deadlock problems because they may be holding critical sections.
>
> Therefore in my multithreaded software I see no other solution than to
> kill
> those threads and the process with TerminateProcess (not ExitProcess)
> after
> I did the necessary cleanup. AFAIK that’s the only way to do it.
>
> //Daniel
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx></mailto:xxxxx>

>

> Windows does have a concept of primary thread for a process.

Well, I don’t know if “main thread” is Win32 concept, but it certainly
does exist in Win32 userland…

As I said, this concept is one which we, as programmers, use to designate
threads. The kernel neither knows nor cares about our vision.

> That is true. If the primary thread terminates, it doesn’t mean the
> process or outstanding threads die.
> The process continues to live until all outstanding threads have
> terminated.

IIRC, Win32 calls ExitProcess() after the main thread’s main() function
(no pun intended) returns control…
I guess the only situation where Joe’s statement applies is the one when
someone is perverse enough
to call ExitThread() in app’s main thread…

ExitThread is another of those calls that if you have written it, you have
almost certainly created an erroneous piece of code.

> Therefore in my multithreaded software I see no other solution than to
> kill those threads and the process with >TerminateProcess (not
> ExitProcess) after I did the necessary cleanup. AFAIK that’s the only
> way to do it.

Well, according to MSDN, ExitProcess() is sufficient …

It does not really terminate the threads cleanly; a multithreaded app that
cannot tell its threads to cleanly terminate represents erroneous code.
joe

Check
http://msdn.microsoft.com/en-us/library/windows/desktop/ms682658(v=vs.85).aspx

Anton Bassov


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

>>> Note that Windows has no concept of a “main thread”; in general, the

>> main
>> thread can exit any time,

Windows does have a concept of primary thread for a process.

I have never seen this. I have seen that when you call CreateProcess, you
get back a handle and ID of the process, and a handle and ID of the thread
that was created, but this thread is free to terminate at any time, and as
long as there is at least one potentially runnable thread, the process is
considered “live”. As far as the scheduler is concerned, there’s a bunch
of threads in the process. Which one is “main” is not a concept it
possesses.

>> and the process is live as long as there is any potentially runnable
>> thread in the process.

That is true. If the primary thread terminates, it doesn’t mean the
process
or outstanding threads die. The process continues to live until all
outstanding threads have terminated.

Very often when closing an application it simply takes WAY too much time
to
wait for all outstanding threads (which may be involved in a lengthy
function call) to finish gracefully and terminating the outstanding
threads
may cause deadlock problems because they may be holding critical sections.

Therefore in my multithreaded software I see no other solution than to
kill
those threads and the process with TerminateProcess (not ExitProcess)
after
I did the necessary cleanup. AFAIK that’s the only way to do it.

TerminateProcess and TerminateThread should never be coded in an
application. They represent an erroneous app. If you think you need
TerminateThread or TerminateProcess, you have a bad design, and need to
fix it.
joe

//Daniel


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

>> Note that Windows has no concept of a “main thread”; in general, the

> main thread can exit any time,
> and the process is live as long as there is any potentially runnable
> thread in the process.

???

If you think about it a little bit you will realize that exiting a
process would be next to impossible if
everything worked the way you describe. In order to be able to exit, it
would first have to terminate all worker threads that it has created,which
means it would have to track their state. However, in practice, IIRC, all
it has to do is to exit its main loop…

That is because the infrastructure contains an ExitProcess. Until that is
executed, a thread is just a thread. Any thread could call ExitProcess
and have the same effect. Any call of ExitProcess other than the implicit
one represents a coding error.

Note that *Windows* does not have the concept of a “main thread”; this is
an illusion imposed on it by the infrastructure that handles the call and
return to main() or WinMain().
joe

Anton Bassov


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

A thread can be requested to terminate at any time, without regard to whether it is currently executing in kernel mode or user mode.

Threads are terminated only when they are ready to return to user mode from kernel mode, were they in kernel mode.

A well-behaved kernel mode component that does inline work on behalf of a user thread will configure their long-running waits such that they can receive notification that a thread termination has been requested, and subsequently break out of their processing in a timely fashion.

  • S (Msft)

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@flounder.com
Sent: Thursday, November 22, 2012 12:10 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Unique process id

>> Note that Windows has no concept of a “main thread”; in general, the
>> main thread can exit any time, and the process is live as long as
>> there is any potentially runnable thread in the process.

???

The thread executing main() or WinMain() calls ExitProcess after return.

Which will not kill threads that are blocked in the kernel. Or at least, it used to be that way.
joe


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

wrote in message news:xxxxx@ntdev…
>

Dude, you gotta learn to read for a change before recklessly dumping your
opinions. Communications with you is almost entirely one way traffic.

//Daniel

Here is the picture of what happens during ExitProcess:

Any thread can call ExitProcess. When it happens, all other threads in the process are hit with a kernel mode equivalent of TerminateThread (ZwTerminateThread?). All IO issued from them will be canceled. Any locks held in user mode are abandoned. Note that TerminateThread is non-blocking. If I/O cancellation takes long time, it doesn’t wait for that.

The remaining thread will call DllMain of all DLLs with DLL_PROCESS_DETACH with lpReserved non-NULL. THE ONLY GOOD ACTION POSSIBLE IS TO RETURN without doing anything. It’s impossible for a DLL to clean up anything in this case, because the process heap can be in inconsistent state.

Starting from Vista, if DllMain of an remaining exiting thread is trying to acquire any CRITICAL_SECTION, the thread will be terminated implicitly. This prevents a deadlock if any critical section is left hanging.

> Dude, you gotta learn to read for a change before recklessly dumping your opinions.

Communications with you is almost entirely one way traffic.

Oh, come on…

Let’s face it - Joe’s statements (BTW, unlike both yours and mine on this thread) are precisely correct and strictly on-topic, so that referring to them as to “recklessly dumping your opinions” seems to be, at least in this context, rude and uncalled for…

Anton Bassov

wrote in message news:xxxxx@ntdev…
> Oh, come on…
>
> Let’s face it - Joe’s statements (BTW, unlike both yours and mine on this
> thread) are precisely correct and strictly on-topic, so that referring to
> them as to “recklessly dumping your opinions” seems to be, at least in
> this context, rude and uncalled for…
>

Personally, I think it’s rude to flood the board with 7 consecutive messages
from the same author with nothing but commonplaces and generalizations while
ignoring one’s arguments against them. That makes me lose my patience
because it’s about impossible to defend against.

You see, there are OS calls that do things on a network that can take
minutes. And I know of one particular one that can potentially take several
hours to return. If we need to wait for them gracefully, that leaves the end
user no other option than to kill off the hanging process with the task
manager.

If I need to choose I prefer to be graceful to the common sense of the end
users rather than to somebody’s limited awareness about software design.
That doesn’t mean I have become very confused or got erroneous code.

//Daniel

xxxxx@hotmail.com wrote:

Let’s face it - Joe’s statements (BTW, unlike both yours and mine on this thread) are precisely correct and strictly on-topic, so that referring to them as to “recklessly dumping your opinions” seems to be, at least in this context, rude and uncalled for…

Daniel has a point, although I’m not sure I would have said it out
loud. Joe has a tendency to be extremely literal (to the point of
annoyance), and spews very technical, black-and-white, good-vs-bad,
judgmental answers to questions that demand a slightly fuzzy interpretation.

Regardless of what the original poster typed, we all pretty much know
what question he was TRYING to ask. There are two approaches to
questions like that. One is to answer the question literally as asked,
knowing that such an answer does not help the poster, demanding that he
ask the question again using the proper vocabulary. The other is to
interpret the intent, restate the question that was really meant, and
answer that. Joe *always* chooses the former. I’m not sure that’s the
most helpful approach.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

>…spews very technical, black-and-white, good-vs-bad, judgmental answers to questions

that demand a slightly fuzzy interpretation.

Well, seems to be just a classical description of a typical poster to NTDEV, don’t you think. IIRC, situation in other Windows-related NGs is even more predictable and boring…

Anton Bassov

> A thread can be requested to terminate at any time, without regard to

whether it is currently executing in kernel mode or user mode.

Some years ago, some folks at ISI had a lot of problems because of this.
ASKING a thread to terminate is not the same as HAVING the thread
terminate, as noted below.

Threads are terminated only when they are ready to return to user mode
from kernel mode, were they in kernel mode.

A well-behaved kernel mode component that does inline work on behalf of a
user thread will configure their long-running waits such that they can
receive notification that a thread termination has been requested, and
subsequently break out of their processing in a timely fashion.

Their experience was that, if we accept the premise that “well-behaved”
components will do this, then many parts of the kernel were not
well-behaved. I no longer recall all the details, but they could get
indefinitely-long hangs. I used to be able to get it just by blocking on
a semaphore; nothing short of TerminateThread could blast it loose. I long
ago recognized that I had to multiwait on the synchronization primitive
AND a “shutdown event”. Timed waits were far too messy because of the
necessary cleanup involved.

Cleanly terminating threads was something I taught in my classes on
Advanced Windows System Programming. It is /always/ possible to tell a
thread to cleanly terminate, unless it is stuck on sone blocking call that
ecfectively overrides the request. Key here is to avoid blocking calls.
Network APIs are the worst offenders, but even file system calls can be
hung up on a non-responding server, and device calls to custom devices,
likewise.

Some years ago, we had to do DNS name resolution on program startup, so it
could re-establish communication. There were no asynchronous DNS calls
then, so I did it all in a backround thread, with an unobtrusive progress
bar displayed on the status bar. An attempt to shut the program down
required putting up a dialog that said “waiting for network response” and
I later added a progress bar that counted down 70 seconds (and recorded
the time right before the call so it was mostly accurate). That thread
could not be broken out of the gethostbyname (if I’ve remembered the call
correctly) function even by a call to ExitThread.

When I talk about “erroneous programs”, I am basing this on many decades
of programming experience (and almost 20 years in Win32) of the kind of
code that gives the illusion of working without actually being correct.
Tiny changes in the assumptions means the program stops working (sometimes
in subtle ways) and the programmer often ignores certain issues I know to
look for, because that part of the program is “known to work”. Tossing in
gratuitous “exithread” and “exitprocess” calls are among the most common
failure modes. I’ve even had a programmer tell me “I was told that
ExitProcess was bad because the OnExit calls aren’t made, so now I call
exit(n)” Of course, when I go looking for the error, the first place I
look is to make sure that all cleanup paths are properly executed. They
usually aren’t. Consider the model

UINT threadfunc(LPVOID p)
{
allocate thread state structure and initialize it
whatever->runthread(threadstate);
deallocate thread state structure
notify soneone the thread has cleanly terminated
return 0;
}

Note that if someone writes an ExitThread call (or one of its many
synonyms) then the lines following the runthread() call are never
executed. But they’re not inportant anyway (and if you believe this, my
uncle was in the Nigerian army and I need your help in getting his assets
out of the country). Since many of these calls are in (untested) error
handling paths, and they leak small amounts of memory, the field report
says “once every 10-14 days we have to restart the service because its
memory footprint has become ridiculously large”. You can’t get this
failure in-house. It is worse if the whatever->runthread() call is being
used to “enter C++ space”. When the ExitThread call happens, all existing
objects in the heap, that would have been cleaned up by their end-of-scope
destructors, are left behind. My “opinions” are backed by years of
experience in finding and fixing these kinds of errors.

I once had a client insist that C++ collections were unreliable because
they leaked memory (I was being asked to recode my solution because I had
used std::map). I said this was simply not true, and he countered by
presenting me with “proof”, a program that leaked massive amounts of heap
space when it had used C++ collections. In less than five minutes I found
an exit() call buried in an error path, and eventually found a couple
dozen. They used aalloc compulsively, “because using malloc() gave us the
same problems” (why the failures were attributed solely to C++ in the
presence of this contradictory information escapes me).

So I have taken to referring to these time bombs as “erroneous code”
because, in fact, many of them /are/ erroneous, and those that are not
will become so under maintenance.

It’s easy to write code that gives the illusion of working. It’s a lot
harder to write code that is actually correct, and will remain so for
years. Key is to not commit certain errors, because if you do, you will
eventually commit them in a context where certain necessary preconditions
are violated by their usage. I call such practices “erroneous”.
joe

  • S (Msft)

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@flounder.com
Sent: Thursday, November 22, 2012 12:10 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Unique process id

>>> Note that Windows has no concept of a “main thread”; in general, the
>>> main thread can exit any time, and the process is live as long as
>>> there is any potentially runnable thread in the process.
>
> ???
>
> The thread executing main() or WinMain() calls ExitProcess after return.

Which will not kill threads that are blocked in the kernel. Or at least,
it used to be that way.
joe

>
> –
> Maxim S. Shatskih
> Windows DDK MVP
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

> That thread could not be broken out of the gethostbyname (if I’ve remembered the call correctly)

function even by a call to ExitThread.

I just wonder how a thread may be blocking and calling ExitThread() at the same time…

OK, I can attribute it to a typo, but look below:

If you make an exit() call the calling process terminates, and, at this point, freeing memory becomes just unnecessary, right. In any case, you will never reach the point of memory leak becoming evident, for understandable reasons…

All your statements seem to be related strictly to ExitThread() - indeed, you have a good chance to leave the address space in inconsistent state unless you know what you are doing. However, ExitProces() just terminates a caller, and, at this point, all cleanup tasks become already meaningless. Therefore, unless you happen to use IPC synch constructs across the process boundary or do something that may affect the behavior of other processes in the system , what all this has to do with ExitProcess() is just above me…

Anton Bassov

wrote in message news:xxxxx@ntdev…
>
> Some years ago, we had to do DNS name resolution on program startup, so it
> could re-establish communication. There were no asynchronous DNS calls
> then, so I did it all in a backround thread, with an unobtrusive progress
> bar displayed on the status bar. An attempt to shut the program down
> required putting up a dialog that said “waiting for network response” and
> I later added a progress bar that counted down 70 seconds (and recorded
> the time right before the call so it was mostly accurate). That thread
> could not be broken out of the gethostbyname (if I’ve remembered the call
> correctly) function even by a call to ExitThread.
>

In our example we are dealing with a problem. A problem that is caused by a
synchronous API that is blocking for way too long. I consider it just an
easy way out to put it in the face of the end user by means of a waiting
dialog so that now it becomes HIS problem instead. Like Windows Live Mail,
it sucks. Like fighting over sand, it doesn’t mean you should, just because
you can. And if the end user is savvy enough he will almost certainly kill
off the hanging app with the task manager. That is worse than terminating
the process in a controlled manner.

In the above example, when it’s time to say goodbye for a process I don’t
put up a dialog box telling the user he should quietly wait. What I do is,
set a flag in my program so that
> notify soneone the thread has cleanly terminated
is no longer required and will be ignored if it happens thereafer
regardless. So terminating the entire process at this point after the
necessary cleanup has been done, is the only right thing to do in my book.
The kernel will clean up all outstanding resources that I do not care about.
Call that erroneous and if you still do consider that there are synchronous
API calls that can block for much longer than 70 seconds.

If you want to carve a rule in stone that says “TerminateThread is evil and
should never be used” you can count me in. TerminateThread can leave your
program in an undefined state and will leak. It’s not even allowed to use as
a last recourse when the program exists because it can potentially deadlock.

If you want to carve a rule in stone that says “TerminateProcess is evil and
should never be used” then I say hold on a second.

My opinion is that kernel programming should be treated as an exact science
in which we have to be pragmatic, correct, unequivocal and 100% loyal to our
principles. Because if we don’t everybody knows the consequences.

Programming in ring 3 in my book can be considered a craftsmanship at least
in part because it offers us the luxury to weigh in various factors other
than sterile academic correctness such as reasonability, user experience and
the human factor.

//Daniel

“Now some of us build and some of us teach,
some of us build and some of us teach.
And some of us kill what some of us eat.
That is a fact of life.”

Wait, wait, wait, wait,wait…

You’re not saying that there is one correct answer to a given problem, right? There is *never* one correct answer to an engineering problem. There are merely choices among correct answers which optimize different factors to obtain the desired results. So, it’s not “an exact science” in MY book.

Peter
OSR

Well, before jumping into a real discussion ( to beat our own brain(s)
), what is the definition of exact science ? Exact does not
necessarily mean unique, in my book though.

For example, pure match is the most exact science in my book, and
there are many places, where problems have a set of solutions. Physics
and its applications ( basically signals and systems ) do follow the
same pattern.

I think the predictable would be the better word, don’t know, but I
think I understand what he meant …

-pro

On Sun, Nov 25, 2012 at 8:58 AM, wrote:
>


>
> Wait, wait, wait, wait,wait…
>
> You’re not saying that there is one correct answer to a given problem, right? There is never one correct answer to an engineering problem. There are merely choices among correct answers which optimize different factors to obtain the desired results. So, it’s not “an exact science” in MY book.
>
> Peter
> OSR
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

wrote in message news:xxxxx@ntdev…
>


>
> Wait, wait, wait, wait,wait…
>
> You’re not saying that there is one correct answer to a given problem,
> right? There is never one correct answer to an engineering problem.
> There are merely choices among correct answers which optimize different
> factors to obtain the desired results. So, it’s not “an exact science” in
> MY book.
>

No that is not what I meant. Perhaps I should have said “engineering
discipline” or at least a process that can be formalized. In any case that
developing software for humans is a different practice with different rules
from developing software for devices.

//Daniel