Fwd: RE: Kernel debug in Windbg using tcp - is it possible?

Michal_Vodicka-2 · January 18, 2007, 1:37pm

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Pavel Lebedinsky[SMTP:xxxxx@hotmail.com]
Reply To: Windows System Software Devs Interest List
Sent: Thursday, January 18, 2007 10:43 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Kernel debug in Windbg using tcp - is it possible?

> Personally, I have no problem with WinDbg command line.
> It should have more intelligent history and completion as
> SI or 4NT have but it is generally useful.

I’m not familiar with SI. What would you like to see improved
in windbg’s command completion?

Completion works some way, although is misses some commands (try .h – .hh isn’t between choices). It doesn’t seem to work with KD extension prefix (try !ndiskd. – nothing). It doesn’t complete available KD extensions. It doesn’t complete symbols – module! should offer all possible matches. So basic functionality is there but it is rather dumb.

With intelligent history I mean you can write a prefix and history displays only matching entries. So for “!a” “!analyze -v” is correct but “.process” isn’t.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

Michal_Vodicka-2 · January 18, 2007, 1:42pm

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Tim Roberts[SMTP:xxxxx@probo.com]
Reply To: Windows System Software Devs Interest List
Sent: Thursday, January 18, 2007 7:22 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Kernel debug in Windbg using tcp - is it possible?

SI had real-time command completion. As you typed each letter, it
showed you the list of possible commands that you might be typing. When
the command was unique, it showed you a one-line usage hint.

If nothing else, this was of great benefit for accidentally discovering
useful commands. I would sometimes just type “a”, then backspace and
“b”, then backspace and “c”, just to see the complete list of available
commands.

Now, that was possible in SI because they had a screen-oriented GUI.
WinDbg, being essentially a command-line app, doesn’t really have a way
to do that. But I, for one, would never have had a clue that the rather
un-mnemonic “sxeld” command existed had I not read it in a post earlier
today.

It’d be enough if it works as TAB completion. But TAB works only for commands starting with . or ! and doesn’t work for letters.

I also found SI real-time help at status line useful. It worked dynamically with commands available at given moment including loaded KD extensions.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

Michal_Vodicka-2 · January 18, 2007, 1:56pm

OK, take it as a result of my frustration from Vista USB stack It is better to write ridiculous posts than examine logs which show new and new bugs which I can’t solve :-/ I really wonder what USB team used to debug their drivers. If anything…

Don’t underestimate aesthetic aspect of engineering. Good work is usually also nice and not only in programming.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Arlie Davis[SMTP:xxxxx@microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Thursday, January 18, 2007 6:15 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Kernel debug in Windbg using tcp - is it possible?

I know engineers love to argue, but this is ridiculous. I can’t belive this thread has become a “debuggers vs. printfs” argument. Puh-leeze. Do whatever it takes to drive your code to quality. Hell, some of the best debugging I’ve done has been with a print-out and a red pen, a cup of coffee, and not a computer in sight.

I expect any competent engineer to have a strong working knowledge of the debugger(s) that apply to the environment they work in. And I expect them to build components with good tracing, internal asserts, etc. They are all different approaches to the same goal, quality.

If you like tracing, fine, go for it. But I have found interactive debuggers to be an incredibly powerful tool for investigating what was *not* anticipated at design-time, and therefore doesn’t have prints / asserts / whatever.

Do engineers in other professions waste as much breath on stuff like this? Do civil engineers argue about whether they should load test their bridges *OR* use finite element analysis? For some weird reason, programming often seems to be more aesthetics than engineering.

From: xxxxx@lists.osr.com [xxxxx@lists.osr.com] On Behalf Of Michal Vodicka [xxxxx@upek.com]
Sent: Wednesday, January 17, 2007 7:30 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Kernel debug in Windbg using tcp - is it possible?

> ----------
> From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Calvin (Hao) Guan[SMTP:xxxxx@broadcom.com]
> Reply To: Windows System Software Devs Interest List
> Sent: Thursday, January 18, 2007 1:20 AM
> To: Windows System Software Devs Interest List
> Subject: RE: [ntdev] Kernel debug in Windbg using tcp - is it possible?
>
> Static analysis is based on certain known rules. It’s not going to help
> if my code doesn’t break these rules.
>
Sure but nobody is perfect and static analysis tool helps to find “logical typos” early. They’re limited but if configured properly, they can eliminate many bugs which could be found by single stepping.

> I used to write an E&M solver to numerically obtain the time domain
> solution for partial differential equations derived from the Maxwell’s
> Equations given arbitrary initial values and boundary conditions under
> sinusoidal excitation. It would take hours to find the solution
> depending on how complicated the circuit components are geometrically
> shaped and physically arranged. I found single stepping and break
> pointing are most efficient ways to debug such program.
>
Wasn’t is because you didn’t write traces with the code? It traces aren’t on place before debugging is started, single stepping can be more efficient in that moment than adding necessary traces.

In my experience, with good traces and asserts any algorithm can be debugged very efficiently.

> Once a while, I heard “single stepping is there for inexperienced”, or>
> “real programmer doesn’t need a debugger”. I disagree. However, I do
> agree that in many cases, tracing can be more efficient.
>
In my experience single stepping is mainly used by inexperienced programmers because quickly helps them to find errors in their code. With more experience they tend to make less such errors and need to debug more complicated scenarios where single stepping is just time waste.

I’m not against debuggers at all. Strategically placed breakpoint can save a lot of time, sometimes. Debuggers are helpful for reverse engineering which is unfortunately necessary for driver development. Traces have other advantages but they have to be adopted by developers. For this purpose, it is necessary to have good infrastructure at first (I mean traces implementation and the way how to control them in runtime and no, ETW isn’t a good one). Second, developers have to be persuaded traces have to be written regularly and immediatelly so they’re integral part of code and not just a tool to solve immediate problem. Fortunately, I was already able to persuade my coworkers to do this and traces became mandatory for the last projects. My main point is traces aren’t there only for them but also for others. If any problem occurs, anybody can enable traces and quickly find what fails. And traces can be used by QA or even customer and developer doesn’t need to waste time reproducing problem at his machine (which can be impossible).

With single stepping you always start from the beginning. Good traces, once written, are always available.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

>
>
>

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

David_J_Craig · January 18, 2007, 3:11pm

How about keeping the command line for windbg active when the keyboard is
used even when the source code window is active? Does windbg allow you to
modify your source and recompile - I don’t think so. The only times the
command line should not be the receiver of keystrokes is when another window
has an edit in progress - memory window, locals when data has been selected,
etc.

“Michal Vodicka” wrote in message
news:xxxxx@ntdev…
OK, take it as a result of my frustration from Vista USB stack It is
better to write ridiculous posts than examine logs which show new and new
bugs which I can’t solve :-/ I really wonder what USB team used to debug
their drivers. If anything…

Don’t underestimate aesthetic aspect of engineering. Good work is usually
also nice and not only in programming.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

> ----------
> From:
> xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com]
> on behalf of Arlie Davis[SMTP:xxxxx@microsoft.com]
> Reply To: Windows System Software Devs Interest List
> Sent: Thursday, January 18, 2007 6:15 AM
> To: Windows System Software Devs Interest List
> Subject: RE: [ntdev] Kernel debug in Windbg using tcp - is it possible?
>
> I know engineers love to argue, but this is ridiculous. I can’t belive
> this thread has become a “debuggers vs. printfs” argument. Puh-leeze. Do
> whatever it takes to drive your code to quality. Hell, some of the best
> debugging I’ve done has been with a print-out and a red pen, a cup of
> coffee, and not a computer in sight.
>
> I expect any competent engineer to have a strong working knowledge of the
> debugger(s) that apply to the environment they work in. And I expect them
> to build components with good tracing, internal asserts, etc. They are
> all different approaches to the same goal, quality.
>
> If you like tracing, fine, go for it. But I have found interactive
> debuggers to be an incredibly powerful tool for investigating what was
> not anticipated at design-time, and therefore doesn’t have prints /
> asserts / whatever.
>
> Do engineers in other professions waste as much breath on stuff like this?
> Do civil engineers argue about whether they should load test their bridges
> OR use finite element analysis? For some weird reason, programming
> often seems to be more aesthetics than engineering.
>
>
> ________________________________________
> From: xxxxx@lists.osr.com
> [xxxxx@lists.osr.com] On Behalf Of Michal Vodicka
> [xxxxx@upek.com]
> Sent: Wednesday, January 17, 2007 7:30 PM
> To: Windows System Software Devs Interest List
> Subject: RE: [ntdev] Kernel debug in Windbg using tcp - is it possible?
>
> > ----------
> > From:
> > xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com]
> > on behalf of Calvin (Hao) Guan[SMTP:xxxxx@broadcom.com]
> > Reply To: Windows System Software Devs Interest List
> > Sent: Thursday, January 18, 2007 1:20 AM
> > To: Windows System Software Devs Interest List
> > Subject: RE: [ntdev] Kernel debug in Windbg using tcp - is it
> > possible?
> >
> > Static analysis is based on certain known rules. It’s not going to help
> > if my code doesn’t break these rules.
> >
> Sure but nobody is perfect and static analysis tool helps to find “logical
> typos” early. They’re limited but if configured properly, they can
> eliminate many bugs which could be found by single stepping.
>
> > I used to write an E&M solver to numerically obtain the time domain
> > solution for partial differential equations derived from the Maxwell’s
> > Equations given arbitrary initial values and boundary conditions under
> > sinusoidal excitation. It would take hours to find the solution
> > depending on how complicated the circuit components are geometrically
> > shaped and physically arranged. I found single stepping and break
> > pointing are most efficient ways to debug such program.
> >
> Wasn’t is because you didn’t write traces with the code? It traces aren’t
> on place before debugging is started, single stepping can be more
> efficient in that moment than adding necessary traces.
>
> In my experience, with good traces and asserts any algorithm can be
> debugged very efficiently.
>
> > Once a while, I heard “single stepping is there for inexperienced”, or>
> > “real programmer doesn’t need a debugger”. I disagree. However, I do
> > agree that in many cases, tracing can be more efficient.
> >
> In my experience single stepping is mainly used by inexperienced
> programmers because quickly helps them to find errors in their code. With
> more experience they tend to make less such errors and need to debug more
> complicated scenarios where single stepping is just time waste.
>
> I’m not against debuggers at all. Strategically placed breakpoint can save
> a lot of time, sometimes. Debuggers are helpful for reverse engineering
> which is unfortunately necessary for driver development. Traces have other
> advantages but they have to be adopted by developers. For this purpose, it
> is necessary to have good infrastructure at first (I mean traces
> implementation and the way how to control them in runtime and no, ETW
> isn’t a good one). Second, developers have to be persuaded traces have to
> be written regularly and immediatelly so they’re integral part of code and
> not just a tool to solve immediate problem. Fortunately, I was already
> able to persuade my coworkers to do this and traces became mandatory for
> the last projects. My main point is traces aren’t there only for them but
> also for others. If any problem occurs, anybody can enable traces and
> quickly find what fails. And traces can be used by QA or even customer and
> developer doesn’t need to waste time reproducing problem at his machine
> (which can be impossible).
>
> With single stepping you always start from the beginning. Good traces,
> once written, are always available.
>
> Best regards,
>
> Michal Vodicka
> UPEK, Inc.
> [xxxxx@upek.com, http://www.upek.com]
>
>
> >
> >
> >
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

Beverly_Brown · January 18, 2007, 3:44pm

On 1/18/07, Arlie Davis wrote:
> If you like tracing, fine, go for it. But I have found interactive debuggers to be an incredibly powerful tool for investigating what was not anticipated at design-time, and therefore doesn’t have prints / asserts / whatever.

Exactly! That is all I was trying to say (And doing a very poor job of
it apparently). What I took issue with was the idea that debuggers
were only for beginners but I know that there are many experienced
driver developers that depend on debuggers as one of the many tools
that they use. Static code checkers and traces are also very valuable
tools.

And let’s face it, if everything could be anticipated at design time
in order to implement every trace you would ever need to debug a
problem, you wouldn’t need them because you would have foreseen
everything and therefore would not have any bugs in your code.

Beverly

OSR_Community_User · January 18, 2007, 3:47pm

Well, if it’s any consolation, I only knew about it because someone instructed me to use it a couple of years ago. Sxdld I had to figure out for myself, but once you know the sx command exists, the rest isn’t quite so bad.

OSR_Community_User · January 18, 2007, 4:33pm

Agreed. Kissinger said essentially the same thing about academia, only
in a considerably less flattering, but equally applicable way.

mm

>> xxxxx@probo.com 2007-01-18 13:34 >>>
Arlie Davis wrote:
I know engineers love to argue, but this is ridiculous.
…
Do engineers in other professions waste as much breath on stuff like
this? Do civil engineers argue about whether they should load test
their bridges *OR* use finite element analysis?

Heck, yes. I’m surprised you wouldn’t expect this. The more
technical
the forum, the more minute the disagreements. I could forward to you
the 20-message thread from my clarinet mailing list debating exactly
what grit of sandpaper is most appropriate for final adjustment of
commercially prepared reeds, or the even longer thread debating
onion-skin pads vs kid leather pads. Now, THOSE are vitally important
issues.

Be happy that we aren’t debating spaces vs hard tabs. I know the
correct answer, of course.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Michal_Vodicka-2 · January 18, 2007, 6:09pm

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Beverly Brown[SMTP:xxxxx@gmail.com]
Reply To: Windows System Software Devs Interest List
Sent: Thursday, January 18, 2007 9:42 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Kernel debug in Windbg using tcp - is it possible?

And let’s face it, if everything could be anticipated at design time
in order to implement every trace you would ever need to debug a
problem, you wouldn’t need them because you would have foreseen
everything and therefore would not have any bugs in your code.

Let’s continue ridiculous discussion to avoid sad reality

You don’t need to anticipate everything. You need to have something which verifies your code is correct. Static analysis verifies predefined rules. Asserts verify expected conditions, validate function parameters, data consistency and so on. Traces allow to display and verify code flow and timing. It is usually enough to quickly find what is wrong when a problem occurs. Once problem is isolated, it usually isn’t hard to examine relevant part of code and find a bug. Yes, breakpoint and debugger can help in this phase if problem is easy to reproduce.

You may not have any bug in your code and still encounter a problem. Driver lives in the OS environment and interracts with it. Could I anticipate Vista USB drivers block some requests for 5 - 30 seconds at some machines? Not even in the worst nightmare. Traces I wrote few years before Vista unfortunate release show me it is the case. Debugger? It occurs once per several minutes of intensive USB communication and apparently depend on whole OS timing. Yes, I’ll probably add an assert there and invoke bugcheck to see OS state in that moment. Next time I’ll anticipate this kind of OS misbehaviour and will use asserts to check if some function doesn’t take too much time.

What I want to say is any problem I can imagine which can be solved using debugger and single stepping can be equally solved with traces. What is more efficient in given moment depends on actual code. How it was prepared for debugging and how it is manageable. On the other hand, there is class of problems which can’t be solved by single stepping and traces still help.

And why I write it? Maybe because years before somebody here criticized Linus T. for his aversion against kernel debugger for Linux. I also didn’t agree then but it led me to start thinking about it and in turn develop more efficient ways how to debug code. So maybe somebody else could start thinking now

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

OSR_Community_User · January 18, 2007, 6:56pm

It might be true that lot of debugging can possibly be done using traces,
asserts, etc. But not having a debugger at hand will surely make me very
very uncomfortable.

The concept of not having good debugger is not new, and at least it did not
start at Linux time. Back in late 70s and early 80s it was quite prevalent
in IBM software divisions. But for some real-time and close to HW interface
ICE was always there with other analyzer.

There are two particular situations when a good debugger is very handy -

If one has to deal with lots of code ( anything more than 50,000 lines
and definitely anything more than 100,000 lines). Usually these kind of
situations does not arise for everybody, but it is there. And who would want
to use debugger ( except for system probing) since it is slow and quite
painful when difficult bugs are there.
There are times when inserting debug trace will not show the condition
you want to reproduce ( specially in a race ).

And there are other situations when one might use it.

The dynamics of OS software is quite complex and time dependent. So I really
don’t see how debugger can be avoided.

One thing I particulary like to see from software community ( any community)
is the resistence of writing more code. Almost always I seem to see people (
perhaps including me) write more codes than necessary and end up bloating
components to be an easy candidate to put thru the periscope ( i.e. the
debugger).

-pro

On 1/18/07, Michal Vodicka wrote:
>
> > ----------
> > From: xxxxx@lists.osr.com[
> SMTP:xxxxx@lists.osr.com] on behalf of Beverly Brown[SMTP:
> xxxxx@gmail.com]
> > Reply To: Windows System Software Devs Interest List
> > Sent: Thursday, January 18, 2007 9:42 PM
> > To: Windows System Software Devs Interest List
> > Subject: Re: [ntdev] Kernel debug in Windbg using tcp - is it
> possible?
> >
> > And let’s face it, if everything could be anticipated at design time
> > in order to implement every trace you would ever need to debug a
> > problem, you wouldn’t need them because you would have foreseen
> > everything and therefore would not have any bugs in your code.
> >
> Let’s continue ridiculous discussion to avoid sad reality
>
> You don’t need to anticipate everything. You need to have something which
> verifies your code is correct. Static analysis verifies predefined rules.
> Asserts verify expected conditions, validate function parameters, data
> consistency and so on. Traces allow to display and verify code flow and
> timing. It is usually enough to quickly find what is wrong when a problem
> occurs. Once problem is isolated, it usually isn’t hard to examine relevant
> part of code and find a bug. Yes, breakpoint and debugger can help in this
> phase if problem is easy to reproduce.
>
> You may not have any bug in your code and still encounter a problem.
> Driver lives in the OS environment and interracts with it. Could I
> anticipate Vista USB drivers block some requests for 5 - 30 seconds at some
> machines? Not even in the worst nightmare. Traces I wrote few years before
> Vista unfortunate release show me it is the case. Debugger? It occurs once
> per several minutes of intensive USB communication and apparently depend on
> whole OS timing. Yes, I’ll probably add an assert there and invoke bugcheck
> to see OS state in that moment. Next time I’ll anticipate this kind of OS
> misbehaviour and will use asserts to check if some function doesn’t take too
> much time.
>
> What I want to say is any problem I can imagine which can be solved using
> debugger and single stepping can be equally solved with traces. What is more
> efficient in given moment depends on actual code. How it was prepared for
> debugging and how it is manageable. On the other hand, there is class of
> problems which can’t be solved by single stepping and traces still help.
>
> And why I write it? Maybe because years before somebody here criticized
> Linus T. for his aversion against kernel debugger for Linux. I also didn’t
> agree then but it led me to start thinking about it and in turn develop more
> efficient ways how to debug code. So maybe somebody else could start
> thinking now
>
> Best regards,
>
> Michal Vodicka
> UPEK, Inc.
> [xxxxx@upek.com, http://www.upek.com]
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

Michal_Vodicka-2 · January 18, 2007, 8:19pm

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Prokash Sinha[SMTP:xxxxx@gmail.com]
Reply To: Windows System Software Devs Interest List
Sent: Friday, January 19, 2007 12:55 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Kernel debug in Windbg using tcp - is it possible?

There are two particular situations when a good debugger is very handy -

If one has to deal with lots of code ( anything more than 50,000 lines and definitely anything more than 100,000 lines). Usually these kind of situations does not arise for everybody, but it is there. And who would want to use debugger ( except for system probing) since it is slow and quite painful when difficult bugs are there.

I’m not sure if I understand you. Having a lot of code especially written by a lot of different people (in our case by several teams) is exactly the case where traces are helpful (if they’re used regularly) and where even good debugger is almost useless. When a problem is reported, we usually only see symptoms in GUI part and the root cause can be everywhere. Traces usually reveal it within minutes and debugger? Where to start and who should do it?

There are times when inserting debug trace will not show the condition you want to reproduce ( specially in a race ).

Yes, traces influence timing and in some cases problem disappears when enabled. However, in my experience debugger is equally useless for races debugging at least unless you find there are races and where they occur (which means you already analyzed the main problem). In such cases I start playing with trace levels so only part of traces is enabled and if I’m lucky, problem can be reproduced. If not, next possibility is to invoke bugcheck when problem is detected and analyze code state. If it doesn’t help, I usually change trace macros implementation to be less intrusive. Traces don’t format and print strings and only quickly store basic info (time, line, parameters) to the big flat buffer driver allocates. Stored info needs postprocessing but it is usually enough to reveal races.

The dynamics of OS software is quite complex and time dependent. So I really don’t see how debugger can be avoided.

This is exactly the reason why I avoid it

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

OSR_Community_User · January 18, 2007, 11:38pm

Does this conversation serve any useful purpose? Who cares whether you like debuggers or trace statements?

Why don’t we move on to a more useful discussion, like whether I like Coke or Pepsi more?

Maybe we can move this to the C2 patterns wiki, which seems to have an incredible knack for bloating every conversation.

From: xxxxx@lists.osr.com [xxxxx@lists.osr.com] On Behalf Of Michal Vodicka [xxxxx@upek.com]
Sent: Thursday, January 18, 2007 5:19 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Kernel debug in Windbg using tcp - is it possible?

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Prokash Sinha[SMTP:xxxxx@gmail.com]
Reply To: Windows System Software Devs Interest List
Sent: Friday, January 19, 2007 12:55 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Kernel debug in Windbg using tcp - is it possible?

There are two particular situations when a good debugger is very handy -

If one has to deal with lots of code ( anything more than 50,000 lines and definitely anything more than 100,000 lines). Usually these kind of situations does not arise for everybody, but it is there. And who would want to use debugger ( except for system probing) since it is slow and quite painful when difficult bugs are there.

I’m not sure if I understand you. Having a lot of code especially written by a lot of different people (in our case by several teams) is exactly the case where traces are helpful (if they’re used regularly) and where even good debugger is almost useless. When a problem is reported, we usually only see symptoms in GUI part and the root cause can be everywhere. Traces usually reveal it within minutes and debugger? Where to start and who should do it?

There are times when inserting debug trace will not show the condition you want to reproduce ( specially in a race ).

Yes, traces influence timing and in some cases problem disappears when enabled. However, in my experience debugger is equally useless for races debugging at least unless you find there are races and where they occur (which means you already analyzed the main problem). In such cases I start playing with trace levels so only part of traces is enabled and if I’m lucky, problem can be reproduced. If not, next possibility is to invoke bugcheck when problem is detected and analyze code state. If it doesn’t help, I usually change trace macros implementation to be less intrusive. Traces don’t format and print strings and only quickly store basic info (time, line, parameters) to the big flat buffer driver allocates. Stored info needs postprocessing but it is usually enough to reveal races.

The dynamics of OS software is quite complex and time dependent. So I really don’t see how debugger can be avoided.

This is exactly the reason why I avoid it

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

OSR_Community_User · January 19, 2007, 11:55am

No this conversation does not serve any useful purpose, at least in my
opinion.

It was a very strong statement, that debugger is almost useless.

Very many smart programmers knows how to use kernel and user mode debuggers,
and that is why these are still in existence.

Traces are good upto a point and that is my take. KM code is inherently
asynchronous and I’ve seen how traces ( with various log level can bloat,
just like the way we are bloating this thread) could become very difficult
to anaylze. Then there are times when you don’t have much of a stack trace,
and you try to reconstruct to get to the real culprit ( needless to say,
quite a few articles were written on it).

At any rate, it does not matter who likes what!. What matters most is how to
use what and when. And good kernel debugger is still very useful to me. ( As
they say, if you want a friend keep a debugger as your pet

_TerminateThread(NULL, NULL, … );

-pro

On 1/18/07, Michal Vodicka wrote:
>
> > ----------
> > From: xxxxx@lists.osr.com[
> SMTP:xxxxx@lists.osr.com] on behalf of Prokash Sinha[
> SMTP:xxxxx@gmail.com]
> > Reply To: Windows System Software Devs Interest List
> > Sent: Friday, January 19, 2007 12:55 AM
> > To: Windows System Software Devs Interest List
> > Subject: Re: [ntdev] Kernel debug in Windbg using tcp - is it
> possible?
> >
> > There are two particular situations when a good debugger is very handy -
> >
> > 1) If one has to deal with lots of code ( anything more than 50,000
> lines and definitely anything more than 100,000 lines). Usually these kind
> of situations does not arise for everybody, but it is there. And who would
> want to use debugger ( except for system probing) since it is slow and quite
> painful when difficult bugs are there.
> >
> I’m not sure if I understand you. Having a lot of code especially written
> by a lot of different people (in our case by several teams) is exactly the
> case where traces are helpful (if they’re used regularly) and where even
> good debugger is almost useless. When a problem is reported, we usually only
> see symptoms in GUI part and the root cause can be everywhere. Traces
> usually reveal it within minutes and debugger? Where to start and who should
> do it?
> >
> > 2) There are times when inserting debug trace will not show the
> condition you want to reproduce ( specially in a race ).
> >
> Yes, traces influence timing and in some cases problem disappears when
> enabled. However, in my experience debugger is equally useless for races
> debugging at least unless you find there are races and where they occur
> (which means you already analyzed the main problem). In such cases I start
> playing with trace levels so only part of traces is enabled and if I’m
> lucky, problem can be reproduced. If not, next possibility is to invoke
> bugcheck when problem is detected and analyze code state. If it doesn’t
> help, I usually change trace macros implementation to be less intrusive.
> Traces don’t format and print strings and only quickly store basic info
> (time, line, parameters) to the big flat buffer driver allocates. Stored
> info needs postprocessing but it is usually enough to reveal races.
> >
> > The dynamics of OS software is quite complex and time dependent. So I
> really don’t see how debugger can be avoided.
> >
> This is exactly the reason why I avoid it
>
> Best regards,
>
> Michal Vodicka
> UPEK, Inc.
> [xxxxx@upek.com, http://www.upek.com]
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

raj_r · January 20, 2007, 12:55pm

On 1/17/07, Peter Wieland wrote:

> I would be interested in a list of features that SoftICE had that made
> it more useful than WinDBG aside from single-machine support. Better
> disassemble? Better breakpoint support? Better single step ability?
> Better functionality without symbols?

i have seen no one talking about using this fine debugger in ring 3 user mode for non source less application in this thread

i know skywing who does a really excellent job of demystyfying windbg in his blog must surely have some real gripes but i see he has posted only one reply to this whole thread

if you compare windbg with ollydbg ( a free user mode debugger authored by oleh yuschuk) you would really see it has a tremendous advantage in terms of userfriendly ness when you are dealing with lets say a malware

for example lets take the issue of finding referances for a certain call
ollydbg would provide you a complete list of all the places from where this call could be called whereas when i break on a certain place in windbg i am really at a wild jungle with a thick blindfold strapped on my eyes

i also have a gripe about the way the disassembly instruction are shown when you are doing symbol less debugging i would actually prefer a switch or commandline option that would entirely turn off this symbol seeking stuff

for example
in this snippet the bogus symbol kdtest!_GetExceptDllInfo confuses more than helping

0:000> u 401012 L1
kdtest+0x1012:
00401012 a10fa14000 mov eax,dword ptr [kdtest!_GetExceptDLLinfo+0x90b6 (0040a10f)]

also you do have a special disassembly window but even in that window scrolling to have a continous view of disassembly is really a pain when the bounds exceeds it prints out a new block and starts from top completly blowing out what you viewing

also one could really do with dynamic labelling and commenting and having it saved along with wisecracks

may be a few more things

but this forum doesnt look like the place to gripe

OSR_Community_User · January 20, 2007, 10:59pm

I would request to just get a copy of softice and use it to see what is
there that could be useful in windbg.

Though I’m not as rigorous as I used to be when it comes to using debuggers,
here is some of them -

– A single session should allow to debug user mode as well as krnl mode.
SoftIce does that ( and my favorite pet :-).
– If I type a bad command or try to say x mydriver!myfunc*, if it windbg
does not find it, it should not go crazy and try to pull symbols over the
internet and make the debugging session slow.

– softice has a selective setting of active symbols, that might be helpful
too.

– windbg gets confused at times in terms of setting sympath, and could not
load private syms.

– windbg is not as snappy as softice.

– command completion is just brilliant in softice. How it would be possible
in windbg is not known to me, since there are bult-in dot commands and
extension commands.

– of course single machine ( as well as live debugging would help, and may
have been in vista ). using vms and create two machine env. for debugging is
not always useful.

– for any workspace, when loaded, it ask for should I save this in base
workspace, say no, checked the box for don’t ask me again, and it asks
again. Extreemly irritating.

– I would not try to take a clean slate for workspace, opened couple
windows, and try to line them up to customize my docking. It is a phd thesis
by iteself :). So either creates more themes, or let the user have bit more
friendly control to snap thoses windows to the respective frames.

Just a thought.

-pro

On 1/20/07, raj_r wrote:
>
>
>
> On 1/17/07, Peter Wieland wrote:
>
> > I would be interested in a list of features that SoftICE had that made
> > it more useful than WinDBG aside from single-machine support. Better
> > disassemble? Better breakpoint support? Better single step ability?
> > Better functionality without symbols?
>
> i have seen no one talking about using this fine debugger in ring 3 user
> mode for non source less application in this thread
>
> i know skywing who does a really excellent job of demystyfying windbg in
> his blog must surely have some real gripes but i see he has posted only one
> reply to this whole thread
>
> if you compare windbg with ollydbg ( a free user mode debugger authored by
> oleh yuschuk) you would really see it has a tremendous advantage in terms
> of userfriendly ness when you are dealing with lets say a malware
>
> for example lets take the issue of finding referances for a certain call
> ollydbg would provide you a complete list of all the places from where
> this call could be called whereas when i break on a certain place in windbg
> i am really at a wild jungle with a thick blindfold strapped on my eyes
>
> i also have a gripe about the way the disassembly instruction are shown
> when you are doing symbol less debugging i would actually prefer a switch or
> commandline option that would entirely turn off this symbol seeking stuff
>
> for example
> in this snippet the bogus symbol kdtest!_GetExceptDllInfo confuses more
> than helping
>
> 0:000> u 401012 L1
> kdtest+0x1012:
> 00401012 a10fa14000 mov eax,dword ptr
> [kdtest!_GetExceptDLLinfo+0x90b6 (0040a10f)]
>
> also you do have a special disassembly window but even in that window
> scrolling to have a continous view of disassembly is really a pain when the
> bounds exceeds it prints out a new block and starts from top completly
> blowing out what you viewing
>
> also one could really do with dynamic labelling and commenting and having
> it saved along with wisecracks
>
> may be a few more things
>
> but this forum doesnt look like the place to gripe
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

David_J_Craig · January 20, 2007, 11:23pm

When adding items to the symbol and source paths it ought to stay in the directory previously selected. Having to drill down from the top again is just inefficient. Why when using VMWare via a COM1 pipe do you have to kill and re-execute windbg just to get a connection. I start WMWare and then have Windows in the VM stop at the boot.ini with a -1 so that I can start windbg before selecting the debug line.
“Prokash Sinha” wrote in message news:xxxxx@ntdev…
I would request to just get a copy of softice and use it to see what is there that could be useful in windbg.

Though I’m not as rigorous as I used to be when it comes to using debuggers, here is some of them -

– A single session should allow to debug user mode as well as krnl mode. SoftIce does that ( and my favorite pet :-).
– If I type a bad command or try to say x mydriver!myfunc*, if it windbg does not find it, it should not go crazy and try to pull symbols over the internet and make the debugging session slow.

– softice has a selective setting of active symbols, that might be helpful too.

– windbg gets confused at times in terms of setting sympath, and could not load private syms.

– windbg is not as snappy as softice.

– command completion is just brilliant in softice. How it would be possible in windbg is not known to me, since there are bult-in dot commands and extension commands.

– of course single machine ( as well as live debugging would help, and may have been in vista ). using vms and create two machine env. for debugging is not always useful.

– for any workspace, when loaded, it ask for should I save this in base workspace, say no, checked the box for don’t ask me again, and it asks again. Extreemly irritating.

– I would not try to take a clean slate for workspace, opened couple windows, and try to line them up to customize my docking. It is a phd thesis by iteself :). So either creates more themes, or let the user have bit more friendly control to snap thoses windows to the respective frames.

Just a thought.

-pro

On 1/20/07, raj_r wrote:

On 1/17/07, Peter Wieland wrote:

> I would be interested in a list of features that SoftICE had that made
> it more useful than WinDBG aside from single-machine support. Better
> disassemble? Better breakpoint support? Better single step ability?
> Better functionality without symbols?

i have seen no one talking about using this fine debugger in ring 3 user mode for non source less application in this thread

i know skywing who does a really excellent job of demystyfying windbg in his blog must surely have some real gripes but i see he has posted only one reply to this whole thread

if you compare windbg with ollydbg ( a free user mode debugger authored by oleh yuschuk) you would really see it has a tremendous advantage in terms of userfriendly ness when you are dealing with lets say a malware

for example lets take the issue of finding referances for a certain call
ollydbg would provide you a complete list of all the places from where this call could be called whereas when i break on a certain place in windbg i am really at a wild jungle with a thick blindfold strapped on my eyes

i also have a gripe about the way the disassembly instruction are shown when you are doing symbol less debugging i would actually prefer a switch or commandline option that would entirely turn off this symbol seeking stuff

for example
in this snippet the bogus symbol kdtest!_GetExceptDllInfo confuses more than helping

0:000> u 401012 L1
kdtest+0x1012:
00401012 a10fa14000 mov eax,dword ptr [kdtest!_GetExceptDLLinfo+0x90b6 (0040a10f)]

also you do have a special disassembly window but even in that window scrolling to have a continous view of disassembly is really a pain when the bounds exceeds it prints out a new block and starts from top completly blowing out what you viewing

also one could really do with dynamic labelling and commenting and having it saved along with wisecracks

may be a few more things

but this forum doesnt look like the place to gripe

—
Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

OSR_Community_User · January 21, 2007, 12:17am

“Prokash Sinha” wrote in message news:xxxxx@ntdev…
…
> One thing I particulary like to see from software community ( any community)
> is the resistence of writing more code. Almost always I seem to see people (
> perhaps including me) write more codes than necessary and end up bloating
> components to be an easy candidate to put thru the periscope ( i.e. the
> debugger).

Very true. The more they write, the more they get to debug.
Keeping the code spam-free really pays off.

–PA

Doron_Holan · January 21, 2007, 1:39am

> – If I type a bad command or try to say x mydriver!myfunc*, if it windbg does

not find it, it should not go crazy and try to pull symbols over the internet
and make the debugging session slow.
This feature already exists, see the help for .symopt, specifically .sympopt +0x100 will turn off unqualified symbol loads

SYMOPT_NO_UNQUALIFIED_LOADS
This symbol option disables the symbol handler’s automatic loading of modules. When this option is set and the debugger attempts to match a symbol, it will only search modules which have already been loaded.

This option can be used as a defense against mistyping a symbol name. Normally, a mistyped symbol will cause the debugger to pause while it searches all unloaded symbol files. When this option is active, a mistyped symbol will not be found in the loaded modules, and then the search will terminate.

This option is off by default. It can be activated by using the -snul command-line option. Once the debugger is running, it can be turned on or off by using .symopt+0x100 or .symopt-0x100, respectively.

d

Maxim_S_Shatskih · January 21, 2007, 4:47am

>Very true. The more they write, the more they get to debug.

Keeping the code spam-free really pays off.

I personally use traces a) in some pieces of logic where bugfixing will be
extremely hard and costly - I also use code coverage (by macros) there and b)
in some “interesting” places of my code/OS interaction.

One of the reasons to have traces is to print the device object addresses etc,
so I will quickly know what address to type in !devobj and such.

All my traces are by KdPrint and do not occur in release build. If I need the
error output in the release build - I use Windows event log (better for support
on customers sites). Even some memory allocation failures are logged this way,
in a case if alloc failure caused some functionality to fail.

Anyway - traces are good tool if properly used. So are breakpoints and then the
memory/structures analyzis - I think that, for me, it is the main debugging
method. Single-stepping is IMHO useful after the aforementioned phase, when you
know that you’re already near the bug.

One more code review by yourself the next day is one of the best debugging
methods. It can easily unweil the bugs which would cost around a week to find
and fix, especially be it at customer’s site.

–
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

OSR_Community_User · January 21, 2007, 8:04am

I like using the (relatively) new DML feature of DbgEng with traces, as
it can be used to massively cut back on default debug output without
recompiling, and still provide the same information with only a click or
two.

mm

>> xxxxx@storagecraft.com 2007-01-21 04:47 >>>
Very true. The more they write, the more they get to debug.
Keeping the code spam-free really pays off.

I personally use traces a) in some pieces of logic where bugfixing will
be
extremely hard and costly - I also use code coverage (by macros) there
and b)
in some “interesting” places of my code/OS interaction.

One of the reasons to have traces is to print the device object
addresses etc,
so I will quickly know what address to type in !devobj and such.

All my traces are by KdPrint and do not occur in release build. If I
need the
error output in the release build - I use Windows event log (better for
support
on customers sites). Even some memory allocation failures are logged
this way,
in a case if alloc failure caused some functionality to fail.

Anyway - traces are good tool if properly used. So are breakpoints and
then the
memory/structures analyzis - I think that, for me, it is the main
debugging
method. Single-stepping is IMHO useful after the aforementioned phase,
when you
know that you’re already near the bug.

One more code review by yourself the next day is one of the best
debugging
methods. It can easily unweil the bugs which would cost around a week
to find
and fix, especially be it at customer’s site.

–
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Michal_Vodicka-2 · January 22, 2007, 1:12pm

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Martin O’Brien[SMTP:xxxxx@evitechnology.com]
Reply To: Windows System Software Devs Interest List
Sent: Sunday, January 21, 2007 2:03 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Kernel debug in Windbg using tcp - is it possible?

I like using the (relatively) new DML feature of DbgEng with traces, as
it can be used to massively cut back on default debug output without
recompiling, and still provide the same information with only a click or
two.

Never heard about it – can you elaborate? Thanks.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]