Kernel debug in Windbg using tcp - is it possible?

NTDEV
21

I hear you Michal. I just posted, but the long and short in my opinion,
is that SI committed suicide by abusive marketing practice, and, while
SI definitely had its issues (although I used the ethernet transport for
years), I think it is a reasonable question to ask how much of a WinDbg
lovefest there would people if people had to pay for it.

mm

>> xxxxx@upek.com 2007-01-16 20:27 >>>

From:
xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com]
on behalf of Don Burn[SMTP:xxxxx@acm.org]
Reply To: Windows System Software Devs Interest List
Sent: Wednesday, January 17, 2007 1:53 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Kernel debug in Windbg using tcp - is it
possible?

Some of us never got SoftIce to not mess up the way the system was
supposed
to work, so I for one would highly recomend against it. I applauded
it
demise or as Mark Twain said ‘‘I didn’t attend the funeral, but I
sent a
nice letter saying I approved of it.’’

Really wise. Nobody forced you to use SoftICE but for some of us it was
the easiest way how to explore live system. It allowed me to make my
work faster and better within all these years. The demise means there is
no choice, just WinDbg. Great post-mortem analysis tool and rather
clumsy debugger. Debugger is matter of personal preferences but it is
always better if there is a choice. And competition which makes software
better.

BTW, for previous poster, I successfully made SoftICE via TCP/IP
working. Just for curiosity; the real value for me was one computer
development.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

22

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Martin O’Brien[SMTP:xxxxx@evitechnology.com]
Reply To: Windows System Software Devs Interest List
Sent: Wednesday, January 17, 2007 3:10 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Kernel debug in Windbg using tcp - is it possible?

I hear you Michal. I just posted, but the long and short in my opinion,
is that SI committed suicide by abusive marketing practice, and, while
SI definitely had its issues (although I used the ethernet transport for
years), I think it is a reasonable question to ask how much of a WinDbg
lovefest there would people if people had to pay for it.

I agree with both suicide and WinDbg price. It isn’t so long the main argument of WinDbg advocates here was no fee.

Well, I didn’t want to awake old SI versus WinDbg thread. The game is over. I wonder if MS developers aren’t able to do what NuMega did or if the necessity to have two computers is the intention. SI was widely used as hackers’ tool mainly because of its one-machine debugging abilities. Or maybe they don’t care. With access to OS sources they don’t need a tool which helps with reverse engineering and they got used to two machines setup.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

23

Thanks, I understand by now you and others like WinDbg more than SoftIce. As
for “The driver crashes unless it is run
under SoftICE, and we have no way to debug it” I recall reading an article
in the NtInsider explaining the perfect Heisenbug when certain code was run
under WinDbg it would run perfectly well and would crash as soon as the
debugger was absent.
( http://www.osronline.com/article.cfm?id=380 )

Interesting issues people here are raising are:

-Wouldn’t it be nice if we could do kernel debugging over a network
connection ?
-Wouldn’t it be nice if we didn’t need to carry around a bunch of hardware
including multiple systems just to be able to do debugging but instead could
just do it on one single machine ?
-Wouldn’t it improve the quality of WinDbg if there were some competition
and another kernel debugger were around ?
-Would WinDbg still be so popular if we had to pay for it ?
-Was MS right to knock out SoftIce from the competition by inducing heavy
kernel restrictions with Patchguard ?
-Wouldn’t it be nice if installing the debugger, getting the symbols and
connections right wouldn’t actually be a much harder task than anything else
in kernel land including writing an encryption file system filter driver ?
-Why WinDdbg doesn’t comply to the user interface guidelines of MS? Wouldn’t
it be nice if they just wouldn’t hexdump the crap out of us just because we
are kernel developers and we can read opcode anyway ? This Windbg is using
an absurd unintuitive syntax which is totally incompatible with both short
term and long term human memory, paged or non paged.

If you have any comments, I would really appreciate your or other experts
views on these. My vision is that the people behind SoftIce really tried
hard and deserve merit for it. No matter what debugger is used, I see
debugging as a non-constructive evil which is sometimes unavoidable and like
a poisonous drug should only be used as a last recourse if really no other
options are available. I prefer investing my time in improving the quality
of my coding rather than setting up quirky debuggers, less there will be a
need for it, at least for the code I write myself.

/Daniel

“Don Burn” wrote in message news:xxxxx@ntdev…
> Actually as a consultant there were times I was forced to use SoftICE.
> About 3 years ago, I started telling my customers there was a 25%
> surcharge if I had to use that piece of crap. Everytime I used it I found
> places where they altered the normal behavior of the system. I did
> benefit once, since I had a customer call me saying “The driver crashes
> unless it is run under SoftICE, and we have no way to debug it” Between
> using WinDBG and driver verifier I found over 10 bugs that SoftICE had
> obscured.
>
>

24

Ah - I love conspiracy theory Tuesdays. I console myself with the
knowledge that if we did charge for WinDBG or if we had ever supported
single machine debugging that an equal number of theories about how it
was a plot by MS to increase (something) would abound.

I would be interested in a list of features that SoftICE had that made
it more useful than WinDBG aside from single-machine support. Better
disassemble? Better breakpoint support? Better single step ability?
Better functionality without symbols?

Ethernet support sounds like one thing. Did it work with any Ethernet
controller, or just one or two? Was there any security on it?

I suspect I can search the archive to find this in bits and pieces - did
anyone ever make an exhaustive list?

-p

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Michal Vodicka
Sent: Tuesday, January 16, 2007 6:41 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Kernel debug in Windbg using tcp - is it possible?

From:
xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com
] on behalf of Martin O’Brien[SMTP:xxxxx@evitechnology.com]
Reply To: Windows System Software Devs Interest List
Sent: Wednesday, January 17, 2007 3:10 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Kernel debug in Windbg using tcp - is it
possible?

I hear you Michal. I just posted, but the long and short in my
opinion,
is that SI committed suicide by abusive marketing practice, and, while
SI definitely had its issues (although I used the ethernet transport
for
years), I think it is a reasonable question to ask how much of a
WinDbg
lovefest there would people if people had to pay for it.

I agree with both suicide and WinDbg price. It isn’t so long the main
argument of WinDbg advocates here was no fee.

Well, I didn’t want to awake old SI versus WinDbg thread. The game is
over. I wonder if MS developers aren’t able to do what NuMega did or if
the necessity to have two computers is the intention. SI was widely used
as hackers’ tool mainly because of its one-machine debugging abilities.
Or maybe they don’t care. With access to OS sources they don’t need a
tool which helps with reverse engineering and they got used to two
machines setup.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

25

Comments inline:

“Daniel Terhell” wrote in message
news:xxxxx@ntdev…
> Thanks, I understand by now you and others like WinDbg more than SoftIce.
> As for “The driver crashes unless it is run
> under SoftICE, and we have no way to debug it” I recall reading an
> article in the NtInsider explaining the perfect Heisenbug when certain
> code was run under WinDbg it would run perfectly well and would crash as
> soon as the debugger was absent.
> ( http://www.osronline.com/article.cfm?id=380 )

Well none of the SoftICE problems were heisenbug’s they were where the
debugger hooked the OS and replaced an OS supplied routine with one of its
own. One of the bugs had been there 4 years (since I had a customer report
back then after I found it!). Heisenbugs will always be there, but the “we
know how the OS should work, better than Microsoft attitude at NuMega was
its own killer”

> Interesting issues people here are raising are:
>
> -Wouldn’t it be nice if we could do kernel debugging over a network
> connection ?

Not particularily, since I would not trust it if it shared a card with the
OS’es use of the network, and if not then I am not gaining much over 1394
if I have to put a card in.

> -Wouldn’t it be nice if we didn’t need to carry around a bunch of
> hardware including multiple systems just to be able to do debugging but
> instead could just do it on one single machine ?

There are way too many heisenbug’s with single machine debugging, such that
I started advocating against it close to 30 years ago. With the power of
todays machines, if you can live with the heisenbugs and depending on the
type of driver you can do this with a virtual machine. I suspect as the
hardware assist’s for VM’s come in more, this will even work for a lot of
drivers that it does not work for today.

> -Wouldn’t it improve the quality of WinDbg if there were some competition
> and another kernel debugger were around ?

While I am not a fan of open source, this is one case where I wish
Microsoft would open the source, of at least the KD engine and important
debugger extensions. Then we might see some competition, or at least a lot
of improvement.

> -Would WinDbg still be so popular if we had to pay for it ?

Yes, because it is the only way to trace down a lot of problems. If
Microsoft started charging, and they opened up the KD engine and extensions
for licensing, maybe I would look at a competitior.

> -Was MS right to knock out SoftIce from the competition by inducing heavy
> kernel restrictions with Patchguard ?

SoftICE was dead before Patchguard, it just hadn’t been laid to rest.
Compuware was not funding it, or fixing bugs so it was dead.

> -Wouldn’t it be nice if installing the debugger, getting the symbols and
> connections right wouldn’t actually be a much harder task than anything
> else in kernel land including writing an encryption file system filter
> driver ?

I don’t find it that hard by any means (it was a lot easier than trying to
completely remove SoftICE from a system!).

> -Why WinDdbg doesn’t comply to the user interface guidelines of MS?
> Wouldn’t it be nice if they just wouldn’t hexdump the crap out of us just
> because we are kernel developers and we can read opcode anyway ? This
> Windbg is using an absurd unintuitive syntax which is totally
> incompatible with both short term and long term human memory, paged or
> non paged.
>

I totally agree the interface is poor, and the reliability of the interface
is at close to an all time low. I am debugging a clustered system right
now where I have two instances, and half the time when I swap from one to
the other most of the windows and the frame stay with the previous
instance!!!

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
http://www.windrvr.com
Remove StopSpam from the email to reply

26

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Peter Wieland[SMTP:xxxxx@windows.microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Wednesday, January 17, 2007 6:17 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Kernel debug in Windbg using tcp - is it possible?

Ah - I love conspiracy theory Tuesdays. I console myself with the
knowledge that if we did charge for WinDBG or if we had ever supported
single machine debugging that an equal number of theories about how it
was a plot by MS to increase (something) would abound.

I had no intention to make conspiracy theories and if I did, I apologize. I just wonder why MS doesn’t implement what NuMega developers did long years ago.

I would be interested in a list of features that SoftICE had that made
it more useful than WinDBG aside from single-machine support. Better
disassemble? Better breakpoint support? Better single step ability?
Better functionality without symbols?

It is hard to say. The main advangate which influences user experience is immediate availability when a hotkey is pressed. For me the single machine support is the most important and even if everything else is worse, it would prevail. Simply, when I need to examine or change something, press a hotkey and can do anything.

Next main difference is UI. SI text mode UI was simple, intuitive and easy to control. WinDbg GUI makes me scream. To be honest, CompuWare attempts to make something similar were even worse.

It’d be possible to compare feature by feature but it’d take a lot of time. I’d say it differently – comparing SI and WinDbg user experience is similar as comparing 4NT and command.com (not cmd.exe). The first has intelligent command history, filename completion and a lot of small improvements and even if you can use both to perform the same tasks, the difference is enormous and the first is much more efficient.

The only thing which is better in WinDbg is access to symbols. For SI symbols had to be prepared/downloaded and translated at first and it wasn’t possible to add more symbols without leaving SI. On the other hand, I really hate WinDbg when I make a typo and it then loads symbols for all modules from symbol server which can’t be broken. It can be probably configured somewhere but who can remember how?

Ethernet support sounds like one thing. Did it work with any Ethernet
controller, or just one or two? Was there any security on it?

Yes, there was security on it. It could be configured what IP addresses are allowed to access it and password could be used. I don’t exactly remember if it worked with any Ethernet controller or with all. I vaguelly remember a there was an universal driver which could hook and utilize any existing NDIS driver but I’m not sure.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

27

I actually like a number of the interfaces in WinDbg, but then again I’m
also a big GDB fan. Clearly, traditional interface design is not really
key to me. I do find WinDbg’s command line interface to be completely
alien, and its help texts less than satisfying.

As regards the other issues (particularly networking):

  • I think Ethernet is a whole lot nicer than serial (especially as
    legacy ports continue to disappear), but I think 1394 is even better.
    For one thing, it’s a lot more suited to point-to-point links than
    Ethernet is (there’s no ARP, among other things, which was a key problem
    with early UDP-based kernel debuggers).

  • Ethernet could be made secure in a physical sense by isolating the
    network segment (most people probably have enough room in their machine
    for an extra card, otherwise they probably wouldn’t be using 1394
    either), but the driver issue is really key. Unless you were to
    implement a KD ethernet driver architecture (which would be monumental
    in its uselessness and the effort to get manufacturers to provide KD
    drivers would be unpleasant at best), you would be restricted to a
    select few chipsets (RTL8139 is about as universal as DEC21040, because
    the cards come in boxes of cereal now). We’re limited to a select few
    (read: one, reliably) 1394 chips, but that’s because there aren’t two
    million common 1394 chips out there. We’re limited to essentially one
    serial chip, if only because everything out there still emulates the

  1. I could name 5 common ethernet chips from 5 years ago off the
    top of my head, and I suspect the problem has only gotten worse now with
    the proliferation of motherboard-embedded ethernet chips, some of which
    do a half-hearted job of emulating existing chips.

It would be nice if there were more peer-to-peer busses on the modern
PC. 1394 is about as good as it gets, but it’s still not really
*common*, and the problem of multiple chipset types makes it difficult.
USB, because of its defined controller interfaces, would be great were
it not for the fact that it’s not peer-to-peer (though OTG comes close).
There are, of course, specialized peer-to-peer USB “networking” cables,
which are essentially two USB devices bridged together (and what I
assume Vista uses for its USB debugging mode). This was used for
homebrew development on the Playstation 2 as well, but it was always
extremely hard to find the right cable. So, 1394 and 16550-style RS-232
win due to relative obscurity and legacy retention, respectively.

Am I leaving out any commonly available peer-to-peer busses?

David Riley
Hardware Engineer
Mantaro Networks
20410 Century Blvd
Germantown, MD 20874
(301)528-2244 x532

Don Burn wrote:

I totally agree the interface is poor, and the reliability of the interface
is at close to an all time low. I am debugging a clustered system right
now where I have two instances, and half the time when I swap from one to
the other most of the windows and the frame stay with the previous
instance!!!