VMWare 4.0 final just went live today, and claims much greater
compatiblity with user and kernel-mode debuggers. I ventured to test
this claim.
Verdict? Not as good as I hoped. They’ve fixed the long-standing
incompatibility between VMWare and SoftIce on the HOST machine, but
SoftIce does not work in any shape or form on the GUEST machine (a step
BACKWARDS from previous versions, which supported at least VGA mode). I
can produce nothing more than a machine lockup when the debugger is
invoked.
Furthermore, WinDbg doesn’t work as advertised when used to debug the
guest. The method described in the VMWare User Guide (and in some old
threads on Usenet) that describes configuring VMWare to hook a serial
port to a pipe on the guest, then hook WinDbg to the pipe on the host
(using an undocumented WinDbg command) simply doesn’t work. WinDbg won’t
connect on break, even though the target machine locks up as if WinDbg
had full control.
The only way I could get WinDbg to work was to download a program called
‘VSPD XP3’ from Eltima Software that simulates a software loopback
between two virtual serial ports (that appear to everyone else as
hardware COM devices). WinDbg and the guest were able to communicate
through this wormhole perfectly well, except now WinDbg crawls along at
typical serial port speeds instead of the much higher speeds I’m led to
believe the pipe method is able to support.
I’d appreciate it if somebody else (Maxim especially, he had a post on
Usenet back in the fall regarding the settings he used to get earlier
versions of the two software packages talking) can try to get WinDbg
working better than I did. (VMWare 4.0 has an eval download).
My setup is this: Windows XP Pro SP1 (both host and guest), Dell
Latitude D600 Pentium M 1.6 ghz, 512 megs memory (soon to be 1 gig)
split about evenly, VMWare 4.0, WinDbg 6.1.17.2, SoftIce 2.7.
(And before you say it’s the Pentium M’s fault, I tried this on a
plain-Jane desktop Pentium 4 with equally dismal results).
My boot.ini options on the guest were this:
/fastdetect /debug /debugport=COM1 /baudrate=38400 (selection of
baudrate didn’t matter)
The VMWare serial port was configured thus:
- Use named pipe
- \.\pipe\windbg
- This end is the server
- The other end is an application
My WinDbg command-line was this:
“” -QY -v -k com:port=\.\pipe\windbg,pipe
- Nicholas Ryan