COM <-> named pipe debugging in vmware workstation

I successfully connect a target macine running inside VirtualPC to
a host-based windbg through a named pipe.

Now I try to do the same on a VMWare workstation 5.0.

I redirect virtual’s COM1 or COM2 to \.\pipe\dbgpipe which is
what I use for VPC but no matter what I do with other VMWare’s
COM port settings (client vs server, app or VM on the other end,
yield on poll), host-based windbg does not see the target
[VMWare virtual] machine but is happy when the target runs
inside a VPC.

boot.ini’s are identical on both virtual machines, with records like

multi(0)disk(0)rdisk(0)partition(1)\WINDOWS=“win + COM1 +
ntbtlog.txt” /fastdetect /debug /debugport=com1
/baudrate=115200 /bootlog

VMWare on-line help brought nothing new. My guess is that
it should work, what can be possibly wrong with a named pipe?

Did anyone have this problem and solved it (or did not have this
VMWare-specific problem at all, for that matter)?

Maybe, it’s some security policy setting that I’m missing?

Any hints would be greatly appreciated.

Regards,
Alex Shvedov

I use this method for debugging all the time. To get it working, I created
the virtual COM port in the virtual machine to be a named pipe, as you did.
Then I opened WInDbg and specified to connect via a named pipe, the only way
I managed this was by running WinDbg from a command line specifying the
following:

start windbg -W Pipe -b -k com:pipe,port=\.\pipe\dbgpipe,resets=0

Note that this will also break on entry into the system, I do this to ensure
connectivity. It also uses the workspace profile I created called Pipe,
which contains the symbol paths etc.

I have encountered an issue with VPC that requires me to restart the
debugger whenever I restart the virtual machine. This is not always the case
with VMWare but it does happen.

Pete

Kernel Drivers
Windows Filesystem and Device Driver Consulting
www.KernelDrivers.com
(303)546-0300

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@Home
Sent: Tuesday, July 19, 2005 1:25 PM
To: Kernel Debugging Interest List
Subject: [windbg] COM <-> named pipe debugging in vmware workstation

I successfully connect a target macine running inside VirtualPC to
a host-based windbg through a named pipe.

Now I try to do the same on a VMWare workstation 5.0.

I redirect virtual’s COM1 or COM2 to \.\pipe\dbgpipe which is
what I use for VPC but no matter what I do with other VMWare’s
COM port settings (client vs server, app or VM on the other end,
yield on poll), host-based windbg does not see the target
[VMWare virtual] machine but is happy when the target runs
inside a VPC.

boot.ini’s are identical on both virtual machines, with records like

multi(0)disk(0)rdisk(0)partition(1)\WINDOWS=“win + COM1 +
ntbtlog.txt” /fastdetect /debug /debugport=com1
/baudrate=115200 /bootlog

VMWare on-line help brought nothing new. My guess is that
it should work, what can be possibly wrong with a named pipe?

Did anyone have this problem and solved it (or did not have this
VMWare-specific problem at all, for that matter)?

Maybe, it’s some security policy setting that I’m missing?

Any hints would be greatly appreciated.

Regards,
Alex Shvedov


You are currently subscribed to windbg as: xxxxx@kerneldrivers.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

Hello,

* On Tue, Jul 19, 2005 at 03:25:09PM -0600 Peter Scott wrote:

I have encountered an issue with VPC that requires me to restart the
debugger whenever I restart the virtual machine. This is not always
the case with VMWare but it does happen.

With VMWare, I have the following issue: When I boot the virtual machine
(XP), the debugger is not allowed to be started (or the VM will not boot
at all). I have to wait until after the VM has started the bootloader, I
have selected the debug flags, pressed enter, and the VM’s screen
dimensions have changed.

HTH,
Spiro.


Spiro R. Trikaliotis xxxxx@trikaliotis.net
Dipl.-Inform. http://www.trikaliotis.net/
OvG-Universitaet, IVS.EUK Fon: +49-391-67-12566
PF 4120, 39016 Magdeburg Fax: +49-391-67-11161

> the only way I managed this was by running WinDbg from a

command line specifying the following:
start windbg -W Pipe -b -k com:pipe,port=\.\pipe\dbgpipe,resets=0
Thanks for replies, Peter and Spiro, at least I know that it may work as
I expected.

Strange, very strange…

My arrangements are exactly the same for both VPC (no
problems) and VMWare, I also figured out that there is no way

  • for now, at least - to save the info about COM port being
    redirected to a pipe in windbg’s workspace so that
    we have to use prosthetics. In my case it is a dbgpipe.cmd
    wich contains (I split the line)

“C:\Program Files\Debugging Tools for Windows\windbg.exe”
-y srv*D:\SYMBOLS*http://msdl.microsoft.com/download/symbols
-k com:pipe,port=\.\pipe\dbgpipe,resets=0,reconnect

(as you see, no break on boot here, it may be saved in a
workspace; symbols in this script are extras also).

Yet again, this script works ok with VPC but not with VMWare,
and I can not spot any difference whatsoever…

The VMWare-based target sits for some notable time, say
30 secs, probably waiting for windbg response, the port state
is reported by VMWare as “connected” (to what ???), and then
the boot continues as usual, and windbg does not see anything.

Mystery. And I thought that all named pipes are created equal:-)

I have encountered an issue with VPC that requires me to restart the
debugger whenever I restart the virtual machine.
Never ever happened to me, I run as many reboots as I want keeping
windbg running. One more mystery…

With VMWare, I have the following issue: When I boot the virtual machine
(XP), the debugger is not allowed to be started (or the VM will not boot
at all). I have to wait until after the VM has started the bootloader, I
have selected the debug flags, pressed enter, and the VM’s screen
dimensions have changed.
Can you tell me what are the VMWare’s and target’s port settings?
client/server, other end is a VM/other end is an app, yield on poll?

Speed seems not to matter (or does it?).

Anyway, I’ll try it your way, normally I start windbg way before the
target, and it’s ok for VPC.

Regards,
Alex Shvedov

----- Original Message -----
From: “Peter Scott”
To: “Kernel Debugging Interest List”
Sent: Tuesday, July 19, 2005 5:25 PM
Subject: RE: [windbg] COM <-> named pipe debugging in vmware workstation

>
> I use this method for debugging all the time. To get it working, I created
> the virtual COM port in the virtual machine to be a named pipe, as you
> did.
> Then I opened WInDbg and specified to connect via a named pipe, the only
> way
> I managed this was by running WinDbg from a command line specifying the
> following:
>
> start windbg -W Pipe -b -k com:pipe,port=\.\pipe\dbgpipe,resets=0
>
> Note that this will also break on entry into the system, I do this to
> ensure
> connectivity. It also uses the workspace profile I created called Pipe,
> which contains the symbol paths etc.
>
> I have encountered an issue with VPC that requires me to restart the
> debugger whenever I restart the virtual machine. This is not always the
> case
> with VMWare but it does happen.
>
> Pete
>
>
> Kernel Drivers
> Windows Filesystem and Device Driver Consulting
> www.KernelDrivers.com
> (303)546-0300
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@Home
> Sent: Tuesday, July 19, 2005 1:25 PM
> To: Kernel Debugging Interest List
> Subject: [windbg] COM <-> named pipe debugging in vmware workstation
>
> I successfully connect a target macine running inside VirtualPC to
> a host-based windbg through a named pipe.
>
> Now I try to do the same on a VMWare workstation 5.0.
>
> I redirect virtual’s COM1 or COM2 to \.\pipe\dbgpipe which is
> what I use for VPC but no matter what I do with other VMWare’s
> COM port settings (client vs server, app or VM on the other end,
> yield on poll), host-based windbg does not see the target
> [VMWare virtual] machine but is happy when the target runs
> inside a VPC.
>
> boot.ini’s are identical on both virtual machines, with records like
>
> multi(0)disk(0)rdisk(0)partition(1)\WINDOWS=“win + COM1 +
> ntbtlog.txt” /fastdetect /debug /debugport=com1
> /baudrate=115200 /bootlog
>
> VMWare on-line help brought nothing new. My guess is that
> it should work, what can be possibly wrong with a named pipe?
>
> Did anyone have this problem and solved it (or did not have this
> VMWare-specific problem at all, for that matter)?
>
> Maybe, it’s some security policy setting that I’m missing?
>
> Any hints would be greatly appreciated.
>
>
> Regards,
> Alex Shvedov
>
>
>
> —
> You are currently subscribed to windbg as: xxxxx@kerneldrivers.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
>
>
> —
> You are currently subscribed to windbg as: xxxxx@bellsouth.net
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>

Hello Alex,

xxxxx@Home schrieb:

> The VMWare-based target sits for some notable time, say 30 secs,
> probably waiting for windbg response, the port state is reported by
> VMWare as “connected” (to what ???), and then the boot continues as
> usual, and windbg does not see anything.

This is exactly the behaviour I had with older WinDBGs. With newer ones,
the target just doesn’t sit 30seconds or so, but forever.

Thus, my work-around is like this: Don’t start WinDBG until the target
waits. Now, start WinDBG and press Ctrl+Break (this is important, the
automatic break-in does NOT work!). After this, I have control over the
target.

Regards,
Spiro.


Spiro R. Trikaliotis
http://www.trikaliotis.net/

> Thus, my work-around is like this: Don’t start WinDBG until the target

waits. Now, start WinDBG and press Ctrl+Break (this is important, the
automatic break-in does NOT work!). After this, I have control over the
target.
Thanks.
The Ctrl+Break adds to the idea I extracted from your previous post.
My guess is that VMWare virtual starts its COM pipe as a server
so it waits for someone to send in a request.
Stupid, as compared to VPC, and - if I’m right - directly contradicts
VM’s own on-line help.

This is exactly the behaviour I had with older WinDBGs.
I had all this with windbg 6.4, just got the latest tools with 6.5.3.7,
let’s see…

Regards,
Alex Shvedov

----- Original Message -----
From: “Spiro Trikaliotis”
To: “Kernel Debugging Interest List”
Sent: Saturday, July 23, 2005 4:04 AM
Subject: Re: [windbg] COM <-> named pipe debugging in vmware workstation

> Hello Alex,
>
> xxxxx@Home schrieb:
>
>> The VMWare-based target sits for some notable time, say 30 secs,
>> probably waiting for windbg response, the port state is reported by
>> VMWare as “connected” (to what ???), and then the boot continues as
>> usual, and windbg does not see anything.
>
> This is exactly the behaviour I had with older WinDBGs. With newer ones,
> the target just doesn’t sit 30seconds or so, but forever.
>
> Thus, my work-around is like this: Don’t start WinDBG until the target
> waits. Now, start WinDBG and press Ctrl+Break (this is important, the
> automatic break-in does NOT work!). After this, I have control over the
> target.
>
> Regards,
> Spiro.
>
> –
> Spiro R. Trikaliotis
> http://www.trikaliotis.net/
>
> —
> You are currently subscribed to windbg as: xxxxx@bellsouth.net
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>

Just to let you know, I posted my command line Windbg, and it works every
time with VMWare. Not the case with VPC. I have the initial BP set and it
hits it each time on the VMWare virtual machine.

OK, every 10-15 reboots of a VMWare virtual pc I need to restart Windbg, I
call that pretty good.

Pete

Kernel Drivers
Windows Filesystem and Device Driver Consulting
www.KernelDrivers.com
(303)546-0300

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@Home
Sent: Saturday, July 23, 2005 8:32 AM
To: Kernel Debugging Interest List
Subject: Re: [windbg] COM <-> named pipe debugging in vmware workstation

Thus, my work-around is like this: Don’t start WinDBG until the target
waits. Now, start WinDBG and press Ctrl+Break (this is important, the
automatic break-in does NOT work!). After this, I have control over the
target.
Thanks.
The Ctrl+Break adds to the idea I extracted from your previous post.
My guess is that VMWare virtual starts its COM pipe as a server
so it waits for someone to send in a request.
Stupid, as compared to VPC, and - if I’m right - directly contradicts
VM’s own on-line help.

This is exactly the behaviour I had with older WinDBGs.
I had all this with windbg 6.4, just got the latest tools with 6.5.3.7,
let’s see…

Regards,
Alex Shvedov

----- Original Message -----
From: “Spiro Trikaliotis”
To: “Kernel Debugging Interest List”
Sent: Saturday, July 23, 2005 4:04 AM
Subject: Re: [windbg] COM <-> named pipe debugging in vmware workstation

> Hello Alex,
>
> xxxxx@Home schrieb:
>
>> The VMWare-based target sits for some notable time, say 30 secs,
>> probably waiting for windbg response, the port state is reported by
>> VMWare as “connected” (to what ???), and then the boot continues as
>> usual, and windbg does not see anything.
>
> This is exactly the behaviour I had with older WinDBGs. With newer ones,
> the target just doesn’t sit 30seconds or so, but forever.
>
> Thus, my work-around is like this: Don’t start WinDBG until the target
> waits. Now, start WinDBG and press Ctrl+Break (this is important, the
> automatic break-in does NOT work!). After this, I have control over the
> target.
>
> Regards,
> Spiro.
>
> –
> Spiro R. Trikaliotis
> http://www.trikaliotis.net/
>
> —
> You are currently subscribed to windbg as: xxxxx@bellsouth.net
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>


You are currently subscribed to windbg as: xxxxx@kerneldrivers.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

> Just to let you know, I posted my command line Windbg, and it works every

time with VMWare. Not the case with VPC.
Mystery. I am happy with VPC 100%.
As I already said, I thought that all named pipes are created equal.
They are not, some named pipes are more named than others:-)

initial BP set and it
hits it each time on the VMWare virtual machine.
This is a secondary problem; in my case windbg hears
VPC all right in all cases, including target reboots, and
does not hear VMW at all.

I stop the stuff by a DbgBreakPoint in my DriverEntry
(in debug/checked build, of course).

Why on earth should it behave differently, I wonder?
It’s just a named pipe, nothing more, after all.
Mystery.

A pro po: starting windbg when wmv is waiting, as Spiro
suggested, does help, but wmv restart is missed by windbg
once again. VMWare does insists on being a server?

Regards,
Alex Shvedov

----- Original Message -----
From: “Peter Scott”
To: “Kernel Debugging Interest List”
Sent: Saturday, July 23, 2005 10:00 PM
Subject: RE: [windbg] COM <-> named pipe debugging in vmware workstation

>
> Just to let you know, I posted my command line Windbg, and it works every
> time with VMWare. Not the case with VPC. I have the initial BP set and it
> hits it each time on the VMWare virtual machine.
>
> OK, every 10-15 reboots of a VMWare virtual pc I need to restart Windbg, I
> call that pretty good.
>
> Pete
>
> Kernel Drivers
> Windows Filesystem and Device Driver Consulting
> www.KernelDrivers.com
> (303)546-0300
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@Home
> Sent: Saturday, July 23, 2005 8:32 AM
> To: Kernel Debugging Interest List
> Subject: Re: [windbg] COM <-> named pipe debugging in vmware workstation
>
>> Thus, my work-around is like this: Don’t start WinDBG until the target
>> waits. Now, start WinDBG and press Ctrl+Break (this is important, the
>> automatic break-in does NOT work!). After this, I have control over the
>> target.
> Thanks.
> The Ctrl+Break adds to the idea I extracted from your previous post.
> My guess is that VMWare virtual starts its COM pipe as a server
> so it waits for someone to send in a request.
> Stupid, as compared to VPC, and - if I’m right - directly contradicts
> VM’s own on-line help.
>
>> This is exactly the behaviour I had with older WinDBGs.
> I had all this with windbg 6.4, just got the latest tools with 6.5.3.7,
> let’s see…
>
> Regards,
> Alex Shvedov
>
>
> ----- Original Message -----
> From: “Spiro Trikaliotis”
> To: “Kernel Debugging Interest List”
> Sent: Saturday, July 23, 2005 4:04 AM
> Subject: Re: [windbg] COM <-> named pipe debugging in vmware workstation
>
>
>> Hello Alex,
>>
>> xxxxx@Home schrieb:
>>
>>> The VMWare-based target sits for some notable time, say 30 secs,
>>> probably waiting for windbg response, the port state is reported by
>>> VMWare as “connected” (to what ???), and then the boot continues as
>>> usual, and windbg does not see anything.
>>
>> This is exactly the behaviour I had with older WinDBGs. With newer ones,
>> the target just doesn’t sit 30seconds or so, but forever.
>>
>> Thus, my work-around is like this: Don’t start WinDBG until the target
>> waits. Now, start WinDBG and press Ctrl+Break (this is important, the
>> automatic break-in does NOT work!). After this, I have control over the
>> target.
>>
>> Regards,
>> Spiro.
>>
>> –
>> Spiro R. Trikaliotis
>> http://www.trikaliotis.net/
>>
>> —
>> You are currently subscribed to windbg as: xxxxx@bellsouth.net
>> To unsubscribe send a blank email to xxxxx@lists.osr.com
>>
>
>
> —
> You are currently subscribed to windbg as: xxxxx@kerneldrivers.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
>
>
> —
> You are currently subscribed to windbg as: xxxxx@bellsouth.net
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>

Hello Alex,

* On Sat, Jul 23, 2005 at 10:32:02AM -0400 xxxxx@Home wrote:

My guess is that VMWare virtual starts its COM pipe as a server so
it waits for someone to send in a request.
Stupid, as compared to VPC, and - if I’m right - directly contradicts
VM’s own on-line help.

Well, at least in my VMWare, I can tell VMWare if I want this side to be
a server or an application… Look into the settings of your serial
port.

Anyway, I am not sure how this would change anything?

* On Sat, Jul 23, 2005 at 11:45:08PM -0400 xxxxx@Home wrote:

I stop the stuff by a DbgBreakPoint in my DriverEntry (in
debug/checked build, of course).

You mean instead of an initial breakpoint? Yes, this works for me, too.

A pro po: starting windbg when wmv is waiting, as Spiro suggested,
does help, but wmv restart is missed by windbg once again. VMWare
does insists on being a server?

I don’t remeber for a VMWare restart, but I remember that going back to
a snapshot in VMWare, you get an error that the pipe could not be opened
(as it already is open).

Perhaps, this is somehow related?

Regards,
Spiro.


Spiro R. Trikaliotis
http://www.trikaliotis.net/

My experience is, after about 10-15 restarts, I need to re-start WinDbg, and
I’m not using virtual anything.


The personal opinion of
Gary G. Little

“Peter Scott” wrote in message
news:xxxxx@windbg…
>
> Just to let you know, I posted my command line Windbg, and it works every
> time with VMWare. Not the case with VPC. I have the initial BP set and it
> hits it each time on the VMWare virtual machine.
>
> OK, every 10-15 reboots of a VMWare virtual pc I need to restart Windbg, I
> call that pretty good.
>
> Pete
>
> Kernel Drivers
> Windows Filesystem and Device Driver Consulting
> www.KernelDrivers.com
> (303)546-0300
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@Home
> Sent: Saturday, July 23, 2005 8:32 AM
> To: Kernel Debugging Interest List
> Subject: Re: [windbg] COM <-> named pipe debugging in vmware workstation
>
>> Thus, my work-around is like this: Don’t start WinDBG until the target
>> waits. Now, start WinDBG and press Ctrl+Break (this is important, the
>> automatic break-in does NOT work!). After this, I have control over the
>> target.
> Thanks.
> The Ctrl+Break adds to the idea I extracted from your previous post.
> My guess is that VMWare virtual starts its COM pipe as a server
> so it waits for someone to send in a request.
> Stupid, as compared to VPC, and - if I’m right - directly contradicts
> VM’s own on-line help.
>
>> This is exactly the behaviour I had with older WinDBGs.
> I had all this with windbg 6.4, just got the latest tools with 6.5.3.7,
> let’s see…
>
> Regards,
> Alex Shvedov
>
>
> ----- Original Message -----
> From: “Spiro Trikaliotis”
> To: “Kernel Debugging Interest List”
> Sent: Saturday, July 23, 2005 4:04 AM
> Subject: Re: [windbg] COM <-> named pipe debugging in vmware workstation
>
>
>> Hello Alex,
>>
>> xxxxx@Home schrieb:
>>
>>> The VMWare-based target sits for some notable time, say 30 secs,
>>> probably waiting for windbg response, the port state is reported by
>>> VMWare as “connected” (to what ???), and then the boot continues as
>>> usual, and windbg does not see anything.
>>
>> This is exactly the behaviour I had with older WinDBGs. With newer ones,
>> the target just doesn’t sit 30seconds or so, but forever.
>>
>> Thus, my work-around is like this: Don’t start WinDBG until the target
>> waits. Now, start WinDBG and press Ctrl+Break (this is important, the
>> automatic break-in does NOT work!). After this, I have control over the
>> target.
>>
>> Regards,
>> Spiro.
>>
>> –
>> Spiro R. Trikaliotis
>> http://www.trikaliotis.net/
>>
>> —
>> You are currently subscribed to windbg as: xxxxx@bellsouth.net
>> To unsubscribe send a blank email to xxxxx@lists.osr.com
>>
>
>
> —
> You are currently subscribed to windbg as: xxxxx@kerneldrivers.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
>
>