KernelMode debugging of UserModeService

WINDBG
1

This might have been addressed in some place, but I’ve not seen anything on
this yet, and I need help on this.

There is a NT service that spawns several process, and some of those
processes also spawns some other processes ( recursively to 3 levels). They
communicate using IPC mechanism(s), and time sensitive. Most of the timing
parameters are either unknown or based on some standards that are mostly
telecom related so changing them would not be wise / easy / or prudent for
my debug scenario.

So essentially, I need to be able to bring the target using boot.ini ( L.E.
XP versions), and break-in and set BPs ( Worse case is to have hard coded
bps ). I’ve few questions -

  1. I’m using the latest build of windbg 32bit binary ( I think 6.8…), and
    found that even if I put bp ( after hard coded break first ), they seem to
    take it then I don’t break in those places. I had this situation before on
    earlier versions. So the case is that bl shows that the break-points are
    active, but never breaks, debug statement right before or after those lines
    prints debug output

  2. If I don’t use hard coded bp to start with, attaching ( using dot cmd
    for attach as well as dot cmd process), does not seem to get me to the
    process context. This would be ideal for putting dynamic break points, but (
    lm t m myserv* ) does not even show the module being loaded, and even if I
    try to do bu, it never fires.

After lookin at much better book from Microsoft about Advanced Debugging, I
noticed the using gflags or debug event filter might help in this situation
since for any user debug event, it looks for user debugger, then if the
events are enabled then looks for kernel debugger.

Not knowing enough, if I turn on the stop on exception ( was in a hurry
wasn’t I :), almost all first chance exceptions and others gets me to
debugger break, and it slows down the whole thing so much that I never be
able to successfully load the root services that spawns other services…

So my question(s) –

  1. What are the minimal flags I need to set so that I can see those bps get
    fired.

  2. Any particular place to look for those flags description, as well as any
    interrelations between gflags and those Event Filters.

THanks
-pro

2

Not having looked at your stuff in detail, two questions:

  1. Have you tried inserting int3s in your processes ?
  2. Have you tried using the i386 debug registers ?

Both of these might need you write your own ISR, but that’s peanuts.

Alberto.

----- Original Message -----
From: Prokash Sinha
To: Kernel Debugging Interest List
Sent: Tuesday, January 08, 2008 10:31 PM
Subject: [windbg] KernelMode debugging of UserModeService

This might have been addressed in some place, but I’ve not seen anything on this yet, and I need help on this.

There is a NT service that spawns several process, and some of those processes also spawns some other processes ( recursively to 3 levels). They communicate using IPC mechanism(s), and time sensitive. Most of the timing parameters are either unknown or based on some standards that are mostly telecom related so changing them would not be wise / easy / or prudent for my debug scenario.

So essentially, I need to be able to bring the target using boot.ini ( L.E. XP versions), and break-in and set BPs ( Worse case is to have hard coded bps ). I’ve few questions -

  1. I’m using the latest build of windbg 32bit binary ( I think 6.8…), and found that even if I put bp ( after hard coded break first ), they seem to take it then I don’t break in those places. I had this situation before on earlier versions. So the case is that bl shows that the break-points are active, but never breaks, debug statement right before or after those lines prints debug output

  2. If I don’t use hard coded bp to start with, attaching ( using dot cmd for attach as well as dot cmd process), does not seem to get me to the process context. This would be ideal for putting dynamic break points, but ( lm t m myserv* ) does not even show the module being loaded, and even if I try to do bu, it never fires.

After lookin at much better book from Microsoft about Advanced Debugging, I noticed the using gflags or debug event filter might help in this situation since for any user debug event, it looks for user debugger, then if the events are enabled then looks for kernel debugger.

Not knowing enough, if I turn on the stop on exception ( was in a hurry wasn’t I :), almost all first chance exceptions and others gets me to debugger break, and it slows down the whole thing so much that I never be able to successfully load the root services that spawns other services…

So my question(s) –

  1. What are the minimal flags I need to set so that I can see those bps get fired.

  2. Any particular place to look for those flags description, as well as any interrelations between gflags and those Event Filters.

THanks
-pro
— You are currently subscribed to windbg as: xxxxx@ieee.org To unsubscribe send a blank email to xxxxx@lists.osr.com

3

Only way I’m successful is to insert hard-coded break point ( either thru
DebugBreak() and friends or if I want to build the release build with enough
of pdb infos then __asm int 3; ). And this is all from kernel mode. *My main
objective is to freeze the systems timers so the IPC communication issues
does not come in my way. This a open telecommunication framework on top of
ACE framework. Since you do multiplatform work, look at the ACE, (or perhaps
you know it ) you might like it :-).

For kmode component, windbg is good for me ( don’t need to use KD).

I was more interested in debugging services, by switching to those context
from kernelmode, then load the symbols and put bps. I can put bps ( visibly
they are enabled), but never hit those bps or sometime it is very rarely.
There are something with codpage sharing and all those internals related to
debugger, but I try not to digress in that direction.

-pro

On Jan 9, 2008 5:50 AM, Alberto Moreira wrote:

> Not having looked at your stuff in detail, two questions:
>
> 1. Have you tried inserting int3s in your processes ?
> 2. Have you tried using the i386 debug registers ?
>
> Both of these might need you write your own ISR, but that’s peanuts.
>
>
> Alberto.
>
>
>
> ----- Original Message -----
> From: Prokash Sinha
> To: Kernel Debugging Interest List
> Sent: Tuesday, January 08, 2008 10:31 PM
> Subject: [windbg] KernelMode debugging of UserModeService
>
> This might have been addressed in some place, but I’ve not seen anything
> on this yet, and I need help on this.
>
> There is a NT service that spawns several process, and some of those
> processes also spawns some other processes ( recursively to 3 levels). They
> communicate using IPC mechanism(s), and time sensitive. Most of the timing
> parameters are either unknown or based on some standards that are mostly
> telecom related so changing them would not be wise / easy / or prudent for
> my debug scenario.
>
> So essentially, I need to be able to bring the target using boot.ini ( L.E.
> XP versions), and break-in and set BPs ( Worse case is to have hard coded
> bps ). I’ve few questions -
>
> 1) I’m using the latest build of windbg 32bit binary ( I think 6.8…),
> and found that even if I put bp ( after hard coded break first ), they seem
> to take it then I don’t break in those places. I had this situation before
> on earlier versions. So the case is that bl shows that the break-points are
> active, but never breaks, debug statement right before or after those lines
> prints debug output
>
> 2) If I don’t use hard coded bp to start with, attaching ( using dot cmd
> for attach as well as dot cmd process), does not seem to get me to the
> process context. This would be ideal for putting dynamic break points, but (
> lm t m myserv* ) does not even show the module being loaded, and even if I
> try to do bu, it never fires.
>
>
> After lookin at much better book from Microsoft about Advanced Debugging,
> I noticed the using gflags or debug event filter might help in this
> situation since for any user debug event, it looks for user debugger, then
> if the events are enabled then looks for kernel debugger.
>
> Not knowing enough, if I turn on the stop on exception ( was in a hurry
> wasn’t I :), almost all first chance exceptions and others gets me to
> debugger break, and it slows down the whole thing so much that I never be
> able to successfully load the root services that spawns other services…
>
> So my question(s) –
>
> 1) What are the minimal flags I need to set so that I can see those bps
> get fired.
>
> 2) Any particular place to look for those flags description, as well as
> any interrelations between gflags and those Event Filters.
>
> THanks
> -pro
> — You are currently subscribed to windbg as: xxxxx@ieee.org To
> unsubscribe send a blank email to xxxxx@lists.osr.com
>
>
> —
> You are currently subscribed to windbg as: unknown lmsubst tag argument:
> ‘’
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>

4

Pro,

I’ve had good luck with doing this with a hardcoded (typically
registry controlled) breakpoint at the start of the service. Once you can
break in with the context of the service everything seems to work, but
trying to put in the initial breakpoint by WinDBG is problematical.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr
Remove StopSpam to reply

“Prokash Sinha” wrote in message news:xxxxx@windbg…
> Only way I’m successful is to insert hard-coded break point ( either thru
> DebugBreak() and friends or if I want to build the release build with
> enough
> of pdb infos then __asm int 3; ). And this is all from kernel mode. My
> main
> objective is to freeze the systems timers so the IPC communication issues
> does not come in my way. This a open telecommunication framework on top of
> ACE framework. Since you do multiplatform work, look at the ACE, (or
> perhaps
> you know it ) you might like it :-).
>
> For kmode component, windbg is good for me ( don’t need to use KD).
>
> I was more interested in debugging services, by switching to those context
> from kernelmode, then load the symbols and put bps. I can put bps (
> visibly
> they are enabled), but never hit those bps or sometime it is very rarely.
> There are something with codpage sharing and all those internals related
> to
> debugger, but I try not to digress in that direction.
>
> -pro
>
> On Jan 9, 2008 5:50 AM, Alberto Moreira wrote:
>
>> Not having looked at your stuff in detail, two questions:
>>
>> 1. Have you tried inserting int3s in your processes ?
>> 2. Have you tried using the i386 debug registers ?
>>
>> Both of these might need you write your own ISR, but that’s peanuts.
>>
>>
>> Alberto.
>>
>>
>>
>> ----- Original Message -----
>> From: Prokash Sinha
>> To: Kernel Debugging Interest List
>> Sent: Tuesday, January 08, 2008 10:31 PM
>> Subject: [windbg] KernelMode debugging of UserModeService
>>
>> This might have been addressed in some place, but I’ve not seen anything
>> on this yet, and I need help on this.
>>
>> There is a NT service that spawns several process, and some of those
>> processes also spawns some other processes ( recursively to 3 levels).
>> They
>> communicate using IPC mechanism(s), and time sensitive. Most of the
>> timing
>> parameters are either unknown or based on some standards that are mostly
>> telecom related so changing them would not be wise / easy / or prudent
>> for
>> my debug scenario.
>>
>> So essentially, I need to be able to bring the target using boot.ini (
>> L.E.
>> XP versions), and break-in and set BPs ( Worse case is to have hard coded
>> bps ). I’ve few questions -
>>
>> 1) I’m using the latest build of windbg 32bit binary ( I think 6.8…),
>> and found that even if I put bp ( after hard coded break first ), they
>> seem
>> to take it then I don’t break in those places. I had this situation
>> before
>> on earlier versions. So the case is that bl shows that the break-points
>> are
>> active, but never breaks, debug statement right before or after those
>> lines
>> prints debug output
>>
>> 2) If I don’t use hard coded bp to start with, attaching ( using dot cmd
>> for attach as well as dot cmd process), does not seem to get me to the
>> process context. This would be ideal for putting dynamic break points,
>> but (
>> lm t m myserv ) does not even show the module being loaded, and even if
>> I
>> try to do bu, it never fires.
>>
>>
>> After lookin at much better book from Microsoft about Advanced Debugging,
>> I noticed the using gflags or debug event filter might help in this
>> situation since for any user debug event, it looks for user debugger,
>> then
>> if the events are enabled then looks for kernel debugger.
>>
>> Not knowing enough, if I turn on the stop on exception ( was in a hurry
>> wasn’t I :), almost all first chance exceptions and others gets me to
>> debugger break, and it slows down the whole thing so much that I never be
>> able to successfully load the root services that spawns other services…
>>
>> So my question(s) –
>>
>> 1) What are the minimal flags I need to set so that I can see those bps
>> get fired.
>>
>> 2) Any particular place to look for those flags description, as well as
>> any interrelations between gflags and those Event Filters.
>>
>> THanks
>> -pro
>> — You are currently subscribed to windbg as: xxxxx@ieee.org To
>> unsubscribe send a blank email to xxxxx@lists.osr.com
>>
>>
>> —
>> You are currently subscribed to windbg as: unknown lmsubst tag argument:
>> ‘’
>> To unsubscribe send a blank email to xxxxx@lists.osr.com
>>
>

5

Thanks Don, and Alberto:

This seems to be the case for me too. Probing registry at the start is a
good idea.

My problem is that communication channels are abstract to carry ( DTMF, SIP,
and other protocols including media streaming), and when in a stress
situation there are channel timeouts happens, these are really not a true
sense timeouts, and retries could enuse. After variable number of resource
unavailablity, things starts to happen in the sense that other processes
starts throwing their hands up …

So in the above context, I was setting break-points ( dynamically ) to see
if it hits those. Okay they are being fired, I get the control. Now I
disable those bps. Go ahead with let these services run. Then I comeback (
breaking and again inserting bps from windbg, specially those error paths
and I never hit them, now I turned back all the previous bps I never hit
them and I know those paths are being executed … I’ve to put hard-coded
0xCC on the error paths too for now…

I wish this gets taken care, this would be so useful.

-pro

On Jan 9, 2008 12:06 PM, Don Burn wrote:

> Pro,
>
> I’ve had good luck with doing this with a hardcoded (typically
> registry controlled) breakpoint at the start of the service. Once you can
> break in with the context of the service everything seems to work, but
> trying to put in the initial breakpoint by WinDBG is problematical.
>
>
> –
> Don Burn (MVP, Windows DDK)
> Windows 2k/XP/2k3 Filesystem and Driver Consulting
> Website: http://www.windrvr.com
> Blog: http://msmvps.com/blogs/WinDrvr
> Remove StopSpam to reply
>
>
> “Prokash Sinha” wrote in message news:xxxxx@windbg…
> > Only way I’m successful is to insert hard-coded break point ( either
> thru
> > DebugBreak() and friends or if I want to build the release build with
> > enough
> > of pdb infos then __asm int 3; ). And this is all from kernel mode. My
> > main
> > objective is to freeze the systems timers so the IPC communication
> issues
> > does not come in my way. This a open telecommunication framework on top
> of
> > ACE framework. Since you do multiplatform work, look at the ACE, (or
> > perhaps
> > you know it ) you might like it :-).
> >
> > For kmode component, windbg is good for me ( don’t need to use KD).
> >
> > I was more interested in debugging services, by switching to those
> context
> > from kernelmode, then load the symbols and put bps. I can put bps (
> > visibly
> > they are enabled), but never hit those bps or sometime it is very
> rarely.
> > There are something with codpage sharing and all those internals related
> > to
> > debugger, but I try not to digress in that direction.
> >
> > -pro
> >
> > On Jan 9, 2008 5:50 AM, Alberto Moreira wrote:
> >
> >> Not having looked at your stuff in detail, two questions:
> >>
> >> 1. Have you tried inserting int3s in your processes ?
> >> 2. Have you tried using the i386 debug registers ?
> >>
> >> Both of these might need you write your own ISR, but that’s peanuts.
> >>
> >>
> >> Alberto.
> >>
> >>
> >>
> >> ----- Original Message -----
> >> From: Prokash Sinha
> >> To: Kernel Debugging Interest List
> >> Sent: Tuesday, January 08, 2008 10:31 PM
> >> Subject: [windbg] KernelMode debugging of UserModeService
> >>
> >> This might have been addressed in some place, but I’ve not seen
> anything
> >> on this yet, and I need help on this.
> >>
> >> There is a NT service that spawns several process, and some of those
> >> processes also spawns some other processes ( recursively to 3 levels).
> >> They
> >> communicate using IPC mechanism(s), and time sensitive. Most of the
> >> timing
> >> parameters are either unknown or based on some standards that are
> mostly
> >> telecom related so changing them would not be wise / easy / or prudent
> >> for
> >> my debug scenario.
> >>
> >> So essentially, I need to be able to bring the target using boot.ini (
> >> L.E.
> >> XP versions), and break-in and set BPs ( Worse case is to have hard
> coded
> >> bps ). I’ve few questions -
> >>
> >> 1) I’m using the latest build of windbg 32bit binary ( I think 6.8…
> .),
> >> and found that even if I put bp ( after hard coded break first ), they
> >> seem
> >> to take it then I don’t break in those places. I had this situation
> >> before
> >> on earlier versions. So the case is that bl shows that the break-points
> >> are
> >> active, but never breaks, debug statement right before or after those
> >> lines
> >> prints debug output
> >>
> >> 2) If I don’t use hard coded bp to start with, attaching ( using dot
> cmd
> >> for attach as well as dot cmd process), does not seem to get me to the
> >> process context. This would be ideal for putting dynamic break points,
> >> but (
> >> lm t m myserv ) does not even show the module being loaded, and even
> if
> >> I
> >> try to do bu, it never fires.
> >>
> >>
> >> After lookin at much better book from Microsoft about Advanced
> Debugging,
> >> I noticed the using gflags or debug event filter might help in this
> >> situation since for any user debug event, it looks for user debugger,
> >> then
> >> if the events are enabled then looks for kernel debugger.
> >>
> >> Not knowing enough, if I turn on the stop on exception ( was in a hurry
> >> wasn’t I :), almost all first chance exceptions and others gets me to
> >> debugger break, and it slows down the whole thing so much that I never
> be
> >> able to successfully load the root services that spawns other
> services…
> >>
> >> So my question(s) –
> >>
> >> 1) What are the minimal flags I need to set so that I can see those bps
> >> get fired.
> >>
> >> 2) Any particular place to look for those flags description, as well as
> >> any interrelations between gflags and those Event Filters.
> >>
> >> THanks
> >> -pro
> >> — You are currently subscribed to windbg as: xxxxx@ieee.org To
> >> unsubscribe send a blank email to xxxxx@lists.osr.com
> >>
> >>
> >> —
> >> You are currently subscribed to windbg as: unknown lmsubst tag
> argument:
> >> ‘’
> >> To unsubscribe send a blank email to xxxxx@lists.osr.com
> >>
> >
>
>
>
> —
> You are currently subscribed to windbg as: xxxxx@gmail.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>

6

> 1) I’m using the latest build of windbg 32bit binary ( I think 6.8…),

and
found that even if I put bp ( after hard coded break first ), they seem to
take it then I don’t break in those places. I had this situation before on
earlier versions. So the case is that bl shows that the break-points are
active, but never breaks, debug statement right before or after those
lines
prints debug output

So my question(s) –

  1. What are the minimal flags I need to set so that I can see those bps
    get
    fired.

Setting user mode breakpoints from kernel debugger is tricky. The
main rule to remember is that *the target system* has to be in the right
process context when you are setting the breakpoint. So if you
Ctrl+C then do .process /p to switch process context, the breakpoint
will appear to set correctly, but may not actually work because
.process /p only changes debugger’s context.

What you need to do instead is use .process /i then continue execution
so that when the target breaks it will be in the right context. Then you
can set breakpoints in user code. There are still various potential
problems here, but most of the time it should work.

In many cases it is easier to debug such problems with a user mode
debugger instead. You can pipe ntsd to kd (ntsd -d) to make sure
the entire system is frozen while you’re looking at something in ntsd.
Another useful option is -o (or .childdbg 1) which automatically
attaches to any child processes.

–
This posting is provided “AS IS” with no warranties, and confers no
rights.

7

Thanks for the tips Pavel.

I will try. It is a must that I freeze the system.

-pro

On Jan 12, 2008 12:54 AM, Pavel Lebedinsky wrote:

> > 1) I’m using the latest build of windbg 32bit binary ( I think 6.8…),
> > and
> > found that even if I put bp ( after hard coded break first ), they seem
> to
> > take it then I don’t break in those places. I had this situation before
> on
> > earlier versions. So the case is that bl shows that the break-points are
> > active, but never breaks, debug statement right before or after those
> > lines
> > prints debug output
>
> > So my question(s) –
> >
> > 1) What are the minimal flags I need to set so that I can see those bps
> > get
> > fired.
>
> Setting user mode breakpoints from kernel debugger is tricky. The
> main rule to remember is that the target system has to be in the right
> process context when you are setting the breakpoint. So if you
> Ctrl+C then do .process /p to switch process context, the breakpoint
> will appear to set correctly, but may not actually work because
> .process /p only changes debugger’s context.
>
> What you need to do instead is use .process /i then continue execution
> so that when the target breaks it will be in the right context. Then you
> can set breakpoints in user code. There are still various potential
> problems here, but most of the time it should work.
>
> In many cases it is easier to debug such problems with a user mode
> debugger instead. You can pipe ntsd to kd (ntsd -d) to make sure
> the entire system is frozen while you’re looking at something in ntsd.
> Another useful option is -o (or .childdbg 1) which automatically
> attaches to any child processes.
>
> –
> This posting is provided “AS IS” with no warranties, and confers no
> rights.
>
>
>
> —
> You are currently subscribed to windbg as: xxxxx@gmail.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>