EX: Resource 83E5B368 held in a position where all ...

Hi All,

With verifier enabled + chk’d kernel and HAL, I get
the following msg, which results in a break:

EX: Resource 83E5B368 held in a position where all
resources acquires should have been released
Break instruction exception - code 80000003 (first
chance)

Has anyone seen this and know how to go about tracking
that resource down?

Thanks in advance!

-Alex


Start your day with Yahoo! - make it your home page
http://www.yahoo.com/r/hs

Alex,

I’ve never seen this before but does

!locks -v 83E5B368

help at all?

/rod

“Alex Leung” wrote in message news:xxxxx@windbg…
> Hi All,
>
> With verifier enabled + chk’d kernel and HAL, I get
> the following msg, which results in a break:
>
> EX: Resource 83E5B368 held in a position where all
> resources acquires should have been released
> Break instruction exception - code 80000003 (first
> chance)
>
> Has anyone seen this and know how to go about tracking
> that resource down?
>
> Thanks in advance!
>
> -Alex

Hi Rod,

Thanks for your reply. Sorry it took so long, but I
finally reproduced it. Below is the windbg output
from the !locks command. Can you conclude anything
from this?

Thanks again,
Alex

EX: Resource 85593368 held in a position where all
resources acquires should have been released
Break instruction exception - code 80000003 (first
chance)
nt!DbgBreakPoint:
811507a8 cc int 3

0: kd> !locks -v 85593368

Resource @ 0x85593368 Exclusively owned
Threads: 8551c470-01<*>

THREAD 8551c470 Cid 0004.0444 Teb: 00000000
Win32Thread: 00000000 RUNNING on processor 0
Not impersonating
DeviceMap e1000418
Owning Process 86bc7bb0 Image:
System
Wait Start TickCount 274762230 Ticks: 1
(0:00:00:00.015)
Context Switch Count 1569 NoStackSwap
UserTime 00:00:00.0000
KernelTime 00:00:00.0421
Start Address nt!ExpWorkerThread (0x81165acc)
Stack Init b9a5f000 Current b9a5e69c Base
b9a5f000 Limit b9a5c000 Call 0
Priority 14 BasePriority 12 PriorityDecrement 0
ChildEBP RetAddr
b9a5ed60 8115fd51 nt!DbgBreakPoint (FPO: [0,0,0])
b9a5ed84 81165c09 nt!ExCheckIfResourceOwned+0x69
(FPO: [Non-Fpo])
b9a5edac 813b5d7c nt!ExpWorkerThread+0x13d (FPO:
[Non-Fpo])
b9a5eddc 81174ed6 nt!PspSystemThreadStartup+0x2e
(FPO: [Non-Fpo])
00000000 00000000 nt!KiThreadStartup+0x16

1 total locks, 1 locks currently held

— Rod Widdowson wrote:

> Alex,
>
> I’ve never seen this before but does
>
> !locks -v 83E5B368
>
> help at all?
>
> /rod
>
>
> “Alex Leung” wrote in message
> news:xxxxx@windbg…
> > Hi All,
> >
> > With verifier enabled + chk’d kernel and HAL, I
> get
> > the following msg, which results in a break:
> >
> > EX: Resource 83E5B368 held in a position where all
> > resources acquires should have been released
> > Break instruction exception - code 80000003 (first
> > chance)
> >
> > Has anyone seen this and know how to go about
> tracking
> > that resource down?
> >
> > Thanks in advance!
> >
> > -Alex
>
>
>
> —
> You are currently subscribed to windbg as:
> xxxxx@yahoo.com
> To unsubscribe send a blank email to
> xxxxx@lists.osr.com
>

____________________________________________________
Start your day with Yahoo! - make it your home page
http://www.yahoo.com/r/hs

You have a work routine that acquired the lock but failed to release it
upon exit; the work queue code has detected this situation and is
reporting it to you.

Figure out which driver/os component owns that lock. I’ve seen this
happen (for example) when exception handling triggers and the exception
handler doesn’t release a held lock.

To figure out the owning component try "!pool

" and/or "ln
" - the goal is to track this back to some owner. If this
shows up in the middle of a device object, dump the device object. If
this shows up as a separate pool tag, try to find the tag owner.

If you think this belongs to your own driver, try to figure out what
work routine was just executed (tough to do once it has been torn down,
unfortunately...)

These are always a challenge to find/fix because the bug is in the past
- you just tripped over the bug detector.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Alex Leung
Sent: Wednesday, August 10, 2005 12:31 PM
To: Kernel Debugging Interest List
Subject: Re:[windbg] Resource 83E5B368 held in a position where all ...

Hi Rod,

Thanks for your reply. Sorry it took so long, but I
finally reproduced it. Below is the windbg output
from the !locks command. Can you conclude anything
from this?

Thanks again,
Alex

EX: Resource 85593368 held in a position where all
resources acquires should have been released
Break instruction exception - code 80000003 (first
chance)
nt!DbgBreakPoint:
811507a8 cc int 3

0: kd> !locks -v 85593368

Resource @ 0x85593368 Exclusively owned
Threads: 8551c470-01<*>

THREAD 8551c470 Cid 0004.0444 Teb: 00000000
Win32Thread: 00000000 RUNNING on processor 0
Not impersonating
DeviceMap e1000418
Owning Process 86bc7bb0 Image:
System
Wait Start TickCount 274762230 Ticks: 1
(0:00:00:00.015)
Context Switch Count 1569 NoStackSwap
UserTime 00:00:00.0000
KernelTime 00:00:00.0421
Start Address nt!ExpWorkerThread (0x81165acc)
Stack Init b9a5f000 Current b9a5e69c Base
b9a5f000 Limit b9a5c000 Call 0
Priority 14 BasePriority 12 PriorityDecrement 0
ChildEBP RetAddr
b9a5ed60 8115fd51 nt!DbgBreakPoint (FPO: [0,0,0])
b9a5ed84 81165c09 nt!ExCheckIfResourceOwned+0x69
(FPO: [Non-Fpo])
b9a5edac 813b5d7c nt!ExpWorkerThread+0x13d (FPO:
[Non-Fpo])
b9a5eddc 81174ed6 nt!PspSystemThreadStartup+0x2e
(FPO: [Non-Fpo])
00000000 00000000 nt!KiThreadStartup+0x16

1 total locks, 1 locks currently held

--- Rod Widdowson wrote:

> Alex,
>
> I've never seen this before but does
>
> !locks -v 83E5B368
>
> help at all?
>
> /rod
>
>
> "Alex Leung" wrote in message
> news:xxxxx@windbg...
> > Hi All,
> >
> > With verifier enabled + chk'd kernel and HAL, I
> get
> > the following msg, which results in a break:
> >
> > EX: Resource 83E5B368 held in a position where all
> > resources acquires should have been released
> > Break instruction exception - code 80000003 (first
> > chance)
> >
> > Has anyone seen this and know how to go about
> tracking
> > that resource down?
> >
> > Thanks in advance!
> >
> > -Alex
>
>
>
> ---
> You are currently subscribed to windbg as:
> xxxxx@yahoo.com
> To unsubscribe send a blank email to
> xxxxx@lists.osr.com
>

____________________________________________________
Start your day with Yahoo! - make it your home page
http://www.yahoo.com/r/hs

---
You are currently subscribed to windbg as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

Tony,

Thanks for your reply; this is very helpful. If I had
known a reply would be that quick, I wouldn’t have
rebooted and lost that state :frowning:

I have a couple of follow-up questions:

To figure out the owning component try "!pool

" and/or "ln "

Is this only valid if that object is in system pool?
If so, are locks usually in allocated in pool?

So if I do a !pool on a location and I get:
...
860e0c90 size: (Allocated) Devi (Protected)
*860e0da8 size: (Allocated) *Devi (Protected)
Pooltag Devi : Device objects

But it does not give me an indication of what Device
Object it is. So I do a "dd 860e0da8" and then start
doing !devobj on all the locations that might be
device objects to look for the culprit. Is this the
right approach?

Also, if I simply want to make sure it's not my lock,
is it sufficient just to get the virtual address of
all my locks and make sure it's not the one that
verifier (or the chk'd build) is complaining about?
Since I only have one spin lock (located in my
DevExt), can I just look into all of my DevExtensions
and check that address against the problemary one?

Thanks again!
Alex

--- Tony Mason wrote:

> You have a work routine that acquired the lock but
> failed to release it
> upon exit; the work queue code has detected this
> situation and is
> reporting it to you.
>
> Figure out which driver/os component owns that lock.
> I've seen this
> happen (for example) when exception handling
> triggers and the exception
> handler doesn't release a held lock.
>
> To figure out the owning component try "!pool
> " and/or "ln
> " - the goal is to track this back to some
> owner. If this
> shows up in the middle of a device object, dump the
> device object. If
> this shows up as a separate pool tag, try to find
> the tag owner.
>
> If you think this belongs to your own driver, try to
> figure out what
> work routine was just executed (tough to do once it
> has been torn down,
> unfortunately...)
>
> These are always a challenge to find/fix because the
> bug is in the past
> - you just tripped over the bug detector.
>
> Regards,
>
> Tony
>
> Tony Mason
> Consulting Partner
> OSR Open Systems Resources, Inc.
> http://www.osr.com
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of
> Alex Leung
> Sent: Wednesday, August 10, 2005 12:31 PM
> To: Kernel Debugging Interest List
> Subject: Re:[windbg] Resource 83E5B368 held in a
> position where all ...
>
> Hi Rod,
>
> Thanks for your reply. Sorry it took so long, but I
> finally reproduced it. Below is the windbg output
> from the !locks command. Can you conclude anything
> from this?
>
> Thanks again,
> Alex
>
> EX: Resource 85593368 held in a position where all
> resources acquires should have been released
> Break instruction exception - code 80000003 (first
> chance)
> nt!DbgBreakPoint:
> 811507a8 cc int 3
>
> 0: kd> !locks -v 85593368
>
> Resource @ 0x85593368 Exclusively owned
> Threads: 8551c470-01<*>
>
> THREAD 8551c470 Cid 0004.0444 Teb: 00000000
> Win32Thread: 00000000 RUNNING on processor 0
> Not impersonating
> DeviceMap e1000418
> Owning Process 86bc7bb0 Image:
>
> System
> Wait Start TickCount 274762230 Ticks:
> 1
> (0:00:00:00.015)
> Context Switch Count 1569 NoStackSwap
> UserTime 00:00:00.0000
> KernelTime 00:00:00.0421
> Start Address nt!ExpWorkerThread (0x81165acc)
> Stack Init b9a5f000 Current b9a5e69c Base
> b9a5f000 Limit b9a5c000 Call 0
> Priority 14 BasePriority 12 PriorityDecrement 0
> ChildEBP RetAddr
> b9a5ed60 8115fd51 nt!DbgBreakPoint (FPO:
> [0,0,0])
> b9a5ed84 81165c09
> nt!ExCheckIfResourceOwned+0x69
> (FPO: [Non-Fpo])
> b9a5edac 813b5d7c nt!ExpWorkerThread+0x13d
> (FPO:
> [Non-Fpo])
> b9a5eddc 81174ed6
> nt!PspSystemThreadStartup+0x2e
> (FPO: [Non-Fpo])
> 00000000 00000000 nt!KiThreadStartup+0x16
>
>
> 1 total locks, 1 locks currently held
>
> --- Rod Widdowson wrote:
>
> > Alex,
> >
> > I've never seen this before but does
> >
> > !locks -v 83E5B368
> >
> > help at all?
> >
> > /rod
> >
> >
> > "Alex Leung" wrote in
> message
> > news:xxxxx@windbg...
> > > Hi All,
> > >
> > > With verifier enabled + chk'd kernel and HAL, I
> > get
> > > the following msg, which results in a break:
> > >
> > > EX: Resource 83E5B368 held in a position where
> all
> > > resources acquires should have been released
> > > Break instruction exception - code 80000003
> (first
> > > chance)
> > >
> > > Has anyone seen this and know how to go about
> > tracking
> > > that resource down?
> > >
> > > Thanks in advance!
> > >
> > > -Alex
> >
> >
> >
> > ---
> > You are currently subscribed to windbg as:
> > xxxxx@yahoo.com
> > To unsubscribe send a blank email to
> > xxxxx@lists.osr.com
> >
>
>
>
>
> ____________________________________________________
> Start your day with Yahoo! - make it your home page
> http://www.yahoo.com/r/hs
>
>
> ---
> You are currently subscribed to windbg as:
> xxxxx@osr.com
> To unsubscribe send a blank email to
> xxxxx@lists.osr.com
>
> ---
> You are currently subscribed to windbg as: unknown
> lmsubst tag argument: ''
> To unsubscribe send a blank email to
> xxxxx@lists.osr.com
>

____________________________________________________
Start your day with Yahoo! - make it your home page
http://www.yahoo.com/r/hs

Alex,

This is exactly the sort of case I like to use to torture my students in
debug class…

There’s nothing that requires the ERESOURCE be allocated from pool. It
could be static within the driver as well, but it exists somewhere - no
guarantees you can find the owner, but the odds are in your favor,
especially if you have symbols for the driver.

Pool memory starts on some 8 byte boundary (in some versions it is 16
bytes, but that’s an 8 byte boundary anyway…). For small allocations
(less than a page,) the first four bytes are control bytes, the 2nd 4
bytes are the tag. A *device object* consists of an object header (try
“dt nt!_OBJECT_HEADER”) and a device object plus the extension - and it
is the extension where the ERESOURCE would be contained anyway.

So taking that address within the pool block, you can compute where the
device object would start:

+ (8)> +

You can just type these values after the "?" operator and get the
results.

Feed those results into either "dt nt!_DEVICE_OBJECT "
or "!devobj ". The latter approach will validate that
you have a device object, the former will dump whatever is at that
memory location, whether or not it is what you said it was.

Assuming "!devobj" works, you have the name of the device object.

If you only have spin locks it isn't yours - this was an ERESOURCE lock.
But just because it isn't your lock, doesn't mean it isn't your bug. My
guess is you are causing an exception and the exception handler in a
driver above you is catching but not freeing the ERESOURCE correctly -
I've seen this happen many times in the past, and finding it this way is
MUCH easier than debugging a deadlock 15 minutes later...

Debugging is often the process of making educated guesses about what
caused a particular problem and using the tools to see if your guesses
are right.

Good luck!

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Alex Leung
Sent: Wednesday, August 10, 2005 3:53 PM
To: Kernel Debugging Interest List
Subject: RE: [windbg] Resource 83E5B368 held in a position where all ...

Tony,

Thanks for your reply; this is very helpful. If I had
known a reply would be that quick, I wouldn't have
rebooted and lost that state :(

I have a couple of follow-up questions:

> To figure out the owning component try "!pool
> " and/or "ln "

Is this only valid if that object is in system pool?
If so, are locks usually in allocated in pool?

So if I do a !pool on a location and I get:
...
860e0c90 size: (Allocated) Devi (Protected)
*860e0da8 size: (Allocated) *Devi (Protected)
Pooltag Devi : Device objects

But it does not give me an indication of what Device
Object it is. So I do a "dd 860e0da8" and then start
doing !devobj on all the locations that might be
device objects to look for the culprit. Is this the
right approach?

Also, if I simply want to make sure it's not my lock,
is it sufficient just to get the virtual address of
all my locks and make sure it's not the one that
verifier (or the chk'd build) is complaining about?
Since I only have one spin lock (located in my
DevExt), can I just look into all of my DevExtensions
and check that address against the problemary one?

Thanks again!
Alex

--- Tony Mason wrote:

> You have a work routine that acquired the lock but
> failed to release it
> upon exit; the work queue code has detected this
> situation and is
> reporting it to you.
>
> Figure out which driver/os component owns that lock.
> I've seen this
> happen (for example) when exception handling
> triggers and the exception
> handler doesn't release a held lock.
>
> To figure out the owning component try "!pool
> " and/or "ln
> " - the goal is to track this back to some
> owner. If this
> shows up in the middle of a device object, dump the
> device object. If
> this shows up as a separate pool tag, try to find
> the tag owner.
>
> If you think this belongs to your own driver, try to
> figure out what
> work routine was just executed (tough to do once it
> has been torn down,
> unfortunately...)
>
> These are always a challenge to find/fix because the
> bug is in the past
> - you just tripped over the bug detector.
>
> Regards,
>
> Tony
>
> Tony Mason
> Consulting Partner
> OSR Open Systems Resources, Inc.
> http://www.osr.com
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of
> Alex Leung
> Sent: Wednesday, August 10, 2005 12:31 PM
> To: Kernel Debugging Interest List
> Subject: Re:[windbg] Resource 83E5B368 held in a
> position where all ...
>
> Hi Rod,
>
> Thanks for your reply. Sorry it took so long, but I
> finally reproduced it. Below is the windbg output
> from the !locks command. Can you conclude anything
> from this?
>
> Thanks again,
> Alex
>
> EX: Resource 85593368 held in a position where all
> resources acquires should have been released
> Break instruction exception - code 80000003 (first
> chance)
> nt!DbgBreakPoint:
> 811507a8 cc int 3
>
> 0: kd> !locks -v 85593368
>
> Resource @ 0x85593368 Exclusively owned
> Threads: 8551c470-01<*>
>
> THREAD 8551c470 Cid 0004.0444 Teb: 00000000
> Win32Thread: 00000000 RUNNING on processor 0
> Not impersonating
> DeviceMap e1000418
> Owning Process 86bc7bb0 Image:
>
> System
> Wait Start TickCount 274762230 Ticks:
> 1
> (0:00:00:00.015)
> Context Switch Count 1569 NoStackSwap
> UserTime 00:00:00.0000
> KernelTime 00:00:00.0421
> Start Address nt!ExpWorkerThread (0x81165acc)
> Stack Init b9a5f000 Current b9a5e69c Base
> b9a5f000 Limit b9a5c000 Call 0
> Priority 14 BasePriority 12 PriorityDecrement 0
> ChildEBP RetAddr
> b9a5ed60 8115fd51 nt!DbgBreakPoint (FPO:
> [0,0,0])
> b9a5ed84 81165c09
> nt!ExCheckIfResourceOwned+0x69
> (FPO: [Non-Fpo])
> b9a5edac 813b5d7c nt!ExpWorkerThread+0x13d
> (FPO:
> [Non-Fpo])
> b9a5eddc 81174ed6
> nt!PspSystemThreadStartup+0x2e
> (FPO: [Non-Fpo])
> 00000000 00000000 nt!KiThreadStartup+0x16
>
>
> 1 total locks, 1 locks currently held
>
> --- Rod Widdowson wrote:
>
> > Alex,
> >
> > I've never seen this before but does
> >
> > !locks -v 83E5B368
> >
> > help at all?
> >
> > /rod
> >
> >
> > "Alex Leung" wrote in
> message
> > news:xxxxx@windbg...
> > > Hi All,
> > >
> > > With verifier enabled + chk'd kernel and HAL, I
> > get
> > > the following msg, which results in a break:
> > >
> > > EX: Resource 83E5B368 held in a position where
> all
> > > resources acquires should have been released
> > > Break instruction exception - code 80000003
> (first
> > > chance)
> > >
> > > Has anyone seen this and know how to go about
> > tracking
> > > that resource down?
> > >
> > > Thanks in advance!
> > >
> > > -Alex
> >
> >
> >
> > ---
> > You are currently subscribed to windbg as:
> > xxxxx@yahoo.com
> > To unsubscribe send a blank email to
> > xxxxx@lists.osr.com
> >
>
>
>
>
> ____________________________________________________
> Start your day with Yahoo! - make it your home page
> http://www.yahoo.com/r/hs
>
>
> ---
> You are currently subscribed to windbg as:
> xxxxx@osr.com
> To unsubscribe send a blank email to
> xxxxx@lists.osr.com
>
> ---
> You are currently subscribed to windbg as: unknown
> lmsubst tag argument: ''
> To unsubscribe send a blank email to
> xxxxx@lists.osr.com
>

____________________________________________________
Start your day with Yahoo! - make it your home page
http://www.yahoo.com/r/hs

---
You are currently subscribed to windbg as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com