Recursive shared ERESOURCE acquisition

NTFSD
1

Our filter driver seems to have hit a deadlock with contention for
ownership of an ERESOURCE lock. At the time of the hang, we break into the
system and see a particular ERESOURCE lock with 1 shared owner, 1 exclusive
waiter, and 10 shared waiters. The shared waiters are all waiting a few
levels deep in NtfsAcquireSharedVcb(), all waiting to do something on the
same volume. The exclusive waiter is waiting a few levels deep in
NtfsAcquireExclusiveVcb(). So far, not so bad.

But the shared owner of the lock seems to be waiting to acquire the SAME
LOCK again, shared. Here’s the stack of the owner (apologies for
wrapping):

ChildEBP RetAddr Args to Child
b99614b4 8042bd4d 00000000 813fb504 ffa4e350 nt!KiSwapThread+0x1b1
b99614dc 804152f2 813e4328 00000000 00000000
nt!KeWaitForSingleObject+0x1a3
b996151c 80414886 811207f8 b99617e8 00000000 nt!ExpWaitForResource+0x1b4
b9961534 bfe9c674 813fb500 00000001 b99617e8
nt!ExAcquireResourceSharedLite+0xc6
b9961544 bfe9c1aa b99617e8 813fb100 00000001
ntfs!NtfsAcquireSharedVcb+0x1b
b99617a0 bfe9b971 b99617e8 811862c8 813fb030 ntfs!NtfsCommonCleanup+0x134
b9961918 8041d915 813fb030 811862c8 811864a0 ntfs!NtfsFsdCleanup+0x113
b996192c b9c56874 811864a0 811864c4 811862d8 nt!IopfCallDriver+0x35
b9961968 b9c58e1e 811b73e8 811862c8 00000000 rxfsfltr!CallAndRelease+0x176
b99619d8 8041d915 811b7330 811862c8 811207f8 rxfsfltr!FilterDispatch+0x449
b99619ec 804c1e67 80064ffc 820c5b40 00000001 nt!IopfCallDriver+0x35
b9961a20 804d840b 820ad030 811b7330 01020000 nt!IopCloseFile+0x267
b9961a4c 8044f37a 820ad030 811207e4 811207f8
nt!ObpDecrementHandleCount+0x13d
b9961b00 80465679 000002e0 e3155528 00000000 nt!NtClose+0x1f0
b9961b00 8042f917 000002e0 e3155528 00000000 nt!KiSystemService+0xc9
b9961b7c b99a42a1 000002e0 000002e0 00000000 nt!ZwClose+0xb

Anyway, you can see he’s waiting a few levels deep in
NtfsAcquireSharedVcb(), actually waiting for the KEVENT object at 813e4328,
which is the SharedWaiters PKEVENT from the ERESOURCE in question.

But this thread already owns the ERESOURCE shared. According to IFS kit
documentation, a second call to ExAcquireResourceSharedLite() by a thread
that already owns the resource should grant the same type of access
recursively. Why is he waiting for it again? Is it possible that the
Exclusive Waiter blocks ALL subsequent attempts to acquire the resource
shared, even if the current thread already owns the resource shared?

Carl Appellof

You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

2

Carl,

If this thread you’ve listed below really does already hold the VCB
resource shared, it will be able to reaquired that same resource shared
even if there is an exclusive waiter in the wait queue.

Are you using the !locks debugger command to see the list of owners and
shared waiters on this ERESOURCE? If so, take a closer look at all the
threads listed. The first thread in the list is not necessarily the
shared owner of the ERESOURCE. I’ve seen occurances when the shared
owner is buried in deep in that list of threads.

Thanks,

Molly Brown
Microsoft Corporation

This posting is provided “AS IS” with no warranties, and confers no
rights. You assume all risk for your use. Copyright 2001 Microsoft
Corporation. All rights reserved.

-----Original Message-----
From: xxxxx@veritas.com [mailto:xxxxx@veritas.com]
Sent: Tuesday, October 30, 2001 10:58 AM
To: File Systems Developers
Subject: [ntfsd] Recursive shared ERESOURCE acquisition

Our filter driver seems to have hit a deadlock with contention for
ownership of an ERESOURCE lock. At the time of the hang, we break into
the
system and see a particular ERESOURCE lock with 1 shared owner, 1
exclusive
waiter, and 10 shared waiters. The shared waiters are all waiting a few

levels deep in NtfsAcquireSharedVcb(), all waiting to do something on
the
same volume. The exclusive waiter is waiting a few levels deep in
NtfsAcquireExclusiveVcb(). So far, not so bad.

But the shared owner of the lock seems to be waiting to acquire the SAME

LOCK again, shared. Here’s the stack of the owner (apologies for
wrapping):

ChildEBP RetAddr Args to Child
b99614b4 8042bd4d 00000000 813fb504 ffa4e350 nt!KiSwapThread+0x1b1
b99614dc 804152f2 813e4328 00000000 00000000
nt!KeWaitForSingleObject+0x1a3
b996151c 80414886 811207f8 b99617e8 00000000 nt!ExpWaitForResource+0x1b4
b9961534 bfe9c674 813fb500 00000001 b99617e8
nt!ExAcquireResourceSharedLite+0xc6
b9961544 bfe9c1aa b99617e8 813fb100 00000001
ntfs!NtfsAcquireSharedVcb+0x1b
b99617a0 bfe9b971 b99617e8 811862c8 813fb030
ntfs!NtfsCommonCleanup+0x134 b9961918 8041d915 813fb030 811862c8
811864a0 ntfs!NtfsFsdCleanup+0x113 b996192c b9c56874 811864a0 811864c4
811862d8 nt!IopfCallDriver+0x35 b9961968 b9c58e1e 811b73e8 811862c8
00000000 rxfsfltr!CallAndRelease+0x176 b99619d8 8041d915 811b7330
811862c8 811207f8 rxfsfltr!FilterDispatch+0x449 b99619ec 804c1e67
80064ffc 820c5b40 00000001 nt!IopfCallDriver+0x35 b9961a20 804d840b
820ad030 811b7330 01020000 nt!IopCloseFile+0x267 b9961a4c 8044f37a
820ad030 811207e4 811207f8
nt!ObpDecrementHandleCount+0x13d
b9961b00 80465679 000002e0 e3155528 00000000 nt!NtClose+0x1f0 b9961b00
8042f917 000002e0 e3155528 00000000 nt!KiSystemService+0xc9 b9961b7c
b99a42a1 000002e0 000002e0 00000000 nt!ZwClose+0xb

Anyway, you can see he’s waiting a few levels deep in
NtfsAcquireSharedVcb(), actually waiting for the KEVENT object at
813e4328,
which is the SharedWaiters PKEVENT from the ERESOURCE in question.

But this thread already owns the ERESOURCE shared. According to IFS kit

documentation, a second call to ExAcquireResourceSharedLite() by a
thread
that already owns the resource should grant the same type of access
recursively. Why is he waiting for it again? Is it possible that the
Exclusive Waiter blocks ALL subsequent attempts to acquire the resource
shared, even if the current thread already owns the resource shared?

Carl Appellof

You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

3

Thanks, Molly. It was indeed the case that the first thread listed by the
!locks extension wasn’t really the owner of the resource.

I still have a deadlock, but its cause is not what I thought.

Carl

You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

4

On 10/31/01, ““Molly Brown” ” wrote:
>
…
>
> Are you using the !locks debugger command to see the list of owners and
> shared waiters on this ERESOURCE? If so, take a closer look at all the
> threads listed. The first thread in the list is not necessarily the
> shared owner of the ERESOURCE. I’ve seen occurances when the shared
> owner is buried in deep in that list of threads.
>

Molly,

Do you or anyone else know if there’s a definitive way of finding the owner
thread for a resource, given the info in the ERESOURCE object? The only
way I’ve found so far is to use process of elimination to examine all the
threads listed by the !locks extension to find the one that’s NOT waiting
on one of the ERESOURCE’s dispatcher objects.

Neither the “OwnerTable” nor the “OwnerThreads” fields of the ERESOURCE
lead me to the right thread directly.

Carl

—
You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

5

Not really. I’ve had to resort to dumping the ERESOURCE object via the “dt” command and then walking the list of the shared or exclusive waiters depending on how it was acquired. The “!locks” command won’t necessarily show you all the owners either so don’t assume that the list is complete. The only sure way to find all the owners is in the lists. (Unless someone can suggest a better way.)

–jerry

----- Original Message -----
From: xxxxx@veritas.com
Sent: Tuesday, November 06, 2001 7:35 PM
To: File Systems Developers
Subject: [ntfsd] RE: Recursive shared ERESOURCE acquisition

On 10/31/01, ““Molly Brown” ” wrote:
>
…
>
> Are you using the !locks debugger command to see the list of owners and
> shared waiters on this ERESOURCE? If so, take a closer look at all the
> threads listed. The first thread in the list is not necessarily the
> shared owner of the ERESOURCE. I’ve seen occurances when the shared
> owner is buried in deep in that list of threads.
>

Molly,

Do you or anyone else know if there’s a definitive way of finding the owner
thread for a resource, given the info in the ERESOURCE object? The only
way I’ve found so far is to use process of elimination to examine all the
threads listed by the !locks extension to find the one that’s NOT waiting
on one of the ERESOURCE’s dispatcher objects.

Neither the “OwnerTable” nor the “OwnerThreads” fields of the ERESOURCE
lead me to the right thread directly.

Carl

—
You are currently subscribed to ntfsd as: xxxxx@msn.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.comGet more from the Web. FREE MSN Explorer download : http://explorer.msn.com

—
You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

6

I soon realized that my post below was referring to finding the complete list of waiters, not the owner. (Not enough coffee yet.) What I meant to say was that you can look at the owners list in the ERESOURCE object for the complete list of owners. You can have several if it’s owned shared.

Keep in mind that the owner list is a list of entries that will point you to the thread. You’ll have to dump each list entry and find the thread ID. If memory serves me, when you dump each entry, say with the “dd” command, the first DWORD is the forward link, followed by the backward link, the thread, and then a pointer to a common header. So if you look at the third DWORD value for each entry, you should find the thread ID. When I’ve looked at deadlocks, this is how I’ve been able to definitively find all the owners.

----- Original Message -----
From: Jerry Kelley
Sent: Wednesday, November 07, 2001 6:27 AM
To: File Systems Developers
Subject: [ntfsd] RE: Recursive shared ERESOURCE acquisition

Not really. I’ve had to resort to dumping the ERESOURCE object via the “dt” command and then walking the list of the shared or exclusive waiters depending on how it was acquired. The “!locks” command won’t necessarily show you all the owners either so don’t assume that the list is complete. The only sure way to find all the owners is in the lists. (Unless someone can suggest a better way.)

–jerry

----- Original Message -----
From: xxxxx@veritas.com
Sent: Tuesday, November 06, 2001 7:35 PM
To: File Systems Developers
Subject: [ntfsd] RE: Recursive shared ERESOURCE acquisition

On 10/31/01, ““Molly Brown” ” wrote:
>
…
>
> Are you using the !locks debugger command to see the list of owners and
> shared waiters on this ERESOURCE? If so, take a closer look at all the
> threads listed. The first thread in the list is not necessarily the
> shared owner of the ERESOURCE. I’ve seen occurances when the shared
> owner is buried in deep in that list of threads.
>

Molly,

Do you or anyone else know if there’s a definitive way of finding the owner
thread for a resource, given the info in the ERESOURCE object? The only
way I’ve found so far is to use process of elimination to examine all the
threads listed by the !locks extension to find the one that’s NOT waiting
on one of the ERESOURCE’s dispatcher objects.

Neither the “OwnerTable” nor the “OwnerThreads” fields of the ERESOURCE
lead me to the right thread directly.

Carl

—
You are currently subscribed to ntfsd as: xxxxx@msn.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

—
You are currently subscribed to ntfsd as: xxxxx@msn.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Get more from the Web. FREE MSN Explorer download : http://explorer.msn.comGet more from the Web. FREE MSN Explorer download : http://explorer.msn.com

—
You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com