Dead Lock in MiMappedPageWriter?

During testing against my filter dirver, I encounter a deadlock in MiMappedPageWriter.
From the output of !locks, I found 3 resources, just as follows:
kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks…

Resource @ 0x895fb514 Shared 1 owning threads
Contention Count = 9
Threads: 8905ea38-01<*>

Resource @ 0x8966bfc8 Shared 1 owning threads
Contention Count = 303
NumberOfSharedWaiters = 1
NumberOfExclusiveWaiters = 1
Threads: 896d23c8-01 89656c68-01<*>
Threads Waiting On Exclusive Access:
8905ea38

Resource @ 0x89360c68 Shared 1 owning threads
Threads: 896d28bb-01<*> *** Actual Thread 896d28b8
2904 total locks, 3 locks currently held

After using !thread to view the detail information for threads, I see that 8905ea38 is waiting for 0x8966bfc8(Exclusive), so the key here is 89656c68 who owned the 0x8966bfc8 shared.

The following is the output for !thread 89656c68:
kd> !thread 89656c68
THREAD 89656c68 Cid 0004.0064 Teb: 00000000 Win32Thread: 00000000 WAIT: (WrVirtualMemory) KernelMode Non-Alertable
805590a0 NotificationEvent
Not impersonating
DeviceMap e1004428
Owning Process 0 Image:
Attached Process 896d3830 Image: System
Wait Start TickCount 17783 Ticks: 8009 (0:00:02:05.140)
Context Switch Count 335
UserTime 00:00:00.000
KernelTime 00:00:00.046
Start Address nt!MiMappedPageWriter (0x8050c928)
Stack Init bad38000 Current bad37d24 Base bad38000 Limit bad35000 Call 0
Priority 17 BasePriority 8 PriorityDecrement 0 DecrementCount 0
ChildEBP RetAddr Args to Child
bad37d3c 80501cd6 89656cd8 89656c68 804fad62 nt!KiSwapContext+0x2e (FPO: [Uses EBP] [0,0,4])
bad37d48 804fad62 806d32d0 805590b0 00000000 nt!KiSwapThread+0x46 (FPO: [0,0,0])
bad37d70 8050c97d 00000000 00000012 00000000 nt!KeWaitForSingleObject+0x1c2 (FPO: [Non-Fpo])
bad37dac 805c7160 00000000 00000000 00000000 nt!MiMappedPageWriter+0x55 (FPO: [Non-Fpo])
bad37ddc 80542dd2 8050c928 00000000 00000000 nt!PspSystemThreadStartup+0x34 (FPO: [Non-Fpo])
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

From above output, I found 89656c68 is waiting for something which is not exist in the output of !locks, is it lead to the deadlock? What is MiMappedPageWriter waiting for?

Given that it’s the Mapped Page Writer, I would guess that it’s waiting for
a write to complete. You need to dump the other threads in the System
process (!process 0 F System) and check if anyone is processing a write
(possibly thread 896d23c8 or 8905ea38).

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntfsd…

During testing against my filter dirver, I encounter a deadlock in
MiMappedPageWriter.
From the output of !locks, I found 3 resources, just as follows:
kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks…

Resource @ 0x895fb514 Shared 1 owning threads
Contention Count = 9
Threads: 8905ea38-01<*>

Resource @ 0x8966bfc8 Shared 1 owning threads
Contention Count = 303
NumberOfSharedWaiters = 1
NumberOfExclusiveWaiters = 1
Threads: 896d23c8-01 89656c68-01<*>
Threads Waiting On Exclusive Access:
8905ea38

Resource @ 0x89360c68 Shared 1 owning threads
Threads: 896d28bb-01<*> *** Actual Thread 896d28b8
2904 total locks, 3 locks currently held

After using !thread to view the detail information for threads, I see that
8905ea38 is waiting for 0x8966bfc8(Exclusive), so the key here is 89656c68
who owned the 0x8966bfc8 shared.

The following is the output for !thread 89656c68:
kd> !thread 89656c68
THREAD 89656c68 Cid 0004.0064 Teb: 00000000 Win32Thread: 00000000 WAIT:
(WrVirtualMemory) KernelMode Non-Alertable
805590a0 NotificationEvent
Not impersonating
DeviceMap e1004428
Owning Process 0 Image:
Attached Process 896d3830 Image: System
Wait Start TickCount 17783 Ticks: 8009 (0:00:02:05.140)
Context Switch Count 335
UserTime 00:00:00.000
KernelTime 00:00:00.046
Start Address nt!MiMappedPageWriter (0x8050c928)
Stack Init bad38000 Current bad37d24 Base bad38000 Limit bad35000 Call 0
Priority 17 BasePriority 8 PriorityDecrement 0 DecrementCount 0
ChildEBP RetAddr Args to Child
bad37d3c 80501cd6 89656cd8 89656c68 804fad62 nt!KiSwapContext+0x2e (FPO:
[Uses EBP] [0,0,4])
bad37d48 804fad62 806d32d0 805590b0 00000000 nt!KiSwapThread+0x46 (FPO:
[0,0,0])
bad37d70 8050c97d 00000000 00000012 00000000 nt!KeWaitForSingleObject+0x1c2
(FPO: [Non-Fpo])
bad37dac 805c7160 00000000 00000000 00000000 nt!MiMappedPageWriter+0x55
(FPO: [Non-Fpo])
bad37ddc 80542dd2 8050c928 00000000 00000000 nt!PspSystemThreadStartup+0x34
(FPO: [Non-Fpo])
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

From above output, I found 89656c68 is waiting for something which is not
exist in the output of !locks, is it lead to the deadlock? What is
MiMappedPageWriter waiting for?

Thanks, Scott. You are correct, 896d23c8 is trying to set FileEndOfFileInformation with AdvanceOnly == TRUE, and it is waiting for 0x8966bfc8 shared(for which 8905ea38 is also waiting exclusively).
But I have no idea how to solve this problem, in fact 0x8966bfc8 is the PagingIoResource of a ShadowFileObject. The Shadow File Object was created in PreCreate routine, after processing some write operations, my filter received a cleanup request. So my filter tried to call FltClose in PreCleanup routine to do the cleanup operation, but it seems the MiMappedPageWriter has not finished doing its job yet, thus the deadlock occurs.
How does file system drivers avoid deadlocks in such condition?

> But I have no idea how to solve this problem, in fact 0x8966bfc8 is the

PagingIoResource of a ShadowFileObject.

I think that it is very important that you understand the deadlock.
Otherwise you’ll just be cutting and pasting the code we give you into your
driver and when you hit the field badness will occur.

  1. What is the thread doing the Ex doing (this may not matter)
  2. Who owns the thread Shared? Why have they not released the resource?
    Are they backed up against some thing which requires that the MPwriter to
    run (like memory shortage)?

Finding the reason for (2) is critical and it will give you a good insight
into the dynamics of the system you are building. When you understand that
go and see whether there are other places where this will bite you (hint:
think about the paging write path and the paging).

Meanwhile, this is probably what you need:

http://msdn.microsoft.com/en-us/library/windows/hardware/ff544367(v=vs.85).aspx

…but as a help to yourself and a courtesy to the readers of this stream I
urge you to research and understand your problem. We all know how hard file
systems are, but we all prefer explaining the system so that you can work
out your solutions, rather than supplying fixes.

Have a great 2015

Dr Rod Widdowson CEng.
Consulting Partner
Steading System Software LLP

Thanks for your response, Rod.
This is the same for me that I wanna to understand the deadlock, it can help me to solve this kind of problems in the future.
But I’m a little confused about the output of !locks, I made some change to my code, the deadlock should gone, but the Mapped Page Writer still stopped, here is the output of !locks:
kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****

Resource @ 0x895fb514 Shared 1 owning threads
Contention Count = 30
Threads: 89189680-01<*>
KD: Scanning for held locks…

Resource @ 0x895b53d0 Shared 1 owning threads
Contention Count = 106
NumberOfExclusiveWaiters = 1
Threads: 89656c68-01<*>
Threads Waiting On Exclusive Access:
89189680

Resource @ 0x89318d68 Exclusively owned
Threads: 89189680-01<*>

Resource @ 0x891b06c8 Shared 1 owning threads
Threads: 896d23cb-01<*> *** Actual Thread 896d23c8
2852 total locks, 4 locks currently held

From the output, it seems there is no deadlock, so it is what that blocking Mapped Page Writer?

For a start this is a different hang from the previous one.

Resource @ 0x895b53d0 Shared 1 owning threads
Contention Count = 106
NumberOfExclusiveWaiters = 1
Threads: 89656c68-01<*>
Threads Waiting On Exclusive Access:
89189680

This is the only hang through a resource that I can see after a quick
glance. So what is 89656c68 doing? That is what is holding the resource
which 89189680 is waiting on.

Looking back to your original post:

Resource @ 0x8966bfc8 Shared 1 owning threads
Contention Count = 303
NumberOfSharedWaiters = 1
NumberOfExclusiveWaiters = 1
Threads: 896d23c8-01 89656c68-01<*>
Threads Waiting On Exclusive Access:
8905ea38

And as you said:

, 896d23c8 is trying to set FileEndOfFileInformation with AdvanceOnly ==
TRUE,
and it is waiting for 0x8966bfc8 shared(for which 8905ea38
is also waiting exclusively).

So, what is 89656c68 waiting on? That is what is blocking 8905ea38 which is
blocking 896d23c8


Yes. It looks like so, but it is still related with the Mapped Page Writer, and I suspect that it is the same deadlock as the previous one.

From my first post, it seems 89656c68 is waiting to do the mapped page writing, and 89189680 is in the processing of FltClose(which was called in PreCleanup routine), it is waiting for PagingIoResource Exclusive. So obvious 89656c68 is holding the PagingIoResource shared and do not want to release it.
What I do not understand is why mapped page writing do not release the PagingIoResource? From my lastest post, it seems Mapped Page Writer is not waiting for something else, it just stopped.

I have solved this problem.
In fact, it is because my filter modified the VDL in FSRTL_COMMON_FCB_HEADER of shadowFileObject.
The memory mapped page writer seems to wait for contents to be flushed to disk, but the VDL was modified by my filte without doing any other operations, so there should not be anything else wait to be flushed, thus the memory mapped page writer wait infinately.
Thanks for all of your help!