Following is the description of a deadlock between two threads fffffa800eae32e0 and fffffa800c159bb0. Both the threads are from different processes. The name of my driver is mydrv.sys which is present in both the threads.
0: kd> !locks
**** DUMP OF ALL RESOURCE OBJECTS ****
KD: Scanning for held locks…
Resource @ 0xfffffa800d52c790 Shared 1 owning threads
Contention Count = 2
Threads: fffffa800c804720-01<*>
KD: Scanning for held locks.
Resource @ 0xfffffa800d5364c0 Exclusively owned
Contention Count = 24
NumberOfExclusiveWaiters = 2
Threads: fffffa800eae32e0-01<*> ---------------------> Thread 1 holding
Threads Waiting On Exclusive Access:
fffffa800c804720 fffffa800c159bb0 ----------> Thread 2 waiting
KD: Scanning for held locks…
Resource @ 0xfffffa800e922e50 Shared 1 owning threads
Threads: fffffa800c159bb0-01<*>
KD: Scanning for held locks…
Resource @ 0xfffffa800e8974e0 Exclusively owned
Contention Count = 99
NumberOfSharedWaiters = 1
Threads: fffffa800c159bb0-01<*> fffffa800eae32e0-01 -----> Thread 2 holding and Thread 1 waiting.
KD: Scanning for held locks…
7443 total locks, 4 locks currently held
0: kd> .thread fffffa800eae32e0
Implicit thread is now fffffa80`0eae32e0
0: kd> k
*** Stack trace for last set context - .thread/.cxr resets it
Child-SP RetAddr Call Site
fffffa6005b701a0 fffff800016b1f8a nt!KiSwapContext+0x7f
fffffa6005b702e0 fffff800016b338a nt!KiSwapThread+0x2fa
fffffa6005b70350 fffff800016b6665 nt!KeWaitForSingleObject+0x2da
fffffa6005b703e0 fffffa60010204ec nt!ExAcquireResourceSharedLite+0x265
fffffa6005b70450 fffffa6001011798 Ntfs!NtfsLookupAllocation+0x2cc
fffffa6005b705c0 fffffa60010b76ff Ntfs!NtfsReserveClusters+0x7e8
fffffa6005b706b0 fffffa6000d2f526 Ntfs!NtfsFastIoCheckIfPossible+0x1df
fffffa6005b70880 fffffa6000d4ca05 fltmgr!FltpPerformFastIoCall+0x366
fffffa6005b708e0 fffffa6004c6d63b fltmgr!FltpFastIoCheckIfPossible+0xc5
fffffa6005b70980 fffffa6004c6d7d6 mydrv!FastIoCheckIsPossibleFunc+0x19f
fffffa6005b70a20 fffff800019389ca mydrv!FastIoWriteFunc+0x96
fffffa6005b70ab0 fffff800016abe73 nt!NtWriteFile+0x599
fffffa6005b70bb0 0000000076ed5afa nt!KiSystemServiceCopyEnd+0x13
000000000864c818 0000000000000000 0x76ed5afa
0: kd> .thread fffffa800c159bb0
Implicit thread is now fffffa80`0c159bb0
0: kd> k
*** Stack trace for last set context - .thread/.cxr resets it
Child-SP RetAddr Call Site
fffffa60019c89b0 fffff800016b1f8a nt!KiSwapContext+0x7f
fffffa60019c8af0 fffff800016b338a nt!KiSwapThread+0x2fa
fffffa60019c8b60 fffff800016c3dfc nt!KeWaitForSingleObject+0x2da
fffffa60019c8bf0 fffffa600101b967 nt!ExAcquireResourceExclusiveLite+0x1cc
fffffa60019c8c60 fffffa60010edfd2 Ntfs!NtfsAcquireExclusiveScb+0x47
fffffa60019c8ca0 fffffa60010ca2af Ntfs!NtfsAllocateClusters+0x82
fffffa60019c8e20 fffffa6001013d78 Ntfs!NtfsReallocateRange+0x14f
fffffa60019c8f10 fffffa6001021bc7 Ntfs!NtfsPrepareComplexBuffers+0x3b8
fffffa60019c8fd0 fffffa600101f963 Ntfs!NtfsPrepareBuffers+0x177
fffffa60019c9050 fffffa600101bf2e Ntfs!NtfsNonCachedIo+0x1c3
fffffa60019c9220 fffffa600101f118 Ntfs!NtfsCommonWrite+0x4ce
fffffa60019c9420 fffffa6000d2fe17 Ntfs!NtfsFsdWrite+0x208
fffffa60019c9690 fffffa6000d2f0dd fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x227
fffffa60019c9700 fffffa6004c70ae2 fltmgr!FltpDispatch+0xcd
fffffa60019c9760 fffffa6004c67181 mydrv!MDFsdDispatch+0x82
fffffa60019c97b0 fffff800016cbbbd mydrv!FSDWrite+0x31
fffffa60019c9800 fffff800016ca8fd nt!IoSynchronousPageWrite+0x14d
fffffa60019c9830 fffff800016be069 nt!MiFlushSectionInternal+0x7ce
fffffa60019c9a30 fffff800016c1989 nt!MmFlushSection+0x2a9
fffffa60019c9af0 fffff800016c5450 nt!CcFlushCache+0x65a
fffffa60019c9be0 fffff800016c5f9b nt!CcWriteBehind+0x1c0
fffffa60019c9c70 fffff800016b9056 nt!CcWorkerThread+0x17b
fffffa60019c9cf0 fffff800018cf8b3 nt!ExpWorkerThread+0x11a
fffffa60019c9d50 fffff800016e64f6 nt!PspSystemThreadStartup+0x57
fffffa60019c9d80 0000000000000000 nt!KiStartSystemThread+0x16
Deactivating mydrv.sys is not creating the deadlock. I am not sure what is the problem in my driver creating the deadlock.
Thanks in advance for your help.
~Manoj
Legacy filter?
Are you passing down the Fastio locking calls? Specifically
FAST_IO_DISPATCH.AcquireForCcFlush ?
wrote in message news:xxxxx@ntfsd…
> Following is the description of a deadlock between two threads
> fffffa800eae32e0 and fffffa800c159bb0. Both the threads are from different
> processes. The name of my driver is mydrv.sys which is present in both the
> threads.
>
> 0: kd> !locks
> DUMP OF ALL RESOURCE OBJECTS
> KD: Scanning for held locks…
>
> Resource @ 0xfffffa800d52c790 Shared 1 owning threads
> Contention Count = 2
> Threads: fffffa800c804720-01<>
> KD: Scanning for held locks.
>
> Resource @ 0xfffffa800d5364c0 Exclusively owned
> Contention Count = 24
> NumberOfExclusiveWaiters = 2
> Threads: fffffa800eae32e0-01<> ---------------------> Thread 1
> holding
> Threads Waiting On Exclusive Access:
> fffffa800c804720 fffffa800c159bb0 ----------> Thread
> 2 waiting
>
> KD: Scanning for held
> locks…
>
> Resource @ 0xfffffa800e922e50 Shared 1 owning threads
> Threads: fffffa800c159bb0-01<>
> KD: Scanning for held locks…
>
> Resource @ 0xfffffa800e8974e0 Exclusively owned
> Contention Count = 99
> NumberOfSharedWaiters = 1
> Threads: fffffa800c159bb0-01<> fffffa800eae32e0-01 -----> Thread 2
> holding and Thread 1 waiting.
> KD: Scanning for held locks…
> 7443 total locks, 4 locks currently held
>
>
>
> 0: kd> .thread fffffa800eae32e0
> Implicit thread is now fffffa800eae32e0<br>><br>> 0: kd> k<br>> ***Stack trace for last set context - .thread/.cxr resets it<br>> Child-SP RetAddr Call Site<br>> fffffa6005b701a0 fffff800016b1f8a nt!KiSwapContext+0x7f<br>> fffffa6005b702e0 fffff800016b338a nt!KiSwapThread+0x2fa<br>> fffffa6005b70350 fffff800016b6665 nt!KeWaitForSingleObject+0x2da<br>> fffffa6005b703e0 fffffa60010204ec nt!ExAcquireResourceSharedLite+0x265<br>> fffffa6005b70450 fffffa6001011798 Ntfs!NtfsLookupAllocation+0x2cc<br>> fffffa6005b705c0 fffffa60010b76ff Ntfs!NtfsReserveClusters+0x7e8<br>> fffffa6005b706b0 fffffa6000d2f526 Ntfs!NtfsFastIoCheckIfPossible+0x1df<br>> fffffa6005b70880 fffffa6000d4ca05 fltmgr!FltpPerformFastIoCall+0x366<br>> fffffa6005b708e0 fffffa6004c6d63b fltmgr!FltpFastIoCheckIfPossible+0xc5<br>> fffffa6005b70980 fffffa6004c6d7d6 mydrv!FastIoCheckIsPossibleFunc+0x19f<br>> fffffa6005b70a20 fffff800019389ca mydrv!FastIoWriteFunc+0x96<br>> fffffa6005b70ab0 fffff800016abe73 nt!NtWriteFile+0x599<br>> fffffa6005b70bb0 0000000076ed5afa nt!KiSystemServiceCopyEnd+0x13<br>> 000000000864c818 0000000000000000 0x76ed5afa<br>><br>><br>> 0: kd> .thread fffffa800c159bb0<br>> Implicit thread is now fffffa800c159bb0
>
> 0: kd> k
>*** Stack trace for last set context - .thread/.cxr resets it
> Child-SP RetAddr Call Site
> fffffa60019c89b0 fffff800016b1f8a nt!KiSwapContext+0x7f
> fffffa60019c8af0 fffff800016b338a nt!KiSwapThread+0x2fa
> fffffa60019c8b60 fffff800016c3dfc nt!KeWaitForSingleObject+0x2da
> fffffa60019c8bf0 fffffa600101b967
> nt!ExAcquireResourceExclusiveLite+0x1cc
> fffffa60019c8c60 fffffa60010edfd2 Ntfs!NtfsAcquireExclusiveScb+0x47
> fffffa60019c8ca0 fffffa60010ca2af Ntfs!NtfsAllocateClusters+0x82
> fffffa60019c8e20 fffffa6001013d78 Ntfs!NtfsReallocateRange+0x14f
> fffffa60019c8f10 fffffa6001021bc7 Ntfs!NtfsPrepareComplexBuffers+0x3b8
> fffffa60019c8fd0 fffffa600101f963 Ntfs!NtfsPrepareBuffers+0x177
> fffffa60019c9050 fffffa600101bf2e Ntfs!NtfsNonCachedIo+0x1c3
> fffffa60019c9220 fffffa600101f118 Ntfs!NtfsCommonWrite+0x4ce
> fffffa60019c9420 fffffa6000d2fe17 Ntfs!NtfsFsdWrite+0x208
> fffffa60019c9690 fffffa6000d2f0dd
> fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x227
> fffffa60019c9700 fffffa6004c70ae2 fltmgr!FltpDispatch+0xcd
> fffffa60019c9760 fffffa6004c67181 mydrv!MDFsdDispatch+0x82
> fffffa60019c97b0 fffff800016cbbbd mydrv!FSDWrite+0x31
> fffffa60019c9800 fffff800016ca8fd nt!IoSynchronousPageWrite+0x14d
> fffffa60019c9830 fffff800016be069 nt!MiFlushSectionInternal+0x7ce
> fffffa60019c9a30 fffff800016c1989 nt!MmFlushSection+0x2a9
> fffffa60019c9af0 fffff800016c5450 nt!CcFlushCache+0x65a
> fffffa60019c9be0 fffff800016c5f9b nt!CcWriteBehind+0x1c0
> fffffa60019c9c70 fffff800016b9056 nt!CcWorkerThread+0x17b
> fffffa60019c9cf0 fffff800018cf8b3 nt!ExpWorkerThread+0x11a
> fffffa60019c9d50 fffff800016e64f6 nt!PspSystemThreadStartup+0x57
> fffffa60019c9d80 0000000000000000 nt!KiStartSystemThread+0x16
>
> Deactivating mydrv.sys is not creating the deadlock. I am not sure what is
> the problem in my driver creating the deadlock.
>
> Thanks in advance for your help.
>
> ~Manoj
>
>
Hi Rod,
Yes, My driver is a legacy filter driver.
Following is my code for the call to check FastIoCheckIfPossible.
PFAST_IO_DISPATCH fastioPtr = ext->filteredDevObj->DriverObject->FastIoDispatch;
if(irql==PASSIVE_LEVEL){
KeRaiseIrql(APC_LEVEL, &irql);
KeRaiseIrqlCalled=TRUE;
}
ret = (*fastioPtr->FastIoCheckIfPossible)( FileObject, FileOffset, Length,
Wait, LockKey, CheckForReadOperation, IoStatus, ext->filteredDevObj );
if(KeRaiseIrqlCalled==TRUE){
KeLowerIrql(irql);
KeRaiseIrqlCalled=FALSE;
}
Before calling this I am getting the LockKey by using FsRtlFastCheckLockForRead or FsRtlFastCheckLockForWrite.
I am not using FAST_IO_DISPATCH.AcquireForCcFlush. It is initialized to NULL.
Thanks
Manoj
On 9/7/2010 3:55 AM, xxxxx@gmail.com wrote:
Following is my code for the call to check FastIoCheckIfPossible.
PFAST_IO_DISPATCH fastioPtr = ext->filteredDevObj->DriverObject->FastIoDispatch;
if(irql==PASSIVE_LEVEL){
KeRaiseIrql(APC_LEVEL,&irql);
KeRaiseIrqlCalled=TRUE;
}
Two things … 1) Why are you raising your IRQL to APC? There is no need.
I am not using FAST_IO_DISPATCH.AcquireForCcFlush. It is initialized to NULL.
- You need to pass all Fast IO requests through, regardless of whether
you actually do anything with them.
Pete
–
Kernel Drivers
Windows File System and Device Driver Consulting
www.KernelDrivers.com
866.263.9295
Hi Pete,
Following are the answers:
-
Why are you raising your IRQL to APC? There is no need.
Ans: NtfsFastIoCheckIfPossible will call ExAcquireFastMutexUnsafe which requires irql=APC_LEVEL otherwise driver verifier will bluescreen on ntfs.sys.
-
You need to pass all Fast IO requests through, regardless of whether
you actually do anything with them.
Ans: Do you think this is the cause of the hang? If yes then I shall try to implement it.
Thanks
Manoj
On 9/7/2010 8:22 AM, xxxxx@gmail.com wrote:
- You need to pass all Fast IO requests through, regardless of whether
you actually do anything with them.
Ans: Do you think this is the cause of the hang? If yes then I shall try to implement it.
You need to pass all FastIO, as well as IRP, requests through to the
underlying file system. If you don’t you are breaking the call chain and
all bets are off.
Pete
–
Kernel Drivers
Windows File System and Device Driver Consulting
www.KernelDrivers.com
866.263.9295
Hi Pete,
I found at following MSDN documentation
http://msdn.microsoft.com/en-us/library/ff541579(VS.85).aspx
Note On Microsoft Windows XP and later, the following fast I/O callback routines are obsolete and should not be used by file system filter drivers:
AcquireForCcFlush
AcquireFileForNtCreateSection
AcquireForModWrite
ReleaseForCcFlush
ReleaseFileForNtCreateSection
ReleaseForModWrite
This deadlock I am facing on Windows 2008 server.
Thanks
Manoj
On 9/8/2010 12:46 AM, xxxxx@gmail.com wrote:
Hi Pete,
I found at following MSDN documentation
http://msdn.microsoft.com/en-us/library/ff541579(VS.85).aspx
Note On Microsoft Windows XP and later, the following fast I/O callback routines are obsolete and should not be used by file system filter drivers:
AcquireForCcFlush
AcquireFileForNtCreateSection
AcquireForModWrite
ReleaseForCcFlush
ReleaseFileForNtCreateSection
ReleaseForModWrite
Regardless of the docs, if you are a legacy filter the above locking
routines DO get called and you need to pass them down. While your
problem may not be related to this, I would start there. Pass down the
lock/unlock calls, if that doesn’t fix the problem then dig more.
One more point, which I just noticed in the stack trace you posted. Why
are you calling the NTFS.FastIoCheckIfPossible from within a FastIoWrite
call? The determination of whether FastIO is possible has already been
made by the time you get the FastIoWrite call. Hence don’t call the
FastIoCheckIfPossible from within your FastIoWrite routine, just pass
down the write call.
Pete
This deadlock I am facing on Windows 2008 server.
Thanks
Manoj
NTFSD is sponsored by OSR
For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
–
Kernel Drivers
Windows File System and Device Driver Consulting
www.KernelDrivers.com
866.263.9295
Hi Pete,
Following is the explanation why I am calling FastIoCheckIfPossible within FastIoWrite call.
My driver is a legacy FS filter driver.
When NTFS, or FAT,processes a fast IO read/write, they only call the fast IO check function of the topmost driver if the fast IO state is set to questionable in the FSTRL_COMMON_FCB_HEADER. This state is set to questionable when a region lock is applied. I, however, off-load region locking from the file system and a side affect of this is that the fast IO state is never set to questionable. This, in turn, causes NTFS to not call my fast IO check function on a fast IO read/write, allowing a read/write from a locked region.
To avoid this I always call my FastIoCheckIfPossible function during a fast IO read/write.
Thanks
Manoj
On 9/13/2010 1:52 AM, xxxxx@gmail.com wrote:
Hi Pete,
Following is the explanation why I am calling FastIoCheckIfPossible within FastIoWrite call.
My driver is a legacy FS filter driver.
When NTFS, or FAT,processes a fast IO read/write, they only call the fast IO check function of the topmost driver if the fast IO state is set to questionable in the FSTRL_COMMON_FCB_HEADER. This state is set to questionable when a region lock is applied. I, however, off-load region locking from the file system and a side affect of this is that the fast IO state is never set to questionable. This, in turn, causes NTFS to not call my fast IO check function on a fast IO read/write, allowing a read/write from a locked region.
Since you are offloading the BR locking on files, can’t you make a
decision whether to allow the fast IO request? Alternatively you could
simply return FALSE always from these routines and your IRP dispatch
handler would be invoked.
Pete
–
Kernel Drivers
Windows File System and Device Driver Consulting
www.KernelDrivers.com
866.263.9295