Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results
The free OSR Learning Library has more than 50 articles on a wide variety of topics about writing and debugging device drivers and Minifilters. From introductory level to advanced. All the articles have been recently reviewed and updated, and are written using the clear and definitive style you've come to expect from OSR over the years.
Check out The OSR Learning Library at: https://www.osr.com/osr-learning-library/
Hi,
I am working on minifilter driver based upon shadow file object design.
In IRP_MJ_CLOSE, we try to delete lowerfileobject through ObDereferenceObject, but after random iterations of file create->read/write->close, it crashes.
EXCEPTION_RECORD: ffffcf01832a7298 -- (.exr 0xffffcf01832a7298)
ExceptionAddress: fffff802410f3caf (nt!ExAcquireFastMutex+0x00000000000000cf)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000001
Parameter[1]: 00000000000001b7
Attempt to write to address 00000000000001b7
CONTEXT: ffffcf01832a6ae0 -- (.cxr 0xffffcf01832a6ae0)
rax=0000000000000000 rbx=0000000000000000 rcx=7ffffffffffffffc
rdx=000000000000003a rsi=ffff960a2f47e3c0 rdi=00000000000001b7
rip=fffff802410f3caf rsp=ffffcf01832a74d0 rbp=0000000000000001
r8=00000000ffffffff r9=ffff960a2d080880 r10=0000000000000000
r11=ffff960a2f47e040 r12=0000000000000000 r13=ffff960a2d978420
r14=0000000000000000 r15=0000000000000012
iopl=0 nv up ei pl zr na po nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010246
nt!ExAcquireFastMutex+0xcf:
fffff802410f3caf f00fba3700 lock btr dword ptr [rdi],0 ds:002b:00000000
000001b7=????????
Call stack:
STACK_TEXT:
ffffcf01832a74d0 fffff802
7e43307c : ffffffffffffffff ffff960a
00000001 ffff960a2f4d9638 ffff960a
2f4d95e8 : nt!ExAcquireFastMutex+0xcf
ffffcf01832a7510 fffff802
7e40360a : ffffcf01832a75e0 ffff960a
2e47d860 ffff960a2d9437e0 ffff960a
2eb2e020 : FLTMGR!FltpCleanupFileObjectContextForCleanup+0x4c
ffffcf01832a7540 fffff802
7e40312e : ffff960a2d080880 ffff960a
2f4d95e8 ffff960a2eb2e020 ffff960a
2d673080 : FLTMGR!FltpPassThrough+0x42a
ffffcf01832a75c0 fffff802
414e15a0 : ffff960a2e47d860 ffff960a
2d943040 0000000000000000 ffff960a
2f8bbbe8 : FLTMGR!FltpDispatch+0x9e
ffffcf01832a7620 fffff802
415ea926 : 0000000000000001 ffff960a
2e47d860 0000000000000001 00000000
00000000 : nt!IopCloseFile+0x150
ffffcf01832a76b0 fffff802
414fd2e8 : ffffcf01832a7929 00000000
00000000 ffff960a2d14cf20 00000000
10000004 : nt!IopDeleteFile+0x109746
ffffcf01832a7730 fffff802
4111cf21 : 0000000000000000 00000000
00000000 ffffcf01832a7929 ffff960a
2e47d860 : nt!ObpRemoveObjectRoutine+0x78
ffffcf01832a7790 fffff802
7f960f87 : 0000000000000000 ffff960a
2ea2dc00 ffff960a2d9784a0 00000000
00000000 : nt!ObfDereferenceObject+0xa1
ffffcf01832a77d0 fffff802
7e4046ca : ffff960a2ea2dcd8 ffffcf01
832a7900 ffffcf01832a78e0 ffff960a
2d9437e0 : MyDriver!PfmCloseCallback+0x437
ffffcf01832a7880 fffff802
7e404278 : ffffcf01832a7a60 ffff960a
2f797900 0000000000000002 ffff960a
2f142000 : FLTMGR!FltpPerformPreCallbacks+0x2ea
ffffcf01832a7990 fffff802
7e403386 : ffff960a2f142010 ffffcf01
832a7a60 ffff960a2f142010 ffffcf01
832a7a70 : FLTMGR!FltpPassThroughInternal+0x88
ffffcf01832a79c0 fffff802
7e40312e : fffffffffffe7960 00000000
00000000 ffff960a2f142368 00000000
00000000 : FLTMGR!FltpPassThrough+0x1a6
ffffcf01832a7a40 fffff802
810f10f5 : ffff960a2f142320 ffff960a
2fdbb520 0000000000000000 00000000
00000001 : FLTMGR!FltpDispatch+0x9e
ffffcf01832a7aa0 fffff802
810fd1d4 : ffff960a2f142010 00000000
00000001 ffff960a2f142010 00000000
00000190 : FSpy+0x10f5
ffffcf01832a7b10 fffff802
414e130d : ffff960a2f142010 ffff960a
2d297080 ffff960a2f142010 ffff960a
2f4e2ec0 : FSpy+0xd1d4
ffffcf01832a7b40 fffff802
414fd2e8 : fffff8024146630c 00000000
00000001 ffff960a2d14cf20 00000000
00000001 : nt!IopDeleteFile+0x12d
ffffcf01832a7bc0 fffff802
414665e0 : 0000000000000000 ffff960a
2d297050 fffff8024146630c fffff802
41314f40 : nt!ObpRemoveObjectRoutine+0x78
ffffcf01832a7c20 fffff802
410dae09 : ffff960a2f47e040 fffff802
4146630c fffff80241314f40 fffff802
413cf280 : nt!ObpProcessRemoveObjectQueue+0x2d4
ffffcf01832a7cc0 fffff802
410ab2c1 : ffff960a2f47e040 00000000
00000080 ffff960a2d021040 ffff960a
2f47e040 : nt!ExpWorkerThread+0xe9
ffffcf01832a7d50 fffff802
41178c76 : ffffcf017ffc7180 ffff960a
2f47e040 fffff802410ab280 00000000
02000005 : nt!PspSystemThreadStartup+0x41
ffffcf01832a7da0 00000000
00000000 : ffffcf01832a8000 ffffcf01
832a2000 0000000000000000 00000000
00000000 : nt!KiStartSystemThread+0x16
Can anyone please help to understand this issue.
Thanks a lot in advance!
Upcoming OSR Seminars | ||
---|---|---|
OSR has suspended in-person seminars due to the Covid-19 outbreak. But, don't miss your training! Attend via the internet instead! | ||
Writing WDF Drivers | 7 Dec 2020 | LIVE ONLINE |
Internals & Software Drivers | 25 Jan 2021 | LIVE ONLINE |
Developing Minifilters | 8 March 2021 | LIVE ONLINE |
Comments
What analysis have you done so far?
FWIW and in case the qoestions don't make it obvious, my money would be on you having a mismatch on your object reference and dereferences
Standard settings for mydriver are enabled in verifier.
Below settings are not enabled
ADDITIONAL FLAGS:
[ ] (0x00000004) Randomized low resources simulation
[ ] (0x00000200) Force pending I/O requests
[ ] (0x00000400) IRP logging
[ ] (0x00002000) Invariant MDL checking for stack
[ ] (0x00004000) Invariant MDL checking for driver
[ ] (0x00008000) Power framework delay fuzzing
[ ] (0x00010000) Port/miniport interface checking
[ ] (0x00040000) Systematic low resources simulation
[ ] (0x00080000) DDI compliance checking (additional)
[ ] (0x00200000) NDIS/WIFI verification
[ ] (0x00800000) Kernel synchronization delay fuzzing
[ ] (0x01000000) VM switch verification
[ ] (0x02000000) Code integrity checks
LowerFileObject(the one geting derefernced), its _FSRTL_ADVANCED_FCB_HEADER and _FAST_MUTEX seems to be valid,
1: kd> dx -r1 ((sfntpffd!_FILE_OBJECT *)0xffff960a2e47d860)
((sfntpffd!_FILE_OBJECT *)0xffff960a2e47d860) : 0xffff960a2e47d860 [Type: _FILE_OBJECT *]
[+0x000] Type : 5 [Type: short]
[+0x002] Size : 216 [Type: short]
[+0x008] DeviceObject : 0xffff960a2d8f9c80 : Device for "\Driver\volmgr" [Type: _DEVICE_OBJECT *]
[+0x010] Vpb : 0xffff960a2d792ea0 [Type: _VPB *]
[+0x018] FsContext : 0xffffbb0cb06a2c00 [Type: void *]
[+0x020] FsContext2 : 0xffffbb0cb06a2e48 [Type: void *]
[+0x028] SectionObjectPointer : 0xffff960a2f8ba518 [Type: _SECTION_OBJECT_POINTERS *]
[+0x030] PrivateCacheMap : 0x0 [Type: void *]
[+0x038] FinalStatus : 0 [Type: long]
[+0x040] RelatedFileObject : 0x0 [Type: _FILE_OBJECT *]
[+0x048] LockOperation : 0x0 [Type: unsigned char]
[+0x049] DeletePending : 0x0 [Type: unsigned char]
[+0x04a] ReadAccess : 0x1 [Type: unsigned char]
[+0x04b] WriteAccess : 0x0 [Type: unsigned char]
[+0x04c] DeleteAccess : 0x0 [Type: unsigned char]
[+0x04d] SharedRead : 0x1 [Type: unsigned char]
[+0x04e] SharedWrite : 0x1 [Type: unsigned char]
[+0x04f] SharedDelete : 0x1 [Type: unsigned char]
[+0x050] Flags : 0x44042 [Type: unsigned long]
[+0x058] FileName : "\enc1\test.txt" [Type: _UNICODE_STRING]
[+0x068] CurrentByteOffset : {0} [Type: _LARGE_INTEGER]
[+0x070] Waiters : 0x0 [Type: unsigned long]
[+0x074] Busy : 0x0 [Type: unsigned long]
[+0x078] LastLock : 0x0 [Type: void *]
[+0x080] Lock [Type: _KEVENT]
[+0x098] Event [Type: _KEVENT]
[+0x0b0] CompletionContext : 0x0 [Type: _IO_COMPLETION_CONTEXT *]
[+0x0b8] IrpListLock : 0x0 [Type: unsigned __int64]
[+0x0c0] IrpList [Type: _LIST_ENTRY]
[+0x0d0] FileObjectExtension : 0xffff960a2d1208a0 [Type: void *]
1: kd> dt nt!_FSRTL_ADVANCED_FCB_HEADER 0xffffbb0cb06a2c00
+0x000 NodeTypeCode : 0n1797
+0x002 NodeByteSize : 0n584
+0x004 Flags : 0x40 '@'
+0x005 IsFastIoPossible : 0x2 ''
+0x006 Flags2 : 0x2 ''
+0x007 Reserved : 0y0000
+0x007 Version : 0y0011
+0x008 Resource : 0xffff960a
2f8ba4a0 _ERESOURCE +0x010 PagingIoResource : 0xffff960a
2f8ba550 _ERESOURCE+0x018 AllocationSize : _LARGE_INTEGER 0xf76000
+0x020 FileSize : _LARGE_INTEGER 0xf750c7
+0x028 ValidDataLength : _LARGE_INTEGER 0xf750a0
+0x030 FastMutex : 0xffff960a
2f8ba468 _FAST_MUTEX +0x038 FilterContexts : _LIST_ENTRY [ 0xffffbb0c
b06a2c38 - 0xffffbb0cb06a2c38 ] +0x048 PushLock : _EX_PUSH_LOCK +0x050 FileContextSupportPointer : 0xffffbb0c
b06a2bf0 -> (null)+0x058 Oplock : (null)
+0x058 ReservedForRemote : (null)
+0x060 ReservedContext : 0xffff960a`2f8b0a08 Void
1: kd> dx -r1 ((ntkrnlmp!_FAST_MUTEX *)0xffff960a2f8ba468)
((ntkrnlmp!_FAST_MUTEX *)0xffff960a2f8ba468) : 0xffff960a2f8ba468 [Type: _FAST_MUTEX *]
[+0x000] Count : 1 [Type: long]
[+0x008] Owner : 0x0 [Type: void *]
[+0x010] Contention : 0x2f3b [Type: unsigned long]
[+0x018] Event [Type: _KEVENT]
[+0x030] OldIrql : 0x0 [Type: unsigned long]
Since this bugcheck happens nt!ExAcquireFastMutex+0xcf, I want to understand which Fast Mutex it is trying to access. I check FastMutex in AdvancedFCBHeader of LowerFileObj, which seems fine.
Any help further is really appreciated.
Thanks in advance!
Interesting
!object
is your friendThat was my assumption. You'll need to pick through the assember and see whather that was true (you know what the address is, so that makes it easier - you may even find evidence in the registers
I'd still be doing a
!pool <addre> 2
and a!verifier 80 <<addr>>
on the file object and the contexts...Thanks for further insights.
As per dump, bsod occurs while accessing rbx register
nt!ExAcquireFastMutex+ce
fffff800`d2878d7e f00fba3300 lock btr dword ptr [rbx],0
rax=ffffd001377b9428
rbx=00000000000001a7
rcx=00000000000001a7
rdx=ffffe0002105baa8
rsi=0000000000000000
rdi=0000000000000000
rip=fffff800d2878d7e
rsp=ffffd001377b93d0
rbp=0000000000000001
r8=0000000000000000
r9=0000000000000012
r10=ffffe00021188bc0
r11=ffffe000205461e0
r12=0000000000000000
r13=0000000000000001
r14=0000000000000000
r15=ffffe0001fa881e0
I did try to use !pool <adddres. 2 for register values, which gave me some valid information w.r.t. this scenario
kd> !pool ffffe0002105baa8 2
Pool page ffffe0002105baa8 region is Nonpaged pool
*ffffe0002105ba50 size: c0 previous size: 110 (Allocated) *FMfc
Pooltag FMfc : FLTMGR_FILE_OBJECT_CONTEXT structure, Binary : fltmgr.sys
0: kd> !pool ffffe00021188bc0 2
Pool page ffffe00021188bc0 region is Nonpaged pool
*ffffe00021188bb0 size: 30 previous size: 90 (Allocated) *FOCX
Pooltag FOCX : File System Run Time File Object Context structure, Binary : nt!fsrtl
0: kd> !pool ffffe000205461e0 2
Pool page ffffe000205461e0 region is Nonpaged pool
*ffffe000205461d0 size: 820 previous size: 70 (Allocated) *FMvo
Pooltag FMvo : FLT_VOLUME structure, Binary : fltmgr.sys
: kd> !pool ffffe0001fa881e0 2
Pool page ffffe0001fa881e0 region is Nonpaged pool
*ffffe0001fa881d0 size: 790 previous size: 30 (Allocated) *FMfr
Pooltag FMfr : FLT_FRAME structure, Binary : fltmgr.sys
where FLT_VOLUME and FLT_FRAME are valid pointers .
Now, I am struggling to relate information of FOCX : File System Run Time File Object Context and FLTMGR_FILE_OBJECT_CONTEXT structures to debug this case.
Any help will be really appreciated!
Thanks a lot!
{{!fltkd.help}} might provide some insight. But I'm struggling a bit.. We are certainly not dealing with a over-dereferenced object
In my driver , we are dereferencing lowerfileobj at single place in Close IRP, not sure how it is leading to over dereferencing of object.
I'd guess based on the name "FltpCleanupFileObjectContextForCleanup" that the fast mutex is part of FltMgr's file object context. Doing a uf on FltpPassThrough on RS5 it looks like you should only get here on an IRP_MJ_CLEANUP, which also makes sense based on the name, but makes no sense based on the stack.
If you do a !thread what IRPs are pending against the thread? Also, you should also enable Verifier on FltMgr.sys. If that doesn't show anything interesting you might also want to enable it on ntoskrnl.exe.
-scott
OSR
Thanks for your reply Scott!
I have listed output of !thread and 2 of IRPs.
0: kd> !thread
THREAD ffffe0001fd0e880 Cid 0004.0078 Teb: 0000000000000000 Win32Thread: 0000000000000000 RUNNING on processor 0
IRP List:
ffffe00021074250: (0006,0310) Flags: 00000404 Mdl: 00000000
ffffe000210e5cf0: (0006,0310) Flags: 00000404 Mdl: 00000000
Not impersonating
DeviceMap ffffc001b020c2a0
Owning Process ffffe0001fac9900 Image: System
Attached Process N/A Image: N/A
Wait Start TickCount 77263 Ticks: 0
Context Switch Count 10716 IdealProcessor: 0
UserTime 00:00:00.000
KernelTime 00:00:01.343
Win32 Start Address nt!ExpWorkerThread (0xfffff800d285e4e0)
Stack Init ffffd001377b9c90 Current ffffd001377b98b0
Base ffffd001377ba000 Limit ffffd001377b4000 Call 0000000000000000
Priority 13 BasePriority 13 PriorityDecrement 0 IoPriority 2 PagePriority 5
Child-SP RetAddr : Args to Child : Call Site
ffffd001
377b81c8 fffff800
d2959a05 : 000000000000007e ffffffff
c0000005 fffff800d2878d7e ffffd001
377b9198 : nt!KeBugCheckExffffd001
377b81d0 fffff800
d2932236 : 00000000016006b5 00000000
00000000 fffff6d800f5c468 ffffb001
eb88c000 : nt! ?? ::FNODOBFM::string'+0x1b15 ffffd001
377b8210 fffff800d294bd3d : 00000000
00000000 ffffd001377b83b0 ffffd001
377b9198 ffffd001377b83b0 : nt!_C_specific_handler+0x86 ffffd001
377b8280 fffff800d28859e9 : 00000000
00000001 fffff800d2803000 ffffd001
377b9100 ffffc00100000000 : nt!RtlpExecuteHandlerForException+0xd ffffd001
377b82b0 fffff800d28dca79 : ffffd001
377b9198 ffffd001377b8eb0 ffffd001
377b9198 0000000000000000 : nt!RtlDispatchException+0x1a5 ffffd001
377b8980 fffff800d2953502 : 00000000
000001a7 ffffd001377b90f8 ffffd001
746c464d 0000000000000081 : nt!KiDispatchException+0x18d ffffd001
377b9060 fffff800d295093c : 00000000
00000000 ffffe0001fc61000 ffffe000
21083c00 fffff800d286656f : nt!KiExceptionDispatch+0xc2 ffffd001
377b9240 fffff800d2878d7e : 00000000
00000001 0000000000000000 ffffe000
2105baa8 fffff8016a6cbe82 : nt!KiPageFault+0x3fc (TrapFrame @ ffffd001
377b9240)ffffd001
377b93d0 fffff801
6a6f59c0 : ffffe0002105baf8 ffffffff
ffffffff ffffe0002105baa8 00000000
00000012 : nt!ExAcquireFastMutex+0xceffffd001
377b9400 fffff801
6a6c7bfd : ffffd001377b9500 ffffd001
377b9480 ffffe00021074518 ffffe000
1fce7f20 : fltmgr!FltpCleanupFileObjectContextForCleanup+0x4cffffd001
377b9430 fffff801
6a6c70aa : ffffe00020546a50 00000000
00000000 ffffe00021074250 00000000
00000000 : fltmgr!FltpPassThrough+0x57effffd001
377b94e0 fffff800
d2bb990d : ffffe00021074250 00000000
00000000 0000000000000000 00000000
00000002 : fltmgr!FltpDispatch+0x9affffd001
377b9540 fffff800
d2d27750 : ffffe0001fce7f20 00000000
00000000 ffffe000656c6946 ffffe000
1fb33f20 : nt!IopCloseFile+0x12dffffd001
377b95d0 fffff800
d2bbd30c : 0000000000000000 ffffe000
1fce7f20 ffffe0001fb33f20 ffffe000
1fce7ef0 : nt! ?? ::NNGAKEGL::string'+0x10950 ffffd001
377b9650 fffff800d286967f : 00000000
00000000 ffffd001377b9720 ffffe000
1fce7f20 ffffe0001fa383d0 : nt!ObpRemoveObjectRoutine+0x64 ffffd001
377b96b0 fffff8016a72a770 : ffffe000
21952360 ffffe0001fc1d598 ffffe000
1fc1d598 ffffe0001fa383d0 : nt!ObfDereferenceObject+0x8f ffffd001
377b96f0 fffff8016a6c80ba : ffffe000
1fc1d598 ffffe0001fc1d4c0 ffffe000
205359a0 ffffe00021948828 : sfntpffd!PfmCloseCallback+0x1c8 [d:\pf-windows\fe-src\ds-client-fe\fe\kernel\windows\minifilter\src\close.c @ 240] ffffd001
377b9790 fffff8016a6c8d0c : ffffd001
377b99a0 fffff800d2946302 ffffe000
20546a00 0000000000000000 : fltmgr!FltpPerformPreCallbacks+0x31a ffffd001
377b98a0 fffff8016a6c7934 : ffffe000
1fc1d4c0 ffffd001377b9920 ffffe000
210e5fb8 fffff800d2ae3180 : fltmgr!FltpPassThroughInternal+0x8c ffffd001
377b98d0 fffff8016a6c70aa : ffffe000
20546a50 ffffe000210e5cf0 ffffe000
210e5cf0 ffffe0001fb33f20 : fltmgr!FltpPassThrough+0x2b5 ffffd001
377b9980 fffff800d2bb96cc : ffffe000
211a5cf0 ffffe0001fa53030 ffffe000
210e5cf0 0000000000000001 : fltmgr!FltpDispatch+0x9a ffffd001
377b99e0 fffff800d2bbd30c : ffffe000
211a5cc0 ffffe000211a5cf0 ffffe000
1fb33f20 ffffe000211a5cc0 : nt!IopDeleteFile+0x128 ffffd001
377b9a60 fffff800d2c6ccd9 : 00000000
00000000 0000000000000000 fffff800
d2aaee00 0000000000000000 : nt!ObpRemoveObjectRoutine+0x64 ffffd001
377b9ac0 fffff800d285eb7f : fffff800
d2c6cc20 ffffe0001fd0e880 fffff800
d2aaee00 0000000000000000 : nt!ObpProcessRemoveObjectQueue+0xb9 ffffd001
377b9b50 fffff800d28d6da2 : 00000000
00000000 ffffd0013b555180 00000000
00000080 ffffe0001fac9900 : nt!ExpWorkerThread+0x69f ffffd001
377b9c00 fffff800d294ac36 : ffffd001
3b555180 ffffe0001fd0e880 ffffe000
1fbd3880 0000000000000000 : nt!PspSystemThreadStartup+0x18a ffffd001
377b9c60 0000000000000000 : ffffd001
377ba000 ffffd001`377b4000 000: kd> !irp ffffe00021074250
Irp is active with 8 stacks 8 is current (= 0xffffe00021074518)
No Mdl: No System Buffer: Thread ffffe0001fd0e880: Irp stack trace.
cmd flg cl Device File Completion-Context
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
kd> !irp ffffe000210e5cf0
Irp is active with 8 stacks 8 is current (= 0xffffe000210e5fb8)
No Mdl: No System Buffer: Thread ffffe0001fd0e880: Irp stack trace.
cmd flg cl Device File Completion-Context
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
[N/A(0), N/A(0)]
0 0 00000000 00000000 00000000-00000000
Can you make a dump file available? Easier than trying to debug over the forum.
-scott
OSR
Where shall I upload dump file?
Something like Dropbox/OneDrive is usually the easiest.
-scott
OSR