Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Home NTFSD

Before Posting...

Please check out the Community Guidelines in the Announcements and Administration Category.

More Info on Driver Writing and Debugging


The free OSR Learning Library has more than 50 articles on a wide variety of topics about writing and debugging device drivers and Minifilters. From introductory level to advanced. All the articles have been recently reviewed and updated, and are written using the clear and definitive style you've come to expect from OSR over the years.


Check out The OSR Learning Library at: https://www.osr.com/osr-learning-library/


WdFilter deadlocks with FltCreateSectionForDataScan

sovaksovak Member Posts: 17

Hello,

I have a minifilter driver registered above wdfilter (Windows Defender). I noticed a strange deadlock in some processes which I am not able to reproduce at all in VMs. It happens randomly with MobaXTerm or powershell when starting from VSCode.

What I do is I open a section into each new unique file in post-create callback with FltCreateSectionForDataScan. After some digging and debugging live machines I managed to catch it live in a debugger. What seems to happen is that WdFilter has registered a precreate callback for IRP_MJ_ACQUIRE_FOR_SECTION_SYNCHRONIZATION and enqueues a file for a scan and basically blocks the operation with it. I can only guess what they do, since FltSendMessage is called - but im guessing they try to map the underlying file in UM. But when running !stacks 2 MyDriver i cannot see my driver blocking the same file again, or any other file from their UM service - so the file is not reentryng my driver and causing a deadlock from a different FO. The deadlock seems to resolve after several minutes since they presumably timeout the FltSendMessage. Is there anything I can do in this case? Will SectionNotificationCallback be called in this case? Im assuming I cant even close the section from the SectionNotificationCallback (if it was triggered) since IRP_MJ_ACQUIRE_FOR_SECTION_SYNCHRONIZATION preoperation has not finished. Also why would WdFilter catch kernel section creation and block until it gets a reply from UM?

Params used for the call

      InitializeObjectAttributes(&oa, nullptr, OBJ_KERNEL_HANDLE, nullptr, nullptr);
      FltCreateSectionForDataScan(instance, file, newCtx, SECTION_QUERY | SECTION_MAP_READ, &oa, nullptr, PAGE_READONLY, SEC_COMMIT | SEC_FILE, 0, &InternalSectionHandle, &SectionObject, &SectionSize);

Call stack attached:

00 ffffef03`47c4ab60 fffff806`2e6229c0     nt!KiSwapContext+0x76
01 ffffef03`47c4aca0 fffff806`2e621eef     nt!KiSwapThread+0x500
02 ffffef03`47c4ad50 fffff806`2e6e30fe     nt!KiCommitThreadWait+0x14f
03 ffffef03`47c4adf0 fffff806`2ea8e291     nt!KeWaitForMultipleObjects+0x2be
04 ffffef03`47c4af00 fffff806`2f74b4fd     nt!FsRtlCancellableWaitForMultipleObjects+0x91
05 ffffef03`47c4af70 fffff806`30368496     FLTMGR!FltSendMessage+0x60d
06 ffffef03`47c4b0d0 fffff806`3036773f     WdFilter+0x38496
07 ffffef03`47c4b240 fffff806`3036a8e9     WdFilter+0x3773f
08 ffffef03`47c4b310 fffff806`3038a8e7     WdFilter+0x3a8e9
09 ffffef03`47c4b3c0 fffff806`2f7464cb     WdFilter+0x5a8e7
0a ffffef03`47c4b400 fffff806`2f742844     FLTMGR!FltpPerformPreCallbacksWorker+0x36b
0b ffffef03`47c4b520 fffff806`2e6a6097     FLTMGR!FltpPreFsFilterOperation+0x184
0c ffffef03`47c4b5d0 fffff806`2ea86591     nt!FsFilterPerformCallbacks+0xe7
0d ffffef03`47c4b640 fffff806`2ea861fb     nt!FsRtlAcquireFileExclusiveCommon+0x121
0e ffffef03`47c4b930 fffff806`2e75c282     nt!FsRtlAcquireToCreateMappedSection+0x5b
0f ffffef03`47c4b9b0 fffff806`2f77d2f4     nt!FsRtlCreateSectionForDataScan+0xb2
10 ffffef03`47c4ba50 fffff806`2f79edfd     FLTMGR!FltCreateSectionForDataScan+0xf4
11 ffffef03`47c4bab0 fffff802`40b0cb50     FLTMGR!FltvCreateSectionForDataScan+0xad
12 ffffef03`47c4bb20 fffff802`40b0e664     MyDriver!MyDriver::MapInfo::MapIfNotMapped+0x1b4 [File @ 3168] 
13 ffffef03`47c4bc30 fffff802`40b083ce     MyDriver!MyDriver::Scan+0x2e0 [File @ 3490] 
14 ffffef03`47c4bf10 fffff806`2f79ccf4     MyDriver!MyDriverPostCreate+0x6fe [File @ 2417] 
15 ffffef03`47c4c0a0 fffff806`2f745b86     FLTMGR!FltvPostOperation+0xb4
16 ffffef03`47c4c130 fffff806`2f74545b     FLTMGR!FltpPerformPostCallbacksWorker+0x346
17 ffffef03`47c4c200 fffff806`2f7471a2     FLTMGR!FltpPassThroughCompletionWorker+0xfb
18 ffffef03`47c4c2a0 fffff806`2f779f54     FLTMGR!FltpLegacyProcessingAfterPreCallbacksCompleted+0x322
19 ffffef03`47c4c310 fffff806`2e775d97     FLTMGR!FltpCreate+0x324
1a ffffef03`47c4c3c0 fffff806`2edcdf2a     nt!IopfCallDriver+0x53
1b ffffef03`47c4c400 fffff806`2e826a65     nt!IovCallDriver+0x266
1c ffffef03`47c4c440 fffff806`2e613944     nt!IofCallDriver+0x20f735
1d ffffef03`47c4c480 fffff806`2ea0558b     nt!IoCallDriverWithTracing+0x34
1e ffffef03`47c4c4d0 fffff806`2ea1b01e     nt!IopParseDevice+0x11bb
1f ffffef03`47c4c640 fffff806`2ea12cea     nt!ObpLookupObjectName+0x3fe
20 ffffef03`47c4c810 fffff806`2ea030ac     nt!ObOpenObjectByNameEx+0x1fa
21 ffffef03`47c4c940 fffff806`2ea01d69     nt!IopCreateFile+0x132c
22 ffffef03`47c4ca00 fffff806`2e8154f8     nt!NtCreateFile+0x79
23 ffffef03`47c4ca90 00007ffe`9e04db04     nt!KiSystemServiceCopyEnd+0x28
24 00000000`0009e548 00007ffe`9d7a73f6     ntdll!NtCreateFile+0x14
25 00000000`0009e550 00007ffe`9d7a901a     wow64!whNtCreateFile+0x106
26 00000000`0009e630 00000000`772a17c3     wow64!Wow64SystemServiceEx+0x15a
27 00000000`0009eef0 00000000`772a11b9     wow64cpu!ServiceNoTurbo+0xb
28 00000000`0009efa0 00007ffe`9d7a38c9     wow64cpu!BTCpuSimulate+0x9
29 00000000`0009efe0 00007ffe`9d7a32bd     wow64!RunCpuSimulation+0xd
2a 00000000`0009f010 00007ffe`9e08378f     wow64!Wow64LdrpInitialize+0x12d
2b 00000000`0009f2c0 00007ffe`9e024deb     ntdll!LdrpInitializeProcess+0x18cf
2c 00000000`0009f6e0 00007ffe`9e024c73     ntdll!LdrpInitialize+0x15f
2d 00000000`0009f780 00007ffe`9e024c1e     ntdll!LdrpInitialize+0x3b
2e 00000000`0009f7b0 00000000`00000000     ntdll!LdrInitializeThunk+0xe

Comments

  • sovaksovak Member Posts: 17
    edited May 2023

    A small update:

    Using FsRtlCreateSectionForDataScan resolves this issue, this confirm my suspision that they are trying to scan the file from precallback of IRP_MJ_ACQUIRE_FOR_SECTION_SYNCHRONIZATION . I have also tested the SectionNotificationCallback but it is sadly never called. But this does not feel right. Is this something I should report to MSFT as a bug ? I might be wrong, but this is something wdfilter should not be doing.

    Edit: Is there any place where I can report the bug @ MSFT? I couldnt find any email or submission page for Defender related issues.

    Post edited by sovak on
  • Scott_Noone_(OSR)Scott_Noone_(OSR) Administrator Posts: 3,678

    Also why would WdFilter catch kernel section creation and block until it gets a reply from UM?

    It's very common for A/V to trigger scans from this callback (regardless of whether it's from UM or KM).

    I'm not sure about your hang but it's curious...What's MpMpEng.exe doing when this happens?

    !process 0 1f MsMpEng.exe
    

    -scott
    OSR

  • sovaksovak Member Posts: 17

    I deadlocked it again

    Call stack for the deadlocked thread is pretty much same as above.

    Trough windbg and my own driver symbols I determined the file in question was "\Users\Sovak\AppData\Roaming\MobaXterm\slash\usr\share\X11\xkb\symbols\mokeypad"

    I ran the command !process 0 1f MsMpEng.exe as you suggested. I can provide the full command output, but I went trough each thread which was currently executing FltCreateSectionForDataScan and found one that matches my file.

    THREAD ffffa80ed1ff0080  Cid 1254.1ea8  Teb: 0000004315040000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
                ffffa80ed4dde710  SynchronizationEvent
            IRP List:
                ffffa80ed493c930: (0006,0118) Flags: 00060000  Mdl: 00000000
            Not impersonating
            DeviceMap                 ffffcd8a8f8370c0
            Owning Process            ffffa80ed067d080       Image:         MsMpEng.exe
            Attached Process          N/A            Image:         N/A
            Wait Start TickCount      9621           Ticks: 449 (0:00:00:07.015)
            Context Switch Count      5488           IdealProcessor: 6             
            UserTime                  00:00:02.937
            KernelTime                00:00:00.265
            Win32 Start Address mprtp!MpPluginGetDevVolumesState (0x00007ffa5d3acff0)
            Stack Init ffff870295861c90 Current ffff8702958610e0
            Base ffff870295862000 Limit ffff87029585c000 Call 0000000000000000
            Priority 11 BasePriority 11 PriorityDecrement 0 IoPriority 2 PagePriority 2
            Scheduling Group: ffffa80ed1fd9080
            Child-SP          RetAddr               Call Site
            ffff8702`95861120 fffff804`4aa1c9c0     nt!KiSwapContext+0x76
            ffff8702`95861260 fffff804`4aa1beef     nt!KiSwapThread+0x500
            ffff8702`95861310 fffff804`4aa1b793     nt!KiCommitThreadWait+0x14f
            ffff8702`958613b0 fffff804`4c77ce9f     nt!KeWaitForSingleObject+0x233
            ffff8702`958614a0 fffff804`4c77d25a     FLTMGR!FltpRundownStreamForSectionConflict+0x5f
            ffff8702`958614e0 fffff804`4d2b987c     FLTMGR!FltCreateSectionForDataScan+0x5a
            ffff8702`95861540 fffff804`4d2a62eb     WdFilter+0x3987c
            ffff8702`95861610 fffff804`4c74bc8a     WdFilter+0x262eb
            ffff8702`958616d0 fffff804`4c781819     FLTMGR!FltpFilterMessage+0xda
            ffff8702`95861730 fffff804`4c744a80     FLTMGR!FltpMsgDispatch+0x179
            ffff8702`958617a0 fffff804`4aa11385     FLTMGR!FltpDispatch+0xe0
            ffff8702`95861800 fffff804`4ae10b1c     nt!IofCallDriver+0x55
            ffff8702`95861840 fffff804`4ae10771     nt!IopSynchronousServiceTail+0x34c
            ffff8702`958618e0 fffff804`4ae0fae6     nt!IopXxxControlFile+0xc71
            ffff8702`95861a20 fffff804`4ac0f4f8     nt!NtDeviceIoControlFile+0x56
            ffff8702`95861a90 00007ffa`7e2ed144     nt!KiSystemServiceCopyEnd+0x28 (TrapFrame @ ffff8702`95861b00)
            00000043`1687d6a8 00007ffa`6aec2b98     ntdll!NtDeviceIoControlFile+0x14
            00000043`1687d6b0 00007ffa`6aec2451     FLTLIB!FilterpDeviceIoControl+0x13c
            00000043`1687d730 00007ffa`5d22b362     FLTLIB!FilterSendMessage+0x31
            00000043`1687d780 00007ffa`5d296056     mprtp!MpPluginGetDevVolumesState+0x2ee32
            00000043`1687d7c0 00007ffa`5d296562     mprtp!MpPluginGetDevVolumesState+0x99b26
            00000043`1687dc80 00007ffa`5d28d53c     mprtp!MpPluginGetDevVolumesState+0x9a032
            00000043`1687dcf0 00007ffa`5c3fbb05     mprtp!MpPluginGetDevVolumesState+0x9100c
            00000043`1687dd50 00007ffa`5c3b4104     mpengine+0x37bb05
            00000043`1687dd90 00007ffa`5c32d0d9     mpengine+0x334104
            00000043`1687ddf0 00000000`00000001     mpengine+0x2ad0d9
            00000043`1687ddf8 00000000`00000000     0x1
    

    And fileinfo

    0: kd> .thread /p /r ffffa80ed1ff0080
    Implicit thread is now ffffa80e`d1ff0080
    Implicit process is now ffffa80e`d067d080
    .cache forcedecodeuser done
    Loading User Symbols....
    
    
    0: kd> .frame /r 5
    05 ffff8702`958614e0 fffff804`4d2b987c     FLTMGR!FltCreateSectionForDataScan+0x5a
    rax=0000000000000000 rbx=0000000000000000 rcx=0000000000000000
    rdx=0000000000000000 rsi=ffffcd8a9fae9370 rdi=ffffa80ecabc5010
    rip=fffff8044c77d25a rsp=ffff8702958614e0 rbp=ffffa80ed5e75370
     r8=0000000000000000  r9=0000000000000000 r10=0000000000000000
    r11=0000000000000000 r12=0000000000000000 r13=ffffcd8a9b237760
    r14=0000000000000005 r15=ffffa80ed5e75370
    iopl=0         nv up di pl nz na pe nc
    cs=0000  ss=0000  ds=0000  es=0000  fs=0000  gs=0000             efl=00000000
    FLTMGR!FltCreateSectionForDataScan+0x5a:
    fffff804`4c77d25a 8bd8            mov     ebx,eax
    0: kd> .thread /p /r ffffa80ed5e75370
    Invalid thread @ ffffa80e`d5e75370 type - context unchanged.
    
    
    0: kd> dt FILE_OBJECT ffffa80ed5e75370
    Wdf01000!FILE_OBJECT
       +0x000 Type             : 0n5
       +0x002 Size             : 0n216
       +0x008 DeviceObject     : 0xffffa80e`c92e98f0 _DEVICE_OBJECT
       +0x010 Vpb              : 0xffffa80e`c93dfcd0 _VPB
       +0x018 FsContext        : 0xffffcd8a`9f828170 Void
       +0x020 FsContext2       : 0xffffcd8a`9f8283e0 Void
       +0x028 SectionObjectPointer : 0xffffa80e`d51a1c98 _SECTION_OBJECT_POINTERS
       +0x030 PrivateCacheMap  : (null) 
       +0x038 FinalStatus      : 0n0
       +0x040 RelatedFileObject : (null) 
       +0x048 LockOperation    : 0 ''
       +0x049 DeletePending    : 0 ''
       +0x04a ReadAccess       : 0x1 ''
       +0x04b WriteAccess      : 0 ''
       +0x04c DeleteAccess     : 0 ''
       +0x04d SharedRead       : 0x1 ''
       +0x04e SharedWrite      : 0 ''
       +0x04f SharedDelete     : 0x1 ''
       +0x050 Flags            : 0x60
       +0x058 FileName         : _UNICODE_STRING "\Users\Sovak\AppData\Roaming\MobaXterm\slash\usr\share\X11\xkb\symbols\mokeypad"
       +0x068 CurrentByteOffset : _LARGE_INTEGER 0x0
       +0x070 Waiters          : 0
       +0x074 Busy             : 0
       +0x078 LastLock         : (null) 
       +0x080 Lock             : _KEVENT
       +0x098 Event            : _KEVENT
       +0x0b0 CompletionContext : (null) 
       +0x0b8 IrpListLock      : 0
       +0x0c0 IrpList          : _LIST_ENTRY [ 0xffffa80e`d5e75430 - 0xffffa80e`d5e75430 ]
       +0x0d0 FileObjectExtension : 0xffffa80e`d4ffcf50 Void
    

    Seems like they are trying to map the file while I am trying to map it as well. They also dont seem to be using SectionNotificationCallback, so theres no way for them to know theres a conflict (if its even possible to know, since my function never returned). Funny thing is, it always deadlocks on the same file for MobaXterm.

  • Scott_Noone_(OSR)Scott_Noone_(OSR) Administrator Posts: 3,678

    That really does look like a bug. Poking around in the debugger it appears that FltMgr is trying to protect against simultaneous calls to FltCreateSectionForDataScan. This is related but distinct from the SectionConflict callback, which is trying to stop writes from failing while a data scan section exists. Hard to blame Defender as it appears to be doing legitimate "A/V stuff",,,

    I strongly suspect that you're the first person to try and call this API above WdFilter. Congratulations 😂 Few things:

    1. What altitude range are you in? And does that range accurately reflect what you're trying to do? There's no point in bothering with this further if you're at the wrong altitude and should be below A/V
    2. Calling FsrtlCreateSectionForDataScan directly bypassed this internal FltMgr synchronization so it makes sense that it would work...It's also not the worst thing in the world as a workaround
    3. What version of Windows is this?

    -scott
    OSR

  • sovaksovak Member Posts: 17

    1) Top of A/V range. What I am working on is behavioral ransomware protection. I get some basic info out of every unique file opened as well as PE structure reading and checking for anomalies. I decided to go for the A/V range instead of monitoring because it can also block files.

    2) I will certainly use the FsRtlCreateSectionForDataScan version to also prevent possible compatibility issues with other AV products.

    3) 22H2 W10 and 22H2 W11 -> the only machines where I tried to test it.

    Should I try to inform MSFT/Defender team that something like this can happen? Sooner or later someone is going to do something similar and if it starts deadlocking infrequently random processes its going to be pain to debug/find the issue in prod.

  • Scott_Noone_(OSR)Scott_Noone_(OSR) Administrator Posts: 3,678

    @sovak said:
    Should I try to inform MSFT/Defender team that something like this can happen? Sooner or later someone is going to do something similar and if it starts deadlocking infrequently random processes its going to be pain to debug/find the issue in prod.

    Absolutely.

    Best if you can put together a very simple filter that demonstrates the issue. Are you going to be involved with Plugfest? That's usually the best time to track these down

    -scott
    OSR

  • sovaksovak Member Posts: 17

    Sure, will do. I believe I have missed the registration windows for the Plugfest.

Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. Sign in or register to get started.

Upcoming OSR Seminars
OSR has suspended in-person seminars due to the Covid-19 outbreak. But, don't miss your training! Attend via the internet instead!
Kernel Debugging 13-17 May 2024 Live, Online
Developing Minifilters 1-5 Apr 2024 Live, Online
Internals & Software Drivers 11-15 Mar 2024 Live, Online
Writing WDF Drivers 20-24 May 2024 Live, Online