Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results
Before Posting...
Please check out the Community Guidelines in the Announcements and Administration Category.More Info on Driver Writing and Debugging
The free OSR Learning Library has more than 50 articles on a wide variety of topics about writing and debugging device drivers and Minifilters. From introductory level to advanced. All the articles have been recently reviewed and updated, and are written using the clear and definitive style you've come to expect from OSR over the years.
Check out The OSR Learning Library at: https://www.osr.com/osr-learning-library/
WdFilter deadlocks with FltCreateSectionForDataScan
Hello,
I have a minifilter driver registered above wdfilter (Windows Defender). I noticed a strange deadlock in some processes which I am not able to reproduce at all in VMs. It happens randomly with MobaXTerm or powershell when starting from VSCode.
What I do is I open a section into each new unique file in post-create callback with FltCreateSectionForDataScan. After some digging and debugging live machines I managed to catch it live in a debugger. What seems to happen is that WdFilter has registered a precreate callback for IRP_MJ_ACQUIRE_FOR_SECTION_SYNCHRONIZATION and enqueues a file for a scan and basically blocks the operation with it. I can only guess what they do, since FltSendMessage is called - but im guessing they try to map the underlying file in UM. But when running !stacks 2 MyDriver i cannot see my driver blocking the same file again, or any other file from their UM service - so the file is not reentryng my driver and causing a deadlock from a different FO. The deadlock seems to resolve after several minutes since they presumably timeout the FltSendMessage. Is there anything I can do in this case? Will SectionNotificationCallback be called in this case? Im assuming I cant even close the section from the SectionNotificationCallback (if it was triggered) since IRP_MJ_ACQUIRE_FOR_SECTION_SYNCHRONIZATION preoperation has not finished. Also why would WdFilter catch kernel section creation and block until it gets a reply from UM?
Params used for the call
InitializeObjectAttributes(&oa, nullptr, OBJ_KERNEL_HANDLE, nullptr, nullptr);
FltCreateSectionForDataScan(instance, file, newCtx, SECTION_QUERY | SECTION_MAP_READ, &oa, nullptr, PAGE_READONLY, SEC_COMMIT | SEC_FILE, 0, &InternalSectionHandle, &SectionObject, &SectionSize);
Call stack attached:
00 ffffef03`47c4ab60 fffff806`2e6229c0 nt!KiSwapContext+0x76 01 ffffef03`47c4aca0 fffff806`2e621eef nt!KiSwapThread+0x500 02 ffffef03`47c4ad50 fffff806`2e6e30fe nt!KiCommitThreadWait+0x14f 03 ffffef03`47c4adf0 fffff806`2ea8e291 nt!KeWaitForMultipleObjects+0x2be 04 ffffef03`47c4af00 fffff806`2f74b4fd nt!FsRtlCancellableWaitForMultipleObjects+0x91 05 ffffef03`47c4af70 fffff806`30368496 FLTMGR!FltSendMessage+0x60d 06 ffffef03`47c4b0d0 fffff806`3036773f WdFilter+0x38496 07 ffffef03`47c4b240 fffff806`3036a8e9 WdFilter+0x3773f 08 ffffef03`47c4b310 fffff806`3038a8e7 WdFilter+0x3a8e9 09 ffffef03`47c4b3c0 fffff806`2f7464cb WdFilter+0x5a8e7 0a ffffef03`47c4b400 fffff806`2f742844 FLTMGR!FltpPerformPreCallbacksWorker+0x36b 0b ffffef03`47c4b520 fffff806`2e6a6097 FLTMGR!FltpPreFsFilterOperation+0x184 0c ffffef03`47c4b5d0 fffff806`2ea86591 nt!FsFilterPerformCallbacks+0xe7 0d ffffef03`47c4b640 fffff806`2ea861fb nt!FsRtlAcquireFileExclusiveCommon+0x121 0e ffffef03`47c4b930 fffff806`2e75c282 nt!FsRtlAcquireToCreateMappedSection+0x5b 0f ffffef03`47c4b9b0 fffff806`2f77d2f4 nt!FsRtlCreateSectionForDataScan+0xb2 10 ffffef03`47c4ba50 fffff806`2f79edfd FLTMGR!FltCreateSectionForDataScan+0xf4 11 ffffef03`47c4bab0 fffff802`40b0cb50 FLTMGR!FltvCreateSectionForDataScan+0xad 12 ffffef03`47c4bb20 fffff802`40b0e664 MyDriver!MyDriver::MapInfo::MapIfNotMapped+0x1b4 [File @ 3168] 13 ffffef03`47c4bc30 fffff802`40b083ce MyDriver!MyDriver::Scan+0x2e0 [File @ 3490] 14 ffffef03`47c4bf10 fffff806`2f79ccf4 MyDriver!MyDriverPostCreate+0x6fe [File @ 2417] 15 ffffef03`47c4c0a0 fffff806`2f745b86 FLTMGR!FltvPostOperation+0xb4 16 ffffef03`47c4c130 fffff806`2f74545b FLTMGR!FltpPerformPostCallbacksWorker+0x346 17 ffffef03`47c4c200 fffff806`2f7471a2 FLTMGR!FltpPassThroughCompletionWorker+0xfb 18 ffffef03`47c4c2a0 fffff806`2f779f54 FLTMGR!FltpLegacyProcessingAfterPreCallbacksCompleted+0x322 19 ffffef03`47c4c310 fffff806`2e775d97 FLTMGR!FltpCreate+0x324 1a ffffef03`47c4c3c0 fffff806`2edcdf2a nt!IopfCallDriver+0x53 1b ffffef03`47c4c400 fffff806`2e826a65 nt!IovCallDriver+0x266 1c ffffef03`47c4c440 fffff806`2e613944 nt!IofCallDriver+0x20f735 1d ffffef03`47c4c480 fffff806`2ea0558b nt!IoCallDriverWithTracing+0x34 1e ffffef03`47c4c4d0 fffff806`2ea1b01e nt!IopParseDevice+0x11bb 1f ffffef03`47c4c640 fffff806`2ea12cea nt!ObpLookupObjectName+0x3fe 20 ffffef03`47c4c810 fffff806`2ea030ac nt!ObOpenObjectByNameEx+0x1fa 21 ffffef03`47c4c940 fffff806`2ea01d69 nt!IopCreateFile+0x132c 22 ffffef03`47c4ca00 fffff806`2e8154f8 nt!NtCreateFile+0x79 23 ffffef03`47c4ca90 00007ffe`9e04db04 nt!KiSystemServiceCopyEnd+0x28 24 00000000`0009e548 00007ffe`9d7a73f6 ntdll!NtCreateFile+0x14 25 00000000`0009e550 00007ffe`9d7a901a wow64!whNtCreateFile+0x106 26 00000000`0009e630 00000000`772a17c3 wow64!Wow64SystemServiceEx+0x15a 27 00000000`0009eef0 00000000`772a11b9 wow64cpu!ServiceNoTurbo+0xb 28 00000000`0009efa0 00007ffe`9d7a38c9 wow64cpu!BTCpuSimulate+0x9 29 00000000`0009efe0 00007ffe`9d7a32bd wow64!RunCpuSimulation+0xd 2a 00000000`0009f010 00007ffe`9e08378f wow64!Wow64LdrpInitialize+0x12d 2b 00000000`0009f2c0 00007ffe`9e024deb ntdll!LdrpInitializeProcess+0x18cf 2c 00000000`0009f6e0 00007ffe`9e024c73 ntdll!LdrpInitialize+0x15f 2d 00000000`0009f780 00007ffe`9e024c1e ntdll!LdrpInitialize+0x3b 2e 00000000`0009f7b0 00000000`00000000 ntdll!LdrInitializeThunk+0xe
Howdy, Stranger!
| Upcoming OSR Seminars | ||
|---|---|---|
| OSR has suspended in-person seminars due to the Covid-19 outbreak. But, don't miss your training! Attend via the internet instead! | ||
| Internals & Software Drivers | 19-23 June 2023 | Live, Online |
| Writing WDF Drivers | 10-14 July 2023 | Live, Online |
| Kernel Debugging | 16-20 October 2023 | Live, Online |
| Developing Minifilters | 13-17 November 2023 | Live, Online |
Comments
A small update:
Using FsRtlCreateSectionForDataScan resolves this issue, this confirm my suspision that they are trying to scan the file from precallback of IRP_MJ_ACQUIRE_FOR_SECTION_SYNCHRONIZATION . I have also tested the SectionNotificationCallback but it is sadly never called. But this does not feel right. Is this something I should report to MSFT as a bug ? I might be wrong, but this is something wdfilter should not be doing.
Edit: Is there any place where I can report the bug @ MSFT? I couldnt find any email or submission page for Defender related issues.
It's very common for A/V to trigger scans from this callback (regardless of whether it's from UM or KM).
I'm not sure about your hang but it's curious...What's MpMpEng.exe doing when this happens?
-scott
OSR
I deadlocked it again
Call stack for the deadlocked thread is pretty much same as above.
Trough windbg and my own driver symbols I determined the file in question was "\Users\Sovak\AppData\Roaming\MobaXterm\slash\usr\share\X11\xkb\symbols\mokeypad"
I ran the command
!process 0 1f MsMpEng.exeas you suggested. I can provide the full command output, but I went trough each thread which was currently executing FltCreateSectionForDataScan and found one that matches my file.THREAD ffffa80ed1ff0080 Cid 1254.1ea8 Teb: 0000004315040000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable ffffa80ed4dde710 SynchronizationEvent IRP List: ffffa80ed493c930: (0006,0118) Flags: 00060000 Mdl: 00000000 Not impersonating DeviceMap ffffcd8a8f8370c0 Owning Process ffffa80ed067d080 Image: MsMpEng.exe Attached Process N/A Image: N/A Wait Start TickCount 9621 Ticks: 449 (0:00:00:07.015) Context Switch Count 5488 IdealProcessor: 6 UserTime 00:00:02.937 KernelTime 00:00:00.265 Win32 Start Address mprtp!MpPluginGetDevVolumesState (0x00007ffa5d3acff0) Stack Init ffff870295861c90 Current ffff8702958610e0 Base ffff870295862000 Limit ffff87029585c000 Call 0000000000000000 Priority 11 BasePriority 11 PriorityDecrement 0 IoPriority 2 PagePriority 2 Scheduling Group: ffffa80ed1fd9080 Child-SP RetAddr Call Site ffff8702`95861120 fffff804`4aa1c9c0 nt!KiSwapContext+0x76 ffff8702`95861260 fffff804`4aa1beef nt!KiSwapThread+0x500 ffff8702`95861310 fffff804`4aa1b793 nt!KiCommitThreadWait+0x14f ffff8702`958613b0 fffff804`4c77ce9f nt!KeWaitForSingleObject+0x233 ffff8702`958614a0 fffff804`4c77d25a FLTMGR!FltpRundownStreamForSectionConflict+0x5f ffff8702`958614e0 fffff804`4d2b987c FLTMGR!FltCreateSectionForDataScan+0x5a ffff8702`95861540 fffff804`4d2a62eb WdFilter+0x3987c ffff8702`95861610 fffff804`4c74bc8a WdFilter+0x262eb ffff8702`958616d0 fffff804`4c781819 FLTMGR!FltpFilterMessage+0xda ffff8702`95861730 fffff804`4c744a80 FLTMGR!FltpMsgDispatch+0x179 ffff8702`958617a0 fffff804`4aa11385 FLTMGR!FltpDispatch+0xe0 ffff8702`95861800 fffff804`4ae10b1c nt!IofCallDriver+0x55 ffff8702`95861840 fffff804`4ae10771 nt!IopSynchronousServiceTail+0x34c ffff8702`958618e0 fffff804`4ae0fae6 nt!IopXxxControlFile+0xc71 ffff8702`95861a20 fffff804`4ac0f4f8 nt!NtDeviceIoControlFile+0x56 ffff8702`95861a90 00007ffa`7e2ed144 nt!KiSystemServiceCopyEnd+0x28 (TrapFrame @ ffff8702`95861b00) 00000043`1687d6a8 00007ffa`6aec2b98 ntdll!NtDeviceIoControlFile+0x14 00000043`1687d6b0 00007ffa`6aec2451 FLTLIB!FilterpDeviceIoControl+0x13c 00000043`1687d730 00007ffa`5d22b362 FLTLIB!FilterSendMessage+0x31 00000043`1687d780 00007ffa`5d296056 mprtp!MpPluginGetDevVolumesState+0x2ee32 00000043`1687d7c0 00007ffa`5d296562 mprtp!MpPluginGetDevVolumesState+0x99b26 00000043`1687dc80 00007ffa`5d28d53c mprtp!MpPluginGetDevVolumesState+0x9a032 00000043`1687dcf0 00007ffa`5c3fbb05 mprtp!MpPluginGetDevVolumesState+0x9100c 00000043`1687dd50 00007ffa`5c3b4104 mpengine+0x37bb05 00000043`1687dd90 00007ffa`5c32d0d9 mpengine+0x334104 00000043`1687ddf0 00000000`00000001 mpengine+0x2ad0d9 00000043`1687ddf8 00000000`00000000 0x1And fileinfo
Seems like they are trying to map the file while I am trying to map it as well. They also dont seem to be using SectionNotificationCallback, so theres no way for them to know theres a conflict (if its even possible to know, since my function never returned). Funny thing is, it always deadlocks on the same file for MobaXterm.
That really does look like a bug. Poking around in the debugger it appears that FltMgr is trying to protect against simultaneous calls to FltCreateSectionForDataScan. This is related but distinct from the SectionConflict callback, which is trying to stop writes from failing while a data scan section exists. Hard to blame Defender as it appears to be doing legitimate "A/V stuff",,,
I strongly suspect that you're the first person to try and call this API above WdFilter. Congratulations 😂 Few things:
-scott
OSR
1) Top of A/V range. What I am working on is behavioral ransomware protection. I get some basic info out of every unique file opened as well as PE structure reading and checking for anomalies. I decided to go for the A/V range instead of monitoring because it can also block files.
2) I will certainly use the FsRtlCreateSectionForDataScan version to also prevent possible compatibility issues with other AV products.
3) 22H2 W10 and 22H2 W11 -> the only machines where I tried to test it.
Should I try to inform MSFT/Defender team that something like this can happen? Sooner or later someone is going to do something similar and if it starts deadlocking infrequently random processes its going to be pain to debug/find the issue in prod.
Absolutely.
Best if you can put together a very simple filter that demonstrates the issue. Are you going to be involved with Plugfest? That's usually the best time to track these down
-scott
OSR
Sure, will do. I believe I have missed the registration windows for the Plugfest.