File System Minifilter driver unload issue

Hello everyone,

I have File system mini filter driver in our product. Issue is “I am not able to stop my filter driver dynamically” and I am not able to find out the reason as well.

When I tried to run !verifier it is not showing any kind of leaks but when I run !fltkd.filter extension I found that reference count of my filter objects are 2, but I am not sure why one additional reference count is there.

Following is the output of !fltkd extension

0: kd> !fltkd.filter 87c09b68 0xf

FLT_FILTER: 87c09b68 “vfsmfd” “263410”
FLT_OBJECT: 87c09b68 [02000001] Filter DRAINING
RundownRef : 0xb855bb55
Could not read field “Count” of nt!_EX_RUNDOWN_WAIT_BLOCK from address: b855bb54
PointerCount : 0x00000002
PrimaryLink : [8b379a6c-8a4d9c6c]
Frame : 8b377000 “Frame 1”
Flags : [00000002] FilteringInitiated
DriverObject : 87bebe38
FilterLink : [8b379a6c-8a4d9c6c]
PreVolumeMount : 00000000 (null)
PostVolumeMount : 00000000 (null)
FilterUnload : ab587510 vfsmfd!FilterUnload
InstanceSetup : b7d84790 fltmgr!FltvInstanceSetup
InstanceQueryTeardown : b7d847b4 fltmgr!FltvInstanceQueryTeardown
InstanceTeardownStart : b7d847d2 fltmgr!FltvInstanceTeardownStart
InstanceTeardownComplete : b7d847f0 fltmgr!FltvInstanceTeardownComplete
ActiveOpens : (87c09c2c) mCount=0
Client Port List : (87c09c58) mCount=0
VerifierExtension : 87a07388
Operations : 87c09c88
OldDriverUnload : 00000000 (null)
SupportedContexts : (87c09bf8)
VolumeContexts : (87c09bf8)
ALLOCATE_CONTEXT_NODE: 882f2e70 [01] LookasideList
Filter : 87c09b68 “vfsmfd”
ContextCleanupCallback : ab5773a0 vfsmfd!VfsmfdContextCleanupCallback
Next : 00000000
ContextType : [0001] VolumeContext
Flags : [01] LookAsideListInited
Could not read field “NonPaged.L.Size” of FltMgr!_ALLOCATE_CONTEXT_LOOKASIDE from address: 882f2e70
InstanceContexts : (87c09bfc)
FileContexts : (87c09c00)
StreamContexts : (87c09c04)
StreamHandleContexts : (87c09c08)
ALLOCATE_CONTEXT_NODE: 882f2f38 [01] LookasideList
Filter : 87c09b68 “vfsmfd”
ContextCleanupCallback : ab5773a0 vfsmfd!VfsmfdContextCleanupCallback
Next : 00000000
ContextType : [0010] StreamHandleContext
Flags : [01] LookAsideListInited
Could not read field “NonPaged.L.Size” of FltMgr!_ALLOCATE_CONTEXT_LOOKASIDE from address: 882f2f38
TransactionContext : (87c09c0c)
InstanceList : (87c09b98)
FLT_INSTANCE: 87cce920 “Default” “263410”
Object usage/reference information:
Could not read offset of field “Count” from type FltMgr!_FLT_VERIFIER_EXTENSION

When I debug my driver code I found that call of FltUnregisterFilter() is hanging, I know this happens when there is some resource leak but I am not able to find it out which resource it is.

Same driver was working previously but suddenly after few new software deployed by IT on my laptop, this issue comes up.
I am not able to figure it out because of which application this is causing or this is really an issue in our driver.
I am very new in driver development and don’t have much internal knowledge, so if some one can help me out to find the problem then this would be really helpful.

Thanks a ton in advance and looking experts to suggest some way out.

Thanks,
Bishnu

Did you enable driver verifier for both FltMgr and your filter ? Once you do that and you try to unload FltMgr should tell you what you’re leaking.

Thanks,
Alex.

Hi Alex,
Thanks for your response.
I tried to enable verifier for both FltMgr and my filter but I am not able to figure it out what is getting leaked. Following is the output of verifier

0: kd> !verifier 0xf

Verify Level bb … enabled options are:
Special pool
Special irql
All pool allocations checked on unload
Io subsystem checking enabled
Deadlock detection enabled
DMA checking enabled

Summary of All Verifier Statistics

RaiseIrqls 0xfe7e8f
AcquireSpinLocks 0x1
Synch Executions 0x0
Trims 0x1

Pool Allocations Attempted 0x2f0564
Pool Allocations Succeeded 0x2f0564
Pool Allocations Succeeded SpecialPool 0x3a561
Pool Allocations With NO TAG 0x0
Pool Allocations Failed 0x0
Resource Allocations Failed Deliberately 0x0

Current paged pool allocations 0x1813 for 001D26F4 bytes
Peak paged pool allocations 0x1868 for 001D6AF0 bytes
Current nonpaged pool allocations 0xa2 for 00018458 bytes
Peak nonpaged pool allocations 0xbc for 0001BEBC bytes

Driver Verification List

Entry State NonPagedPool PagedPool Module

8b48df08 Loaded 00018458 001d26f4 fltmgr.sys

Current Pool Allocations 000000a2 00001813
Current Pool Bytes 00018458 001d26f4
Peak Pool Allocations 000000bb 00001868
Peak Pool Bytes 0001be64 001d6af0

PoolAddress SizeInBytes Tag CallersAddress
8b688f90 0x00000070 FMfi b7d87970
8b692f80 0x00000080 FMpl b7d84c43
8b377000 0x00002000 FMpl b7d84c54
8b33d000 0x00002000 FMfr b7d7c049
8b67eff8 0x00000004 FMus b7d7c933
8b690ff0 0x0000000c FMus b7d7c933
8b68ac00 0x00000400 FMpr b7d7bd5e
8b69a800 0x00000800 FMla b7d7ba9a
8b6b8df8 0x00000208 FMfl b7d834ec
8b6a8ce0 0x00000320 FMcr b7d77ac8
8b6b4ff0 0x0000000c FMus b7d7c933
8b34e000 0x00002000 FMfr b7d7c049
8b6c6ff0 0x0000000c FMus b7d7c933
8b6c0c00 0x00000400 FMpr b7d7bd5e
8b6c4800 0x00000800 FMla b7d7ba9a
8b6cce80 0x00000180 FMfl b7d834ec
e884f830 0x000000f8 FMfn b7d7fa9f
8b7c6c18 0x000003e4 FMvo b7d845a2
8b7c4f00 0x000000fc FMic b7d6c120
8b7daf00 0x000000fc FMic b7d6c120
8b7d0c18 0x000003e4 FMvo b7d845a2
8b7def28 0x000000d4 FMfn b7d7fa9f
8b772db0 0x0000024c FMis b7d7ccc0
8b776ff0 0x00000010 FMrr b7d84aca
8b784e00 0x00000200 FMrr b7d84b00
8b720db0 0x00000250 FMct b7d77d43
8b6d4f10 0x000000ec FMfn b7d7fa9f
8b79ce40 0x000001bc FMis b7d7ccc0
8b7a6ff0 0x00000010 FMrr b7d84aca
8b7a4e00 0x00000200 FMrr b7d84b00
8b7a0db0 0x00000250 FMct b7d77d43
e8953d78 0x00000128 FMfn b7d7fa9f
8b60ced8 0x00000124 FMfn b7d7fa9f
e8d34568 0x00000168 FMfn b7d7fa9f
e8cf94a8 0x000001e8 FMfn b7d7fa9f
8b676f10 0x000000ec FMfn b7d7fa9f
8b63eee0 0x00000120 FMfn b7d7fa9f
8b718e28 0x000001d4 FMic b7d6c120
8b754f08 0x000000f4 FMfn b7d7fa9f
8b6c2ee8 0x00000114 FMfn b7d7fa9f
8b6eae28 0x000001d4 FMic b7d6c120
e887e9f8 0x000000f8 FMfn b7d7fa9f
8b736f10 0x000000ec FMfn b7d7fa9f
8b608ec8 0x00000138 FMfn b7d7fa9f
8b63cee0 0x00000120 FMfn b7d7fa9f
8b6a0e28 0x000001d4 FMic b7d6c120
8b76ef18 0x000000e8 FMfn b7d7fa9f
8b640e28 0x000001d4 FMic b7d6c120
8b796e20 0x000001dc FMfn b7d7fa9f
8b69cf10 0x000000ec FMfn b7d7fa9f
8b6cef10 0x000000ec FMfn b7d7fa9f
8b79ef10 0x000000ec FMfn b7d7fa9f
8b6def10 0x000000ec FMfn b7d7fa9f
8b67af18 0x000000e8 FMfn b7d7fa9f
8b798f18 0x000000e4 FMfn b7d7fa9f
8b700f10 0x000000f0 FMfn b7d7fa9f
8b62ef18 0x000000e8 FMfn b7d7fa9f
8b72cf18 0x000000e8 FMfn b7d7fa9f
8b7b2f10 0x000000ec FMfn b7d7fa9f
8b7faf08 0x000000f4 FMfn b7d7fa9f
8b7e8f18 0x000000e8 FMfn b7d7fa9f
8b73cf10 0x000000f0 FMfn b7d7fa9f
8b6aef18 0x000000e8 FMfn b7d7fa9f
8b7d2f18 0x000000e8 FMfn b7d7fa9f



.

e8c7f008 0x00000150 FMfn b7d7fa9f
e8c96008 0x00000158 FMfn b7d7fa9f
e8bd0008 0x000000d8 FMfn b7d7fa9f
e8c1d930 0x00000170 FMfn b7d7fa9f
e8cdd850 0x00000170 FMfn b7d7fa9f
e8a3a880 0x00000150 FMfn b7d7fa9f

8b48de88 Loaded 00000000 00000000 vfsmfd.sys

Current Pool Allocations 00000000 00000000
Current Pool Bytes 00000000 00000000
Peak Pool Allocations 00000005 00000020
Peak Pool Bytes 00000430 00000910

-----------------------------------------------
Fault injection trace log
-----------------------------------------------

Driver Verifier didn’t inject any faults.

-----------------------------------------------
Track irql trace log
-----------------------------------------------
Error: cannot read nt!_VI_TRACK_IRQL type size.

Can you please help me if you see anything wrong in above trace. My filter name is “vfsmfd”.

Thanks,
Bishnu

> I tried to enable verifier for both FltMgr and my filter but I am not able

to figure it out what is getting leaked. Following is the output of
verifier

You need to be running Win7 (possibly Vista). Have you turned out debug
output from the kernel? Fltmgr will usually (after a short pause) tell you
all you need to know.

Hello Rod,

Thanks for your reply.
But we are facing this issue only with Win XP SP3, not with Win 7.

Just want to know does verifier has some additional capabilities on Vista or Win7 but not in Win XP ?

Thanks,
Bishnu

> Just want to know does verifier has some additional capabilities on Vista or Win7 but not in Win XP ?

Surely.

Verifier gets better in each next version. It is a good idea to test with Verifier on latest Windows.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

Hi Maxim,

Thanks for the information.
But right now I don’t have Win 7 test machine with all required software to test this issue on Win 7 and it is not reproducible on other Win 7 machine as well.

In my above verifier dump I see that there are whole lot of entries for ‘FMfn’ tag from address b7d7fa9f. According to documentation FMfn tag used by filter manager when DDK method FltGetFileNameInformation get called. I see in my code each FltGetFileNameInformation() is match with its corresponding FltReleaseFileNameINformation(), then also in verifier dump I see lot many these tags from same caller address b7d7fa9f.

Is there any way by which I can check more detail about caller from its address ?

Thanks,
Bishnu

FileNameInformation items are reference counted and from all I can infer
from observation, hang off either the Filtermanager’s own Stream or File
context (as part of the caching). So having a bunch of these hanging around
might just indicate that you are leaking a StreamContext. Or not.

I hear what you say about the difficulty of getting test kit together but
you are going to have to eventually (actually quite soon) and my guess is
that it might be quicker to build the Win7 test box that you are going to
need than try to debug this on XP… Indeed you are not going to find this
by debugging, only by desk checking the code.

Sorry…

Hi Rod,

I really appreciate your help.
I am facing this issue only in Win XP machines, but not in Win 7.
On Win 7 same driver is getting unload properly. We don’t have any code in our driver which execute OS version specific.
Can you please let me know what could be the difference in both the OS and since this issue is not there on Win 7 even verifier is not showing anything bad :frowning:

Thanks,
Bishnu

> I am facing this issue only in Win XP machines, but not in Win 7.

Oh nasty. I have never had that.

Can you please let me know what could be the difference in both the OS

Where do you start? The list is endless. Off the top of my head:

  • Manifold bugs have a been fixed.
  • Loads of new function (transactions, File Contexts, Link enumeration).
  • IoCreateFileSpeicfyDeviceHint does weird things when faced with reparse on
    disk points.
  • SR is no longer there but you are pretty much guaranteed to not be along
    on the filteramanager stack.
  • Filtermanager is always loaded at boot time.
  • Fewer (no?) Stack base file objects
  • A completely different approach to filtering the network
  • Several new filesystem types.
  • The list goes on.

But that’s just scratching the surface…

It’s possible that this is a result of a bug in XP SP3 that is being exposed by the new software IT installed on your laptop. The bug can cause a filter to not unload.

Does the IT-installed stuff include a new filter that sits below yours? If you can run without it does the problem go away?

Christian [MSFT]
This posting is provided “AS IS” with no warranties, and confers no rights.

It’s a new SW rolled out by IT, but the filter sits on top of ours.
Can you please elaborate the bug in XP SP3 and when does it get surfaced ?
Any workarounds for this ?

Thanks.

Hi Christian,

Yes in new IT patch new AV patch has been rolled out, which sits on top of our filter driver. But same version patch does not hang in Win 7.
We are facing this issue in Win XP only and following is the call stack

THREAD 8a6b1020 Cid 0004.0030 Teb: 00000000 Win32Thread: 00000000
WAIT: (Executive) KernelMode Non-Alertable
b851bb1c SynchronizationEvent
Not impersonating
DeviceMap e10010e8
Owning Process 0 Image:
Attached Process 8a6b4490 Image: System
Wait Start TickCount 10537 Ticks: 4670 (0:00:01:12.968)
Context Switch Count 4996
UserTime 00:00:00.000
KernelTime 00:00:01.265
Start Address nt!ExpWorkerThread (0x805386ea)
Stack Init b851c000 Current b851baa8 Base b851c000 Limit b8519000 Call 0
Priority 12 BasePriority 12 PriorityDecrement 0 DecrementCount 16
*** ERROR: Module load completed but symbols could not be loaded for vfsmfd.sys
ChildEBP RetAddr
b851bac0 80503864 nt!KiSwapContext+0x2f (FPO: [Uses EBP] [0,0,4])
b851bacc 804fb094 nt!KiSwapThread+0x8a (FPO: [0,0,0])
b851baf4 b7d4ec1b nt!KeWaitForSingleObject+0x1c2 (FPO: [Non-Fpo])
b851bb34 b7d40ff7 fltMgr!
FltpExWaitForRundownProtectionReleaseCacheAware+0x83 (FPO: [Non-Fpo])
b851bb40 b7d46a24 fltMgr!FltpDrainPendingCallbacksForInstance+0x17
(FPO: [Non-Fpo])
b851bb70 b7d4d816 fltMgr!FltpFreeInstance+0x74 (FPO: [Non-Fpo])
b851bb94 9df6f5bb fltMgr!FltUnregisterFilter+0x96 (FPO: [Non-Fpo])
WARNING: Stack unwind information not available. Following frames may be wrong.
b851bd3c b7d48fb3 vfsmfd+0x5bb
b851bd60 80581493 fltMgr!FltpMiniFilterDriverUnload+0xab (FPO: [Non-
Fpo])
b851bd7c 805387d9 nt!IopLoadUnloadDriver+0x19 (FPO: [Non-Fpo])
b851bdac 805cffbe nt!ExpWorkerThread+0xef (FPO: [Non-Fpo])
b851bddc 8054616e nt!PspSystemThreadStartup+0x34 (FPO: [Non-Fpo])
00000000 00000000 nt!KiThreadStartup+0x16

If you can tell me how should I confirm if this is an issue in Win XP SP3.

Thanks,
Bishnu

There’s another thread somewhere that is holding the rundown lock and not progressing. You need to find that thread.

Windows XP and Windows 7 are very different OS platforms, so it may just happen on XP because of the specific order of operations on that platform.

Tony
OSR

There isn’t another thread holding the rundown. This is a regression in the XP SP3 version of Filter Manager. It happens under these circumstances:

  1. You have two minifilters, A and B, with A above B in the same frame.
  2. Filter B has registered to filter reads/writes
  3. Filter A issues a read/write using FltRead/WriteFile()

When this happens a reference may leak. Since that reference never goes away the filter cannot unload. Unfortunately there is no workaround.

Because XP SP3 stopped getting mainstream support in April 2009 the only way to get a fix for this is if you have an extended support agreement for XP. If so you should contact your MSFT Technical Account Manager (TAM) to see about getting a fix.

Christian [MSFT]
This posting is provided “AS IS” with no warranties, and confers no rights.

Fortunately, nobody runs XP anymore. :wink:

Tony
OSR

Who told you that!?!? I suspect Seagate still installs XP as the “official” OS for their systems, and I KNOW that HGST’s (Western Digital and Hitachi Global Storage) IT still images new machines to XP. Of course us developers then put Ubuntu, Win 7, 8 or what ever it is that we need. Of course if IT has to work on it, guess what we get back?

Gary Little
H (952) 223-1349
C (952) 454-4629
xxxxx@comcast.net

On Oct 30, 2012, at 8:52 PM, Tony Mason wrote:

>


>
> Fortunately, nobody runs XP anymore. :wink:
>
> Tony
> OSR
>
>
> —
> NTFSD is sponsored by OSR
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Oh, I get it ? you were making a funny ? :slight_smile:

Gary Little
H (952) 223-1349
C (952) 454-4629
xxxxx@comcast.net

On Oct 30, 2012, at 8:58 PM, Gary Little wrote:

> Who told you that!?!? I suspect Seagate still installs XP as the “official” OS for their systems, and I KNOW that HGST’s (Western Digital and Hitachi Global Storage) IT still images new machines to XP. Of course us developers then put Ubuntu, Win 7, 8 or what ever it is that we need. Of course if IT has to work on it, guess what we get back?
>
> Gary Little
> H (952) 223-1349
> C (952) 454-4629
> xxxxx@comcast.net
>
>
> On Oct 30, 2012, at 8:52 PM, Tony Mason wrote:
>
>>


>>
>> Fortunately, nobody runs XP anymore. :wink:
>>
>> Tony
>> OSR
>>
>>
>> —
>> NTFSD is sponsored by OSR
>>
>> For our schedule of debugging and file system seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTFSD is sponsored by OSR
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer