Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Before Posting...
Please check out the Community Guidelines in the Announcements and Administration Category.

More Info on Driver Writing and Debugging


The free OSR Learning Library has more than 50 articles on a wide variety of topics about writing and debugging device drivers and Minifilters. From introductory level to advanced. All the articles have been recently reviewed and updated, and are written using the clear and definitive style you've come to expect from OSR over the years.


Check out The OSR Learning Library at: https://www.osr.com/osr-learning-library/


Call to RegUnLoadKey hangs VMware Virtual Machine

OSR_Community_UserOSR_Community_User Member Posts: 110,217
I have an issue with a windows desktop application I develop which is causing two VMs running Windows 7 Sp1 and windows 10 to hang both on a VMware ESXi.
After searching the dump file of the VM I found out that the problem is happening because the Registry is exclusively locked by a thread and not released.
Further investigating stack's thread I found out that the aforementioned thread is unloading a user's registry profile (hive) using RegUnLoadKey function.
The stack of the thread:
# ChildEBP RetAddr Args to Child
00 9ef03510 82a8ef6d 85ad8030 00000000 807cc120 nt!KiSwapContext+0x26
01 9ef03548 82a8ddc7 85ad80f0 85ad8030 9ef03688 nt!KiSwapThread+0x266
02 9ef03570 82a8760f 85ad8030 85ad80f0 0000002a nt!KiCommitThreadWait+0x1df
03 9ef035e8 8b0b1be9 9ef03688 00000000 bc65de01 nt!KeWaitForSingleObject+0x393
04 9ef03648 8b0b260b 9ef03668 bc65de84 bc65de70 fileinfo!FIPfBlockedOpBlock+0x15b
05 9ef03718 8b0b2774 bc65de70 00000004 00000001 fileinfo!FIPfFileFSOpStart+0x183
06 9ef0373c 8b0aabcc bc65de70 9ef037a8 882c3e3c fileinfo!FIPfFileFSOpUnbufferedWriteStart+0x106
07 9ef0379c 8b077df7 882c3e3c 9ef037bc 9ef037ec fileinfo!FIPreReadWriteCallback+0x188
08 9ef03808 8b07ad34 9ef0385c c2c18e28 00000000 fltmgr!FltpPerformPreCallbacks+0x361
09 9ef03820 8b07b24d 9ef0385c 00000000 864bced8 fltmgr!FltpPassThroughInternal+0x40
0a 9ef03844 8b07b70c 04f03801 864bced8 00000000 fltmgr!FltpPassThrough+0x203
0b 9ef03874 82a47129 864bced8 c2c18e28 c2c18e28 fltmgr!FltpDispatch+0xb4
0c 9ef0388c 82c3f7af c93973b0 c2c18e28 c2c18fdc nt!IofCallDriver+0x63
0d 9ef038ac 82c85cb8 864bced8 c93973b0 00000001 nt!IopSynchronousServiceTail+0x1f8
0e 9ef03948 82a4ddb6 864bced8 800031cc 00000000 nt!NtWriteFile+0x6e8
0f 9ef03948 82a4d215 864bced8 800031cc 00000000 nt!KiSystemServicePostCall
10 9ef039e4 82c21b87 80002e34 800031cc 00000000 nt!ZwWriteFile+0x11
11 9ef03a60 82c245a5 80002e34 c9479248 00000001 nt!CmpDoFileWrite+0x15c
12 9ef03a7c 82c225f0 c3241008 00000000 c9479248 nt!CmpFileWrite+0x33
13 9ef03ac8 82c21f4c c000014d 00000000 a2053101 nt!HvWriteDirtyDataToHive+0x155
14 9ef03adc 82bfc207 adfd0008 00000020 c3241008 nt!HvOptimizedSyncHive+0x27
15 9ef03afc 82c9ddf4 adfd0008 00000001 00000000 nt!CmFlushKey+0x22e
16 9ef03b24 82c9f225 00000000 9ef03be8 1dcfbbeb nt!CmUnloadKey+0x4c
17 9ef03c14 82c9ecdb 00000000 00000000 85ad8030 nt!NtUnloadKey2+0x53f
18 9ef03c28 82a4ddb6 05b3f538 05b3f550 76df6c74 nt!NtUnloadKey+0x10
19 9ef03c28 76df6c74 05b3f538 05b3f550 76df6c74 nt!KiSystemServicePostCall
1a 05b3f524 76df653c 76c00ed0 05b3f538 7675171f ntdll!KiFastSystemCallRet
1b 05b3f528 76c00ed0 05b3f538 7675171f 00000018 ntdll!NtUnloadKey+0xc
1c 05b3f550 76bf9cbe 0000034c 05b3f570 73f1041c kernel32!LocalBaseRegUnLoadKey+0x5a
1d 05b3f5a4 005a38cf 80000003 037a7d08 005a3c10 kernel32!RegUnLoadKeyW+0x85
1e 05b3f5b0 005a3c10 037a7d08 73dd7ee1 0427a870 Agent_2!Registry::CRegistryLoader::UnLoad+0xf
As far I understand the registry unload key function locks the registry and then some functions are called to write some data to the hive file and that causes the filter manager to call the fileinfo filter, however I am not sure why the fileinfo fails to finish successfully (it seems its operation is blocked).
I would appreciate if anyone could give me any info about how to further debug the system calls and the "fileinfo" or any ideas of what might be going wrong so I can fix the issue.
Let me know if more information is needed.
Thanks,
Kostas

Comments

  • Scott_Noone_(OSR)Scott_Noone_(OSR) Administrator Posts: 3,302
    Try a !stacks 2 and see if there are any threads that jump out as
    interesting (e.g. running in the fileinfo filter or blocked elsewhere in the
    file system).

    -scott
    OSR
    @OSRDrivers

    wrote in message news:[email protected]

    I have an issue with a windows desktop application I develop which is
    causing two VMs running Windows 7 Sp1 and windows 10 to hang both on a
    VMware ESXi.
    After searching the dump file of the VM I found out that the problem is
    happening because the Registry is exclusively locked by a thread and not
    released.
    Further investigating stack's thread I found out that the aforementioned
    thread is unloading a user's registry profile (hive) using RegUnLoadKey
    function.
    The stack of the thread:
    # ChildEBP RetAddr Args to Child
    00 9ef03510 82a8ef6d 85ad8030 00000000 807cc120 nt!KiSwapContext+0x26
    01 9ef03548 82a8ddc7 85ad80f0 85ad8030 9ef03688 nt!KiSwapThread+0x266
    02 9ef03570 82a8760f 85ad8030 85ad80f0 0000002a nt!KiCommitThreadWait+0x1df
    03 9ef035e8 8b0b1be9 9ef03688 00000000 bc65de01
    nt!KeWaitForSingleObject+0x393
    04 9ef03648 8b0b260b 9ef03668 bc65de84 bc65de70
    fileinfo!FIPfBlockedOpBlock+0x15b
    05 9ef03718 8b0b2774 bc65de70 00000004 00000001
    fileinfo!FIPfFileFSOpStart+0x183
    06 9ef0373c 8b0aabcc bc65de70 9ef037a8 882c3e3c
    fileinfo!FIPfFileFSOpUnbufferedWriteStart+0x106
    07 9ef0379c 8b077df7 882c3e3c 9ef037bc 9ef037ec
    fileinfo!FIPreReadWriteCallback+0x188
    08 9ef03808 8b07ad34 9ef0385c c2c18e28 00000000
    fltmgr!FltpPerformPreCallbacks+0x361
    09 9ef03820 8b07b24d 9ef0385c 00000000 864bced8
    fltmgr!FltpPassThroughInternal+0x40
    0a 9ef03844 8b07b70c 04f03801 864bced8 00000000 fltmgr!FltpPassThrough+0x203
    0b 9ef03874 82a47129 864bced8 c2c18e28 c2c18e28 fltmgr!FltpDispatch+0xb4
    0c 9ef0388c 82c3f7af c93973b0 c2c18e28 c2c18fdc nt!IofCallDriver+0x63
    0d 9ef038ac 82c85cb8 864bced8 c93973b0 00000001
    nt!IopSynchronousServiceTail+0x1f8
    0e 9ef03948 82a4ddb6 864bced8 800031cc 00000000 nt!NtWriteFile+0x6e8
    0f 9ef03948 82a4d215 864bced8 800031cc 00000000 nt!KiSystemServicePostCall
    10 9ef039e4 82c21b87 80002e34 800031cc 00000000 nt!ZwWriteFile+0x11
    11 9ef03a60 82c245a5 80002e34 c9479248 00000001 nt!CmpDoFileWrite+0x15c
    12 9ef03a7c 82c225f0 c3241008 00000000 c9479248 nt!CmpFileWrite+0x33
    13 9ef03ac8 82c21f4c c000014d 00000000 a2053101
    nt!HvWriteDirtyDataToHive+0x155
    14 9ef03adc 82bfc207 adfd0008 00000020 c3241008 nt!HvOptimizedSyncHive+0x27
    15 9ef03afc 82c9ddf4 adfd0008 00000001 00000000 nt!CmFlushKey+0x22e
    16 9ef03b24 82c9f225 00000000 9ef03be8 1dcfbbeb nt!CmUnloadKey+0x4c
    17 9ef03c14 82c9ecdb 00000000 00000000 85ad8030 nt!NtUnloadKey2+0x53f
    18 9ef03c28 82a4ddb6 05b3f538 05b3f550 76df6c74 nt!NtUnloadKey+0x10
    19 9ef03c28 76df6c74 05b3f538 05b3f550 76df6c74 nt!KiSystemServicePostCall
    1a 05b3f524 76df653c 76c00ed0 05b3f538 7675171f ntdll!KiFastSystemCallRet
    1b 05b3f528 76c00ed0 05b3f538 7675171f 00000018 ntdll!NtUnloadKey+0xc
    1c 05b3f550 76bf9cbe 0000034c 05b3f570 73f1041c
    kernel32!LocalBaseRegUnLoadKey+0x5a
    1d 05b3f5a4 005a38cf 80000003 037a7d08 005a3c10 kernel32!RegUnLoadKeyW+0x85
    1e 05b3f5b0 005a3c10 037a7d08 73dd7ee1 0427a870
    Agent_2!Registry::CRegistryLoader::UnLoad+0xf
    As far I understand the registry unload key function locks the registry and
    then some functions are called to write some data to the hive file and that
    causes the filter manager to call the fileinfo filter, however I am not sure
    why the fileinfo fails to finish successfully (it seems its operation is
    blocked).
    I would appreciate if anyone could give me any info about how to further
    debug the system calls and the "fileinfo" or any ideas of what might be
    going wrong so I can fix the issue.
    Let me know if more information is needed.
    Thanks,
    Kostas

    -scott
    OSR

  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    Hi Scott,

    Thanks for your reply.
    I run the !stacks 2 command as you suggested.

    There was only one more thread with fileinfo appearing in its stack and it was owned by SearchProtocolHost however there wasn't something interesting there. Here is the stack:

    ChildEBP RetAddr Args to Child
    9ef43190 82a8ef6d c92f42f8 00000000 807cc120 nt!KiSwapContext+0x26 (FPO: [Uses EBP] [0,0,4])
    9ef431c8 82a8ddc7 c92f43b8 c92f42f8 d6fed0e8 nt!KiSwapThread+0x266
    9ef431f0 82a8760f c92f42f8 c92f43b8 00000000 nt!KiCommitThreadWait+0x1df
    9ef4326c 82c1cced d6fed0e8 00000000 00000000 nt!KeWaitForSingleObject+0x393
    9ef432ac 82c1e315 00000001 9ef432d4 00000000 nt!FsRtlCancellableWaitForMultipleObjects+0x74
    9ef432cc 8b077e17 d6fed0e8 00000000 00000000 nt!FsRtlCancellableWaitForSingleObject+0x1a
    9ef43338 8b07ad34 9ef4337c c2dce268 00000000 fltmgr!FltpPerformPreCallbacks+0x381 (FPO: [Non-Fpo])
    9ef43350 8b08e254 9ef4337c 8b0925ae 00000000 fltmgr!FltpPassThroughInternal+0x40 (FPO: [Non-Fpo])
    9ef43364 8b08e919 9ef4337c c2dce268 d52a75c8 fltmgr!FltpCreateInternal+0x24 (FPO: [Non-Fpo])
    9ef433a8 82a47129 864bced8 864bd210 d52a7624 fltmgr!FltpCreate+0x2c9 (FPO: [Non-Fpo])
    9ef433c0 82c5b535 1dcbb35f 9ef43570 00000000 nt!IofCallDriver+0x63
    9ef434a0 82c3a8fe 864aeaa8 851de970 c93b8d20 nt!IopParseDevice+0xf08
    9ef4351c 82c4aeac 00000000 9ef43570 00000240 nt!ObpLookupObjectName+0x510
    9ef4357c 82c417db 9ef43860 851de970 00000000 nt!ObOpenObjectByName+0x165
    9ef435f8 82c830d1 9ef43834 000000a1 9ef43860 nt!IopCreateFile+0x673
    9ef43654 8b090fca 9ef43834 000000a1 9ef43860 nt!IoCreateFileEx+0x9e
    9ef436e0 8b091192 851fb008 00000000 9ef43834 fltmgr!FltpCreateFile+0xba (FPO: [Non-Fpo])
    9ef43764 8b0b32d3 851fb008 00000000 9ef43834 fltmgr!FltCreateFileEx2+0x5a (FPO: [Non-Fpo])
    9ef43844 82c92038 00000000 badeac80 00000110 fileinfo!FIPfInterfaceOpen+0x2a9 (FPO: [Non-Fpo])
    9ef438a8 82c980c9 9ef43954 00000000 000000a1 nt!PfpOpenHandleCreate+0xc0
    9ef4391c 82c97deb 9ef439d4 b42d2b90 9ef43954 nt!PfSnGetSectionObject+0x9a
    9ef439b4 82c97b1f 00f439d4 00000000 00000000 nt!PfSnPrefetchSections+0x1d4
    9ef43b34 82c70964 c2e12000 9ef43b64 9ef43b70 nt!PfSnPrefetchScenario+0x193
    9ef43bc8 82c887ab 82c6ea3b c86ee3d8 9ef43c20 nt!PfSnBeginAppLaunch+0x382
    9ef43bd8 82c6e64e 1dcbbbdf 00000000 00000000 nt!PfProcessCreateNotification+0x65
    9ef43c20 82ac1a19 00000000 76df6c58 00000001 nt!PspUserThreadStartup+0x113
    00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x19


    As I am not experienced with Windbg and Winows system debugging I wasn't sure how to check for filesystem related threads, so I searched for fltmgr calls and I found one more thread.
    Again this doesn't seem to be doing anything interesting.

    8d396878 82a8ef6d 851f6a70 00000000 8d315120 nt!KiSwapContext+0x26 (FPO: [Uses EBP] [0,0,4])
    8d3968b0 82a8ddc7 851f6b30 851f6a70 887dab80 nt!KiSwapThread+0x266
    8d3968d8 82a8760f 851f6a70 851f6b30 00000000 nt!KiCommitThreadWait+0x1df
    8d396950 947f8494 887dab80 00000000 00000000 nt!KeWaitForSingleObject+0x393
    WARNING: Stack unwind information not available. Following frames may be wrong.
    8d396bb0 947f7ab3 88637520 000016fc 87946d40 mbam+0x1494
    8d396bcc 8b09ab8e 864d7af8 bd9f856c 87943cf8 mbam+0xab3
    8d396c00 82a8e37b 00000000 00000000 851f6a70 fltmgr!FltpProcessDeferredIoWorkItem+0x3a (FPO: [Non-Fpo])
    8d396c50 82c1d4e7 00000000 0e06eb6f 00000000 nt!ExpWorkerThread+0x10d
    8d396c90 82ac1a19 82a8e26e 00000000 00000000 nt!PspSystemThreadStartup+0x9e
    00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x19

    Is there any other way I can find threads with filesystem calls?
    Any other ideas?

    Thanks again for your time.
    Kostas
  • Scott_Noone_(OSR)Scott_Noone_(OSR) Administrator Posts: 3,302
    Nice work tracking down some more interesting threads.

    So, FileInfo doesn't want the operation to proceed until it can open a file.
    The FileInfo open request was pended and FltMgr is waiting for the open
    request to complete. In the other thread, you have the Malwarebytes
    anti-malware driver waiting in a work item for something ELSE to complete.
    If I had to guess, it's probably waiting for its user mode engine to say if
    the FileInfo open should be allowed to proceed or not.

    If you disable Malwarebytes does the problem go away?

    -scott
    OSR
    @OSRDrivers

    wrote in message news:[email protected]

    Hi Scott,

    Thanks for your reply.
    I run the !stacks 2 command as you suggested.

    There was only one more thread with fileinfo appearing in its stack and it
    was owned by SearchProtocolHost however there wasn't something interesting
    there. Here is the stack:

    ChildEBP RetAddr Args to Child
    9ef43190 82a8ef6d c92f42f8 00000000 807cc120 nt!KiSwapContext+0x26 (FPO:
    [Uses EBP] [0,0,4])
    9ef431c8 82a8ddc7 c92f43b8 c92f42f8 d6fed0e8 nt!KiSwapThread+0x266
    9ef431f0 82a8760f c92f42f8 c92f43b8 00000000 nt!KiCommitThreadWait+0x1df
    9ef4326c 82c1cced d6fed0e8 00000000 00000000 nt!KeWaitForSingleObject+0x393
    9ef432ac 82c1e315 00000001 9ef432d4 00000000
    nt!FsRtlCancellableWaitForMultipleObjects+0x74
    9ef432cc 8b077e17 d6fed0e8 00000000 00000000
    nt!FsRtlCancellableWaitForSingleObject+0x1a
    9ef43338 8b07ad34 9ef4337c c2dce268 00000000
    fltmgr!FltpPerformPreCallbacks+0x381 (FPO: [Non-Fpo])
    9ef43350 8b08e254 9ef4337c 8b0925ae 00000000
    fltmgr!FltpPassThroughInternal+0x40 (FPO: [Non-Fpo])
    9ef43364 8b08e919 9ef4337c c2dce268 d52a75c8 fltmgr!FltpCreateInternal+0x24
    (FPO: [Non-Fpo])
    9ef433a8 82a47129 864bced8 864bd210 d52a7624 fltmgr!FltpCreate+0x2c9 (FPO:
    [Non-Fpo])
    9ef433c0 82c5b535 1dcbb35f 9ef43570 00000000 nt!IofCallDriver+0x63
    9ef434a0 82c3a8fe 864aeaa8 851de970 c93b8d20 nt!IopParseDevice+0xf08
    9ef4351c 82c4aeac 00000000 9ef43570 00000240 nt!ObpLookupObjectName+0x510
    9ef4357c 82c417db 9ef43860 851de970 00000000 nt!ObOpenObjectByName+0x165
    9ef435f8 82c830d1 9ef43834 000000a1 9ef43860 nt!IopCreateFile+0x673
    9ef43654 8b090fca 9ef43834 000000a1 9ef43860 nt!IoCreateFileEx+0x9e
    9ef436e0 8b091192 851fb008 00000000 9ef43834 fltmgr!FltpCreateFile+0xba
    (FPO: [Non-Fpo])
    9ef43764 8b0b32d3 851fb008 00000000 9ef43834 fltmgr!FltCreateFileEx2+0x5a
    (FPO: [Non-Fpo])
    9ef43844 82c92038 00000000 badeac80 00000110
    fileinfo!FIPfInterfaceOpen+0x2a9 (FPO: [Non-Fpo])
    9ef438a8 82c980c9 9ef43954 00000000 000000a1 nt!PfpOpenHandleCreate+0xc0
    9ef4391c 82c97deb 9ef439d4 b42d2b90 9ef43954 nt!PfSnGetSectionObject+0x9a
    9ef439b4 82c97b1f 00f439d4 00000000 00000000 nt!PfSnPrefetchSections+0x1d4
    9ef43b34 82c70964 c2e12000 9ef43b64 9ef43b70 nt!PfSnPrefetchScenario+0x193
    9ef43bc8 82c887ab 82c6ea3b c86ee3d8 9ef43c20 nt!PfSnBeginAppLaunch+0x382
    9ef43bd8 82c6e64e 1dcbbbdf 00000000 00000000
    nt!PfProcessCreateNotification+0x65
    9ef43c20 82ac1a19 00000000 76df6c58 00000001 nt!PspUserThreadStartup+0x113
    00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x19


    As I am not experienced with Windbg and Winows system debugging I wasn't
    sure how to check for filesystem related threads, so I searched for fltmgr
    calls and I found one more thread.
    Again this doesn't seem to be doing anything interesting.

    8d396878 82a8ef6d 851f6a70 00000000 8d315120 nt!KiSwapContext+0x26 (FPO:
    [Uses EBP] [0,0,4])
    8d3968b0 82a8ddc7 851f6b30 851f6a70 887dab80 nt!KiSwapThread+0x266
    8d3968d8 82a8760f 851f6a70 851f6b30 00000000 nt!KiCommitThreadWait+0x1df
    8d396950 947f8494 887dab80 00000000 00000000 nt!KeWaitForSingleObject+0x393
    WARNING: Stack unwind information not available. Following frames may be
    wrong.
    8d396bb0 947f7ab3 88637520 000016fc 87946d40 mbam+0x1494
    8d396bcc 8b09ab8e 864d7af8 bd9f856c 87943cf8 mbam+0xab3
    8d396c00 82a8e37b 00000000 00000000 851f6a70
    fltmgr!FltpProcessDeferredIoWorkItem+0x3a (FPO: [Non-Fpo])
    8d396c50 82c1d4e7 00000000 0e06eb6f 00000000 nt!ExpWorkerThread+0x10d
    8d396c90 82ac1a19 82a8e26e 00000000 00000000 nt!PspSystemThreadStartup+0x9e
    00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x19

    Is there any other way I can find threads with filesystem calls?
    Any other ideas?

    Thanks again for your time.
    Kostas

    -scott
    OSR

  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    Thanks for the reply Scott, I will try what you suggest and let you know. It might take a while as I do not have access to the machine.
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    Hi Scott,

    I would like to thank you for one more time.
    The problem is indeed caused by Malwarebytes software.
Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Upcoming OSR Seminars
OSR has suspended in-person seminars due to the Covid-19 outbreak. But, don't miss your training! Attend via the internet instead!
Kernel Debugging 30 Mar 2020 OSR Seminar Space
Developing Minifilters 15 Jun 2020 LIVE ONLINE
Writing WDF Drivers 22 June 2020 LIVE ONLINE
Internals & Software Drivers 28 Sept 2020 Dulles, VA