BSOD with access violation in ExAcquireFastMutex

NTFSD
1

My company’s file system driver likely has a subtle bug, but I am wondering
if anyone can point me in the right direction…

A BSOD occurs, often when first booting a Win2003 Server machine. The
minidump stack trace is very consistent, always ending with an access
violation in ExAcquireFastMutex. Our code does not appear in the stack
trace. Below is some output from windbg. Please let me know what other info
I could provide that would help.

It is not clear to me what our software is doing that would cause an access
violation in this stack path.

Thanks in advance for your guidance!
Charlie

KERNEL_MODE_EXCEPTION_NOT_HANDLED_M (1000008e)
This is a very common bugcheck. Usually the exception address pinpoints
the driver/function that caused the problem. Always note this address
as well as the link date of the driver/image that contains this address.
Some common problems are exception code 0x80000003. This means a hard
coded breakpoint or assertion was hit, but this system was booted
/NODEBUG. This is not supposed to happen as developers should never have
hardcoded breakpoints in retail code, but …
If this happens, make sure a debugger gets connected, and the
system is booted /DEBUG. This will let us see why this breakpoint is
happening.
Arguments:
Arg1: c0000005, The exception code that was not handled
Arg2: 8083ed90, The address that the exception occurred at
Arg3: f88929e8, Trap Frame
Arg4: 00000000

Debugging Details:

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at “0x%08lx”
referenced memory at “0x%08lx”. The memory could not be “%s”.

FAULTING_IP:
nt!ExAcquireFastMutex+13
8083ed90 f00fc106 lock xadd [esi],eax

TRAP_FRAME: f88929e8 – (.trap fffffffff88929e8)
ErrCode = 00000002
eax=ffffffff ebx=f8892b00 ecx=0000003d edx=00000001 esi=00000000
edi=819c0598
eip=8083ed90 esp=f8892a5c ebp=f8892a74 iopl=0 nv up ei ng nz na po
nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000
efl=00010286
nt!ExAcquireFastMutex+0x13:
8083ed90 f00fc106 lock xadd [esi],eax
ds:0023:00000000=00000000
Resetting default scope

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP

BUGCHECK_STR: 0x8E

CURRENT_IRQL: 1

LAST_CONTROL_TRANSFER: from 8086f015 to 8083ed90

STACK_TEXT:
f8892a60 8086f015 819c0598 00000000 f8892b28 nt!ExAcquireFastMutex+0x13
f8892a74 f830287a 819dbc78 819ba9c8 00000000
nt!FsRtlLookupPerStreamContextInternal+0x13
f8892ad8 f8314a43 819ba9c8 819c0598 00000000
fltMgr!FltpGetStreamListCtrl+0x5a
f8892af4 f82ff7c1 819ba9c8 819c0598 81e99020
fltMgr!FltpCleanupStreamListCtrlForFileObjectClose+0x17
f8892b10 f82ffcf1 f8892b28 819c0598 81f64d10 fltMgr!FltpPassThrough+0x93
f8892b40 80828c95 81e99020 81e314e8 81e314f8 fltMgr!FltpDispatch+0x10d
f8892b54 8090afbf 80a6ea90 00000000 819c0580 nt!IofCallDriver+0x45
f8892b8c 80904834 009c0598 819c0580 00000000 nt!IopDeleteFile+0x13a
f8892ba4 80828af5 819c0598 00000000 00000798 nt!ObpRemoveObjectRoutine+0xde
f8892bc4 80905472 e1001ca8 81faa818 81fa6020 nt!ObfDereferenceObject+0x67
f8892bdc 80905528 e1001ca8 819c0598 00000798
nt!ObpCloseHandleTableEntry+0x139
f8892c20 80905573 00000798 00000000 f8892c3c nt!ObpCloseHandle+0x82
f8892c30 8082337b 80000798 f8892d14 80821380 nt!NtClose+0x1b
f8892c30 80821380 80000798 f8892d14 80821380 nt!KiFastCallEntry+0xf8
f8892cac 808ca842 80000798 e1dc5178 e1f771b8 nt!ZwClose+0x11
f8892d14 808ca8aa e1dc5158 00000000 e1dc5148 nt!CcPfQueryVolumeInfo+0xc4
f8892d3c 808e2e3e 81f62368 e1f77198 00000010 nt!CcPfUpdateVolumeList+0xce
f8892d80 808203bd 81f62368 00000000 81fa6020
nt!CcPfGetFileNamesWorkerRoutine+0x180
f8892dac 80905d2c 81f62368 00000000 00000000 nt!ExpWorkerThread+0xeb
f8892ddc 80828499 80820300 00000001 00000000 nt!PspSystemThreadStartup+0x2e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

STACK_COMMAND: kb

FOLLOWUP_IP:
fltMgr!FltpGetStreamListCtrl+5a
f830287a eb02 jmp fltMgr!FltpGetStreamListCtrl+0x5e
(f830287e)

FAULTING_SOURCE_CODE:

SYMBOL_STACK_INDEX: 2

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: fltMgr!FltpGetStreamListCtrl+5a

MODULE_NAME: fltMgr

IMAGE_NAME: fltMgr.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 42435ba1

FAILURE_BUCKET_ID: 0x8E_fltMgr!FltpGetStreamListCtrl+5a

BUCKET_ID: 0x8E_fltMgr!FltpGetStreamListCtrl+5a

Followup: MachineOwner

2

Have you tried to look at which file is being closed here? It might
provide you with some additional insight.

From the stack trace, were I to guess (it’s tough from just !analyze
output) I’d say that this file object has somehow become corrupted.

Is your filter a mini-filter? If it is, perhaps knowing which file it
was will give you insight into whether or not you’d have associated
context with it. If your filter is a legacy filter, do you use filter
contexts? If you don’t use filter contexts, do you create file objects
(e.g., IoCreateStreamFileObjectXXX calls.)

So, places to look:

  • Is the file object even valid. I’ve seen obscure issues when someone
    tries to use a stack-based file object for performing I/O.
  • Are you using contexts?
  • Does the knowledge of which file this is provide you with any insight.

With an actual crash dump it might be possible to dig out more from
this.

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Charlie
Sent: Tuesday, September 12, 2006 8:55 AM
To: ntfsd redirect
Subject: [ntfsd] BSOD with access violation in ExAcquireFastMutex

My company’s file system driver likely has a subtle bug, but I am
wondering
if anyone can point me in the right direction…

A BSOD occurs, often when first booting a Win2003 Server machine. The
minidump stack trace is very consistent, always ending with an access
violation in ExAcquireFastMutex. Our code does not appear in the stack
trace. Below is some output from windbg. Please let me know what other
info
I could provide that would help.

It is not clear to me what our software is doing that would cause an
access
violation in this stack path.

Thanks in advance for your guidance!
Charlie

KERNEL_MODE_EXCEPTION_NOT_HANDLED_M (1000008e)
This is a very common bugcheck. Usually the exception address pinpoints
the driver/function that caused the problem. Always note this address
as well as the link date of the driver/image that contains this address.
Some common problems are exception code 0x80000003. This means a hard
coded breakpoint or assertion was hit, but this system was booted
/NODEBUG. This is not supposed to happen as developers should never
have
hardcoded breakpoints in retail code, but …
If this happens, make sure a debugger gets connected, and the
system is booted /DEBUG. This will let us see why this breakpoint is
happening.
Arguments:
Arg1: c0000005, The exception code that was not handled
Arg2: 8083ed90, The address that the exception occurred at
Arg3: f88929e8, Trap Frame
Arg4: 00000000

Debugging Details:

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at “0x%08lx”
referenced memory at “0x%08lx”. The memory could not be “%s”.

FAULTING_IP:
nt!ExAcquireFastMutex+13
8083ed90 f00fc106 lock xadd [esi],eax

TRAP_FRAME: f88929e8 – (.trap fffffffff88929e8)
ErrCode = 00000002
eax=ffffffff ebx=f8892b00 ecx=0000003d edx=00000001 esi=00000000
edi=819c0598
eip=8083ed90 esp=f8892a5c ebp=f8892a74 iopl=0 nv up ei ng nz na
po
nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000
efl=00010286
nt!ExAcquireFastMutex+0x13:
8083ed90 f00fc106 lock xadd [esi],eax
ds:0023:00000000=00000000
Resetting default scope

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP

BUGCHECK_STR: 0x8E

CURRENT_IRQL: 1

LAST_CONTROL_TRANSFER: from 8086f015 to 8083ed90

STACK_TEXT:
f8892a60 8086f015 819c0598 00000000 f8892b28 nt!ExAcquireFastMutex+0x13
f8892a74 f830287a 819dbc78 819ba9c8 00000000
nt!FsRtlLookupPerStreamContextInternal+0x13
f8892ad8 f8314a43 819ba9c8 819c0598 00000000
fltMgr!FltpGetStreamListCtrl+0x5a
f8892af4 f82ff7c1 819ba9c8 819c0598 81e99020
fltMgr!FltpCleanupStreamListCtrlForFileObjectClose+0x17
f8892b10 f82ffcf1 f8892b28 819c0598 81f64d10 fltMgr!FltpPassThrough+0x93
f8892b40 80828c95 81e99020 81e314e8 81e314f8 fltMgr!FltpDispatch+0x10d
f8892b54 8090afbf 80a6ea90 00000000 819c0580 nt!IofCallDriver+0x45
f8892b8c 80904834 009c0598 819c0580 00000000 nt!IopDeleteFile+0x13a
f8892ba4 80828af5 819c0598 00000000 00000798
nt!ObpRemoveObjectRoutine+0xde
f8892bc4 80905472 e1001ca8 81faa818 81fa6020
nt!ObfDereferenceObject+0x67
f8892bdc 80905528 e1001ca8 819c0598 00000798
nt!ObpCloseHandleTableEntry+0x139
f8892c20 80905573 00000798 00000000 f8892c3c nt!ObpCloseHandle+0x82
f8892c30 8082337b 80000798 f8892d14 80821380 nt!NtClose+0x1b
f8892c30 80821380 80000798 f8892d14 80821380 nt!KiFastCallEntry+0xf8
f8892cac 808ca842 80000798 e1dc5178 e1f771b8 nt!ZwClose+0x11
f8892d14 808ca8aa e1dc5158 00000000 e1dc5148 nt!CcPfQueryVolumeInfo+0xc4
f8892d3c 808e2e3e 81f62368 e1f77198 00000010
nt!CcPfUpdateVolumeList+0xce
f8892d80 808203bd 81f62368 00000000 81fa6020
nt!CcPfGetFileNamesWorkerRoutine+0x180
f8892dac 80905d2c 81f62368 00000000 00000000 nt!ExpWorkerThread+0xeb
f8892ddc 80828499 80820300 00000001 00000000
nt!PspSystemThreadStartup+0x2e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

STACK_COMMAND: kb

FOLLOWUP_IP:
fltMgr!FltpGetStreamListCtrl+5a
f830287a eb02 jmp fltMgr!FltpGetStreamListCtrl+0x5e
(f830287e)

FAULTING_SOURCE_CODE:

SYMBOL_STACK_INDEX: 2

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: fltMgr!FltpGetStreamListCtrl+5a

MODULE_NAME: fltMgr

IMAGE_NAME: fltMgr.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 42435ba1

FAILURE_BUCKET_ID: 0x8E_fltMgr!FltpGetStreamListCtrl+5a

BUCKET_ID: 0x8E_fltMgr!FltpGetStreamListCtrl+5a

Followup: MachineOwner

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

3

David Kane of Symantec pointed us in this direction:
http://www.osronline.com/showThread.cfm?link=69126

Specifically the posting by Neal Christiansen.

Acting upon this tip, we replaced our usage of FSRTL_COMMON_FCB_HEADER with
FSRTL_ADVANCED_FCB_HEADER, initialized the new components, and our BSOD has
gone away. There may be some new Windows code that is expecting that the
ADVANCED structure is used…

Charlie

“Charlie” wrote in message news:xxxxx@ntfsd…

My company’s file system driver likely has a subtle bug, but I am
wondering if anyone can point me in the right direction…

A BSOD occurs, often when first booting a Win2003 Server machine. The
minidump stack trace is very consistent, always ending with an access
violation in ExAcquireFastMutex. Our code does not appear in the stack
trace. Below is some output from windbg. Please let me know what other
info I could provide that would help.

It is not clear to me what our software is doing that would cause an
access violation in this stack path.

Thanks in advance for your guidance!
Charlie

KERNEL_MODE_EXCEPTION_NOT_HANDLED_M (1000008e)
This is a very common bugcheck. Usually the exception address pinpoints
the driver/function that caused the problem. Always note this address
as well as the link date of the driver/image that contains this address.
Some common problems are exception code 0x80000003. This means a hard
coded breakpoint or assertion was hit, but this system was booted
/NODEBUG. This is not supposed to happen as developers should never have
hardcoded breakpoints in retail code, but …
If this happens, make sure a debugger gets connected, and the
system is booted /DEBUG. This will let us see why this breakpoint is
happening.
Arguments:
Arg1: c0000005, The exception code that was not handled
Arg2: 8083ed90, The address that the exception occurred at
Arg3: f88929e8, Trap Frame
Arg4: 00000000

Debugging Details:

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at “0x%08lx”
referenced memory at “0x%08lx”. The memory could not be “%s”.

FAULTING_IP:
nt!ExAcquireFastMutex+13
8083ed90 f00fc106 lock xadd [esi],eax

TRAP_FRAME: f88929e8 – (.trap fffffffff88929e8)
ErrCode = 00000002
eax=ffffffff ebx=f8892b00 ecx=0000003d edx=00000001 esi=00000000
edi=819c0598
eip=8083ed90 esp=f8892a5c ebp=f8892a74 iopl=0 nv up ei ng nz na po
nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010286
nt!ExAcquireFastMutex+0x13:
8083ed90 f00fc106 lock xadd [esi],eax ds:0023:00000000=00000000
Resetting default scope

CUSTOMER_CRASH_COUNT: 1

DEFAULT_BUCKET_ID: DRIVER_FAULT_SERVER_MINIDUMP

BUGCHECK_STR: 0x8E

CURRENT_IRQL: 1

LAST_CONTROL_TRANSFER: from 8086f015 to 8083ed90

STACK_TEXT:
f8892a60 8086f015 819c0598 00000000 f8892b28 nt!ExAcquireFastMutex+0x13
f8892a74 f830287a 819dbc78 819ba9c8 00000000
nt!FsRtlLookupPerStreamContextInternal+0x13
f8892ad8 f8314a43 819ba9c8 819c0598 00000000
fltMgr!FltpGetStreamListCtrl+0x5a
f8892af4 f82ff7c1 819ba9c8 819c0598 81e99020
fltMgr!FltpCleanupStreamListCtrlForFileObjectClose+0x17
f8892b10 f82ffcf1 f8892b28 819c0598 81f64d10 fltMgr!FltpPassThrough+0x93
f8892b40 80828c95 81e99020 81e314e8 81e314f8 fltMgr!FltpDispatch+0x10d
f8892b54 8090afbf 80a6ea90 00000000 819c0580 nt!IofCallDriver+0x45
f8892b8c 80904834 009c0598 819c0580 00000000 nt!IopDeleteFile+0x13a
f8892ba4 80828af5 819c0598 00000000 00000798
nt!ObpRemoveObjectRoutine+0xde
f8892bc4 80905472 e1001ca8 81faa818 81fa6020 nt!ObfDereferenceObject+0x67
f8892bdc 80905528 e1001ca8 819c0598 00000798
nt!ObpCloseHandleTableEntry+0x139
f8892c20 80905573 00000798 00000000 f8892c3c nt!ObpCloseHandle+0x82
f8892c30 8082337b 80000798 f8892d14 80821380 nt!NtClose+0x1b
f8892c30 80821380 80000798 f8892d14 80821380 nt!KiFastCallEntry+0xf8
f8892cac 808ca842 80000798 e1dc5178 e1f771b8 nt!ZwClose+0x11
f8892d14 808ca8aa e1dc5158 00000000 e1dc5148 nt!CcPfQueryVolumeInfo+0xc4
f8892d3c 808e2e3e 81f62368 e1f77198 00000010 nt!CcPfUpdateVolumeList+0xce
f8892d80 808203bd 81f62368 00000000 81fa6020
nt!CcPfGetFileNamesWorkerRoutine+0x180
f8892dac 80905d2c 81f62368 00000000 00000000 nt!ExpWorkerThread+0xeb
f8892ddc 80828499 80820300 00000001 00000000
nt!PspSystemThreadStartup+0x2e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

STACK_COMMAND: kb

FOLLOWUP_IP:
fltMgr!FltpGetStreamListCtrl+5a
f830287a eb02 jmp fltMgr!FltpGetStreamListCtrl+0x5e
(f830287e)

FAULTING_SOURCE_CODE:

SYMBOL_STACK_INDEX: 2

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: fltMgr!FltpGetStreamListCtrl+5a

MODULE_NAME: fltMgr

IMAGE_NAME: fltMgr.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 42435ba1

FAILURE_BUCKET_ID: 0x8E_fltMgr!FltpGetStreamListCtrl+5a

BUCKET_ID: 0x8E_fltMgr!FltpGetStreamListCtrl+5a

Followup: MachineOwner