KeExpandKernelStackAndCallout crash

I’ve gotten this same crash in the same process context twice now and I’m assuming it’s the result of one of my drivers. Apparently Chrome doesn’t like something I’m doing. However, I’m not doing any recursion or using any stack space so the message about extending the stack doesn’t make sense to me. The other unusual thing is that I don’t have driver verifier enabled on this system for any drivers so the error message is even more suspect. This has only occurred on a Win8.1 x64 system and I haven’t seen it on any of my Win7 systems. If anyone has any tips on how to investigate this further please let me know.

Either way, here’s the !analyze -v

Loading Dump File [V:\MEMORY.DMP]
Kernel Bitmap Dump File: Only kernel address space is available

Symbol search path is: SRV*C:\windows\symbols*http://msdl.microsoft.com/download/symbols;
Executable search path is:
Windows 8 Kernel Version 9600 MP (2 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS Personal
Built by: 9600.17736.amd64fre.winblue_r9.150322-1500
Machine Name:
Kernel base = 0xfffff80142872000 PsLoadedModuleList = 0xfffff80142b4b850
Debug session time: Wed Apr 29 18:29:38.420 2015 (UTC - 4:00)
System Uptime: 10 days 16:41:57.274
Loading Kernel Symbols



Loading User Symbols
PEB is paged out (Peb.Ldr = 00000000`ff4ab018). Type “.hh dbgerr001” for details
Loading unloaded module list

*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck C4, {91, 1, ffffe00196ae8880, 0}

Probably caused by : ntkrnlmp.exe ( nt!RtlpGetStackLimits+ee )

Followup: MachineOwner

1: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught. This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this driver.
If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA will
be among the most commonly seen crashes.
Arguments:
Arg1: 0000000000000091, A driver switched stacks using a method that is not supported by
the operating system. The only supported way to extend a kernel
mode stack is by using KeExpandKernelStackAndCallout.
Arg2: 0000000000000001
Arg3: ffffe00196ae8880
Arg4: 0000000000000000

Debugging Details:

BUGCHECK_STR: 0xc4_91

DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT

PROCESS_NAME: chrome.exe

CURRENT_IRQL: 1

EXCEPTION_RECORD: 0000000000000005 – (.exr 0x5)
Cannot read Exception record @ 0000000000000005

LAST_CONTROL_TRANSFER: from fffff8014293e552 to fffff801429c2ca0

STACK_TEXT:
fffff801444690c8 fffff8014293e552 : 00000000000000c4 0000000000000091 0000000000000001 ffffe00196ae8880 : nt!KeBugCheckEx
fffff801444690d0 fffff8014293af61 : 0000000000000000 fffff80138a088cb ffffe00100401802 ffff2541b40e1542 : nt!RtlpGetStackLimits+0xee
fffff80144469110 fffff8014293f45e : fffff8014446a008 fffff80144469d10 fffff8014446a008 ffffe00196ae8c00 : nt!RtlDispatchException+0x61
fffff801444697e0 fffff801429ce8c2 : 0000000000000005 ffffe00196ae8c00 ffffe0019a438c60 0000000000000000 : nt!KiDispatchException+0x646
fffff80144469ed0 fffff801429ccdfe : 0000000000000000 0000000000000000 fffff80142b75180 0000000000000001 : nt!KiExceptionDispatch+0xc2
fffff8014446a0b0 fffff801429c7e0e : fffff8013b888dff ffffe001904411b0 ffffe00100000000 ffffe00196ae8800 : nt!KiGeneralProtectionFault+0xfe
fffff8014446a248 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!KiDpcInterrupt+0x1de

STACK_COMMAND: kb

FOLLOWUP_IP:
nt!RtlpGetStackLimits+ee
fffff801`4293e552 cc int 3

SYMBOL_STACK_INDEX: 1

SYMBOL_NAME: nt!RtlpGetStackLimits+ee

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: nt

IMAGE_NAME: ntkrnlmp.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 550f41a6

BUCKET_ID_FUNC_OFFSET: ee

FAILURE_BUCKET_ID: 0xc4_91_nt!RtlpGetStackLimits

BUCKET_ID: 0xc4_91_nt!RtlpGetStackLimits

Followup: MachineOwner

I think the crash code is a red herring, my guess would be a stack
corruption. The fact that it’s happened in the Chrome process is probably
also unrelated.

Does any of your code run the context of a DPC? Do you have any traces of
the last thing(s) going on in your driver before the crash?

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntdev…

I’ve gotten this same crash in the same process context twice now and I’m
assuming it’s the result of one of my drivers. Apparently Chrome doesn’t
like something I’m doing. However, I’m not doing any recursion or using any
stack space so the message about extending the stack doesn’t make sense to
me. The other unusual thing is that I don’t have driver verifier enabled on
this system for any drivers so the error message is even more suspect. This
has only occurred on a Win8.1 x64 system and I haven’t seen it on any of my
Win7 systems. If anyone has any tips on how to investigate this further
please let me know.

Either way, here’s the !analyze -v

Loading Dump File [V:\MEMORY.DMP]
Kernel Bitmap Dump File: Only kernel address space is available

Symbol search path is:
SRV*C:\windows\symbols*http://msdl.microsoft.com/download/symbols;
Executable search path is:
Windows 8 Kernel Version 9600 MP (2 procs) Free x64
Product: WinNt, suite: TerminalServer SingleUserTS Personal
Built by: 9600.17736.amd64fre.winblue_r9.150322-1500
Machine Name:
Kernel base = 0xfffff80142872000 PsLoadedModuleList = 0xfffff80142b4b850
Debug session time: Wed Apr 29 18:29:38.420 2015 (UTC - 4:00)
System Uptime: 10 days 16:41:57.274
Loading Kernel Symbols



Loading User Symbols
PEB is paged out (Peb.Ldr = 00000000`ff4ab018). Type “.hh dbgerr001” for
details
Loading unloaded module list

*******************************************************************************
*
*
* Bugcheck Analysis
*
*
*
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck C4, {91, 1, ffffe00196ae8880, 0}

Probably caused by : ntkrnlmp.exe ( nt!RtlpGetStackLimits+ee )

Followup: MachineOwner

1: kd> !analyze -v
*******************************************************************************
*
*
* Bugcheck Analysis
*
*
*
*******************************************************************************

DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught. This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this
driver.
If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA
will
be among the most commonly seen crashes.
Arguments:
Arg1: 0000000000000091, A driver switched stacks using a method that is not
supported by
the operating system. The only supported way to extend a kernel
mode stack is by using KeExpandKernelStackAndCallout.
Arg2: 0000000000000001
Arg3: ffffe00196ae8880
Arg4: 0000000000000000

Debugging Details:

BUGCHECK_STR: 0xc4_91

DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT

PROCESS_NAME: chrome.exe

CURRENT_IRQL: 1

EXCEPTION_RECORD: 0000000000000005 – (.exr 0x5)
Cannot read Exception record @ 0000000000000005

LAST_CONTROL_TRANSFER: from fffff8014293e552 to fffff801429c2ca0

STACK_TEXT:
fffff801444690c8 fffff8014293e552 : 00000000000000c4 0000000000000091
0000000000000001 ffffe00196ae8880 : nt!KeBugCheckEx
fffff801444690d0 fffff8014293af61 : 0000000000000000 fffff80138a088cb
ffffe00100401802 ffff2541b40e1542 : nt!RtlpGetStackLimits+0xee
fffff80144469110 fffff8014293f45e : fffff8014446a008 fffff80144469d10
fffff8014446a008 ffffe00196ae8c00 : nt!RtlDispatchException+0x61
fffff801444697e0 fffff801429ce8c2 : 0000000000000005 ffffe00196ae8c00
ffffe0019a438c60 0000000000000000 : nt!KiDispatchException+0x646
fffff80144469ed0 fffff801429ccdfe : 0000000000000000 0000000000000000
fffff80142b75180 0000000000000001 : nt!KiExceptionDispatch+0xc2
fffff8014446a0b0 fffff801429c7e0e : fffff8013b888dff ffffe001904411b0
ffffe00100000000 ffffe00196ae8800 : nt!KiGeneralProtectionFault+0xfe
fffff8014446a248 0000000000000000 : 0000000000000000 0000000000000000
0000000000000000 0000000000000000 : nt!KiDpcInterrupt+0x1de

STACK_COMMAND: kb

FOLLOWUP_IP:
nt!RtlpGetStackLimits+ee
fffff801`4293e552 cc int 3

SYMBOL_STACK_INDEX: 1

SYMBOL_NAME: nt!RtlpGetStackLimits+ee

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: nt

IMAGE_NAME: ntkrnlmp.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 550f41a6

BUCKET_ID_FUNC_OFFSET: ee

FAILURE_BUCKET_ID: 0xc4_91_nt!RtlpGetStackLimits

BUCKET_ID: 0xc4_91_nt!RtlpGetStackLimits

Followup: MachineOwner

I do not queue any DPCs. I have a worker thread to handle inverted call completions but nothing crazy. All I have right now are the memory dumps so I’ll have to setup the system for remote debugging and enable some logging. I’m sure it will occur again.

I figured that the error message was not probably not applicable which is unfortunate for me. In the meantime, are there any WinDbg commands I can try that might pull some more information from the memory dump?

Can you provide the output of the following:

  1. kv

  2. From the resulting kv output, find the trap frame listed next to
    nt!KiGeneralProtectionFault and execute:

.trap 0x



3. dps @rsp

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntdev...

I do not queue any DPCs. I have a worker thread to handle inverted call
completions but nothing crazy. All I have right now are the memory dumps so
I'll have to setup the system for remote debugging and enable some logging.
I'm sure it will occur again.

I figured that the error message was not probably not applicable which is
unfortunate for me. In the meantime, are there any WinDbg commands I can
try that might pull some more information from the memory dump?

Thank you very much. That definitely helped and allowed me to figure out the issue. I actually thought it was a different driver causing the problems but it turns out it was my minifilter’s PostRead function. So back to one of your original questions, yes I do have code running in the context of a DPC (although I didn’t think so at the time because with the other driver everything runs at PASSIVE). I was trying to do some things when IRQL == DISPATCH that shouldn’t be done at that level. A little FltDoCompletionProcessingWhenSafe and all is good again.

Just curious as to why this was a guaranteed crash on Win8.1 but I have never had an issue on Win7 and driver verifier never complained? I can’t imagine that I’ve never experienced a PostRead at DISPATCH on Win7 in all the time I’ve been developing/testing/using the driver.

I’d say bad luck…

Do you enable Driver Verifier on your driver and on FltMgr.sys? You really
need both in order to get proper coverage.

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntdev…

Thank you very much. That definitely helped and allowed me to figure out
the issue. I actually thought it was a different driver causing the
problems but it turns out it was my minifilter’s PostRead function. So back
to one of your original questions, yes I do have code running in the context
of a DPC (although I didn’t think so at the time because with the other
driver everything runs at PASSIVE). I was trying to do some things when
IRQL == DISPATCH that shouldn’t be done at that level. A little
FltDoCompletionProcessingWhenSafe and all is good again.

Just curious as to why this was a guaranteed crash on Win8.1 but I have
never had an issue on Win7 and driver verifier never complained? I can’t
imagine that I’ve never experienced a PostRead at DISPATCH on Win7 in all
the time I’ve been developing/testing/using the driver.

Can I know what is the reason to use this command?
Is RSP relevant to calling stack?
In my knowledge rsp is just a stack pointer to adjust the stack size for function’s local variable and function’s parameters.
If there is any document or book discusses about this, please let me know…

//dlcu

On x86 based architectures the return address for subroutine calls are
stored on the stack. For example, an x86 CALL instruction such as the
following:

CALL FOO

Is (effectively):

PUSH
JMP FOO

When the subroutine wants to return, it simply pops the return address of
the stack into the instruction pointer register.

The dps instruction dumps a series of pointers and attempts to locate the
nearest debugging symbols to the resulting addresses. If you use this
instruction with the stack pointer you can see return addresses on the
stack. For example, if you have this stack:

kd> k L4
ChildEBP RetAddr
eec8b3b8 80500cf0 nt!KiSwapContext+0x2e
eec8b3c4 804f9d72 nt!KiSwapThread+0x46
eec8b3ec ba54b246 nt!KeWaitForSingleObject+0x1c2
eec8b4d4 804ee129 Ntfs!NtfsFsdCreate+0x291

You can see the return addresses (plus a bunch of other stuff) if you dump
the raw stack with dps:

kd> dps @esp
eec8b3ac eec8b3ec
eec8b3b0 85864b18
eec8b3b4 ffdff120
eec8b3b8 eec8b438
eec8b3bc 80500cf0 nt!KiSwapThread+0x46
eec8b3c0 85864b88
eec8b3c4 85864b18
eec8b3c8 804f9d72 nt!KeWaitForSingleObject+0x1c2
eec8b3cc 804f9bb0 nt!KeWaitForSingleObject
eec8b3d0 00000000
eec8b3d4 85ad0c70
eec8b3d8 eec8b2b4
eec8b3dc 00000044
eec8b3e0 eec8b4c4
eec8b3e4 ba50e75b Ntfs!_except_handler3
eec8b3e8 00000000
eec8b3ec eec8b4d4
eec8b3f0 ba54b246 Ntfs!NtfsFsdCreate+0x291

An additional trick to this is that when a subroutine returns, it pops the
return address off the stack but does not scrub the resulting memory
location. Thus you can sometimes see the return addresses of unwound
function calls, which can be useful in determining what a thread was doing
prior to the crash.

That was my goal in this case, we expected some kind of stack corruption so
I was hoping to pick off fragments of the OPs code running on the unwound
portion of the stack. Or, at worst, to see some kind of pattern in the
corruption (i.e. all zeroes, string contents, etc.)

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntdev…



Can I know what is the reason to use this command?
Is RSP relevant to calling stack?
In my knowledge rsp is just a stack pointer to adjust the stack size for
function’s local variable and function’s parameters.
If there is any document or book discusses about this, please let me know…

//dlcu

xxxxx@hotmail.com wrote:

Can I know what is the reason to use this command?
Is RSP relevant to calling stack?
In my knowledge rsp is just a stack pointer to adjust the stack size for function’s local variable and function’s parameters.

Well, yes, but also for all the functions before it as well. The stack
is basically your history, showing how we got where we are. As such, it
is an incredibly useful tool. The traceback you get from the “kb”
command is nothing more than an interpreted version of what is at rsp.
Indeed, the stack can even be traced back to the user-mode originator.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.