Debugging crash dump -- Bad buffer passed by user-mode process

WINDBG
1

Hi,

I don’t have a lot of experience with debugging and I could use some help here.

I have a crash dump that shows a Bugcheck D1.
!analyze -v indicates that the faulting process is our user-mode process.
The kernel call stack shows the IRP contents to be NULL right from the point that our topmost volume filter driver gets a chance to process it.
I don’t even know what IRP that is as it’s not of interest to our driver since it is forwarding it further without even looking at it.

Is it possible to find the fault in the user mode process from the crash dump?
It feels like it’s passing a bad buffer to the IOCTL.

The bottom-most entry in the call stack displays an address that probably is the switch from user to kernel mode.

Can I fetch the return address (for this last entry in the kernel stack) which would most likely point to a function in my user process?

I’m just thinking out loud here and would appreciate if someone could point me in the right direction.

2

If you post the output we might be able to provide you with greater insight. It can’t be the UM process that causes a KM crash - the kernel doesn’t “trust” UM code anyway, so it’s responsible for protecting against obvious errors.

What do you mean the “IRP contents to be NULL”? Fair warning: if you are on x64 and looking at the arguments displayed on a stack backtrace, they aren’t valid (because the parameters are passed in registers now). So if you’re saying “the second parameter on the stack is NULL” then it’s a red herring. The !analyze -v output should provide more information though.

Tony
OSR

3

Yeah it is a 64-bit system.
I selected the function entry in the Calls window and then went to the Locals window to check the IRP.
I found a NULL value to IRP->Tail.Overlay.CurrentStackLocation.

Wouldn’t Windbg automatically fetch and initialize this parameter from the register knowing that it’s a 64-bit dump?

I further found out the faulting thread and ran !thread on it. I notice that there’s a “Win32 Start Address” field that holds the address of the user function.

Can we not get to peek at the user function given that we know it’s address in memory as we have the symbols and the sources?
This is a complete memory dump so I thought there’s got to be some way to reach to the user mode process and look at the calling function.

Please advise.

P.S. Unfortunately I cannot share the !analyze -v output here due to an NDA. I so much wish I could.

4

If you are using 64-bit release build of your driver then windbg will not
be able to show the right values of the arguments since they are not saved
on the stack home space. If you understand x64 calling convention then you
can get that value if it was stored on the stack by some function. A
quicker way would be to use* !stack -p* command from CMKD debugger
extension by codemachine to see the actual values of arguments.
http://www.codemachine.com/tool_cmkd.html

Yes it is possible, if you are in the right process context and have given
the right symbol path for your user-mode process, then doing *.reload /user*
will help in loading the user mode symbols. Hope this information helps
you. Without !analyze -v output there is not much I can say.

On Wed, Jun 8, 2016 at 10:52 PM, wrote:

> Yeah it is a 64-bit system.
> I selected the function entry in the Calls window and then went to the
> Locals window to check the IRP.
> I found a NULL value to IRP->Tail.Overlay.CurrentStackLocation.
>
> Wouldn’t Windbg automatically fetch and initialize this parameter from the
> register knowing that it’s a 64-bit dump?
>
> I further found out the faulting thread and ran !thread on it. I notice
> that there’s a “Win32 Start Address” field that holds the address of the
> user function.
>
> Can we not get to peek at the user function given that we know it’s
> address in memory as we have the symbols and the sources?
> This is a complete memory dump so I thought there’s got to be some way to
> reach to the user mode process and look at the calling function.
>
> Please advise.
>
> P.S. Unfortunately I cannot share the !analyze -v output here due to an
> NDA. I so much wish I could.
>
>
> —
> WINDBG is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

5

xxxxx@gmail.com wrote:

Yeah it is a 64-bit system.
I selected the function entry in the Calls window and then went to the Locals window to check the IRP.
I found a NULL value to IRP->Tail.Overlay.CurrentStackLocation.

Wouldn’t Windbg automatically fetch and initialize this parameter from the register knowing that it’s a 64-bit dump?

Many people overestimate the clairvoyance of WinDbg. You should doubt
any surprising result until you can check it yourself. Unless it’s a
debug build, it may be impossible for it to find the locals.

I further found out the faulting thread and ran !thread on it. I notice that there’s a “Win32 Start Address” field that holds the address of the user function.

That holds the address of the starting function for the thread.

Can we not get to peek at the user function given that we know it’s address in memory as we have the symbols and the sources?
This is a complete memory dump so I thought there’s got to be some way to reach to the user mode process and look at the calling function.

Sometimes you can. WinDbg does have the ability to poke around in user
mode, but remember that only the currently actives process is accessible
through the page tables. Unless your thread is currently active in one
of the processors, there’s no (good) way to find the memory for that
process.

P.S. Unfortunately I cannot share the !analyze -v output here due to an NDA. I so much wish I could.

That’s highly doubtful. What protected information do you think you’re
going to find? If the driver name reveals a vendor, then just use a
text editor to change it to xxxxx.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

6

Thanks a lot for your inputs everyone… Here’s the output for the “!analyze -v” command.
Thanks for the tip Tim. :slight_smile:

*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: 0000000000000030, memory referenced
Arg2: 0000000000000002, IRQL
Arg3: 0000000000000000, value 0 = read operation, 1 = write operation
Arg4: fffff880014c7ed2, address which referenced memory

Debugging Details:

READ_ADDRESS: 0000000000000030

CURRENT_IRQL: 2

FAULTING_IP:

READ_ADDRESS: 0000000000000030

CURRENT_IRQL: 2

FAULTING_IP:
[XXXXDriver1]!ScsiWmiFunc2+5a
fffff880`014c7ed2 458b5a30 mov r11d,dword ptr [r10+30h]

DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT

BUGCHECK_STR: 0xD1

PROCESS_NAME: [YYYUserProcess]

TRAP_FRAME: fffff880092aded0 – (.trap 0xfffff880092aded0)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=fffff8800892a3bc rbx=0000000000000000 rcx=fffff8800892a300
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=fffff880014c7ed2 rsp=fffff880092ae068 rbp=fffffa804acc0008
r8=0000000000000001 r9=0000000000000088 r10=0000000000000000
r11=fffff880092adf30 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
[XXXXDriver1]!ScsiWmiFunc2+0x5a:
fffff880014c7ed2 458b5a30 mov r11d,dword ptr [r10+30h] ds:0000000000000030=???
Resetting default scope

LAST_CONTROL_TRANSFER: from fffff80001880be9 to fffff80001881640

STACK_TEXT:
fffff880092add88 fffff80001880be9 : 000000000000000a 0000000000000030 0000000000000002 0000000000000000 : nt!KeBugCheckEx
fffff880092add90 fffff8000187f860 : 3830303063636134 0000000000000000 0000000000000000 0000000000000000 : nt!KiBugCheckDispatch+0x69
fffff880092aded0 fffff880014c7ed2 : fffff880014c8435 fffff8800892a3a0 0000000000000088 ffff000008642f46 : nt!KiPageFault+0x260
fffff880092ae068 fffff880014c8435 : fffff8800892a3a0 0000000000000088 ffff000008642f46 0000000000000088 : [XXXXDriver1]!ScsiWmiFunc2+0x5a
fffff880092ae070 fffff880014ad977 : 0000000000000001 fffff880014e3db0 0000000000000001 fffffa804acc0008 : [XXXXDriver1]!ScsiWmiFunc1+0x29
fffff880092ae0b0 fffff880014c81e1 : fffffa8000000088 fffff8800892a3a0 fffff8a000000002 fffffa804acc0160 : [XXXXDriver1]!FetchWmiQueryInfo+0x257
fffff880092ae140 fffff880014c84bb : 0000000000000000 0000000000000002 0000000000000001 fffffa804acc0008 : [XXXXDriver1]!ScsiWmiRequest+0x1ed
fffff880092ae1b0 fffff880014ad4d6 : fffff8800892a390 0000000000000000 0000000000000001 fffff6fc400310c0 : [XXXXDriver1]!DispatchWmi+0x77
fffff880092ae220 fffff8800149224f : 0000000000000002 0000000000000000 fffffa804acc6d28 00000000000000fe : [XXXXDriver1]!WmiSrbEx+0xa6
fffff880092ae270 fffff8800157a269 : fffffa804ab4f101 fffffa804ab400fe 0000000000000000 fffffa804acc3028 : [XXXXDriver1]!MyStartIo+0xea3
fffff880092ae3b0 fffff80001def62f : fffffa80ac391d20 0000000000000000 fffff8800892a0b8 fffff8000198afe9 : storport!RaidpAdapterContinueScatterGather+0x149
fffff880092ae420 fffff8800158fb1b : 0000000000000400 fffffa804ab4f1b0 fffff880092ae600 0000000000000000 : hal!HalBuildScatterGatherList+0x203
fffff880092ae490 fffff880015902b2 : 0000000000000000 fffff8800892a000 fffffa804ab4f1b0 fffff88001597110 : storport!RaidAdapterScatterGatherExecute+0x8b
fffff880092ae4f0 fffff880015902e8 : fffff8800892a010 0000000000000400 0000000000000000 0000000000000400 : storport!RaidAdapterExecuteXrb+0x22
fffff880092ae520 fffff880015d0abf : 0000000000000001 0000000000000000 ffffbb8c053ff4e0 fffff8a000238400 : storport!RaidAdapterRaiseIrqlAndExecuteXrb+0x18
fffff880092ae550 fffff880015d0d49 : fffff8800892a000 fffffa805434c501 fffff880092ae6e8 fffffa804ab4f060 : storport!RaWmiPassToMiniPort+0x27f
fffff880092ae5b0 fffff880015d0ed0 : fffffa805434c510 fffff80000000008 fffff880092ae740 0000000000000042 : storport!RaWmiIrpNormalRequest+0x49
fffff880092ae5e0 fffff880015d0fa7 : fffff880092ae6e8 fffffa805434c510 fffff88001597110 fffffa804ab4f060 : storport!RaWmiDispatchIrp+0x140
fffff880092ae640 fffff88001d87e99 : fffffa805434c510 fffffa805434c510 fffffa805434c510 fffff880092ae810 : storport!RaDriverSystemControlIrp+0x67
fffff880092ae680 fffff88001d87498 : fffffa804ab4fd00 fffffa80000e0082 fffffa80542a4001 fffffa805434c510 : ZZZZScsiFlt!SFiltDeviceExt::SendToNextDriver+0x69
fffff880092ae6c0 fffff80001b1b6c4 : fffff88000000000 fffffa804ab4f060 fffffa804ab4f060 fffffa80542a4001 : ZZZZScsiFlt!SFiltDriver::PtDispatchSysCtrl+0xe4
fffff880092ae730 fffff80001c54933 : 0000000000000000 0000000000000000 fffffa805434c540 fffffa80542a4001 : nt!WmipForwardWmiIrp+0x16c
fffff880092ae7b0 fffff80001bc11eb : fffffa804d164500 fffffa804bdac4b0 0000000000000400 0000000000000000 : nt!WmipQuerySetExecuteSI+0x293
fffff880092ae8c0 fffff80001b9bf97 : fffffa804bdac4b0 fffff880092aeca0 fffff880092aeca0 fffffa805434c510 : nt! ?? ::NNGAKEGL::string'+0x319e9 fffff880092aea10 fffff80001b9c7f6 : 0000000000000000 0000000000000988 0000000000000000 0000000000000000 : nt!IopXxxControlFile+0x607 fffff880092aeb40 fffff800018808d3 : 0000000000000000 fffffa804bbfda00 00000000746c6644 fffff880092aec38 : nt!NtDeviceIoControlFile+0x56 fffff880092aebb0 0000000076e9138a : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!KiSystemServiceCopyEnd+0x13 000000000780d928 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 00000000`00000000 : 0x76e9138a

STACK_COMMAND: kb

FOLLOWUP_IP:
[XXXXDriver1]!ScsiWmiFunc2+5a
fffff880`014c7ed2 458b5a30 mov r11d,dword ptr [r10+30h]

FAULTING_SOURCE_LINE: d:\storage\scsiwmi\wmilib.c

FAULTING_SOURCE_FILE: d:\storage\scsiwmi\wmilib.c

FAULTING_SOURCE_LINE_NUMBER: 132

FAULTING_SOURCE_CODE:
No source found for ‘d:\storage\scsiwmi\wmilib.c’

SYMBOL_STACK_INDEX: 3

SYMBOL_NAME: [XXXXDriver1]!ScsiWmiFunc2+5a

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: [XXXXDriver1]

IMAGE_NAME: [XXXXDriver1].sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4c7e90ef

FAILURE_BUCKET_ID: X64_0xD1_[XXXXDriver1]!ScsiWmiFunc2+5a

BUCKET_ID: X64_0xD1_[XXXXDriver1]!ScsiWmiFunc2+5a

Followup: MachineOwner

!ScsiWmiFunc2+5a [d:\storage\scsiwmi\wmilib.c @ 132]
fffff880`014c7ed2 458b5a30 mov r11d,dword ptr [r10+30h]

DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT

BUGCHECK_STR: 0xD1

PROCESS_NAME: [YYYUserProcess].exe

TRAP_FRAME: fffff880092aded0 – (.trap 0xfffff880092aded0)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=fffff8800892a3bc rbx=0000000000000000 rcx=fffff8800892a300
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=fffff880014c7ed2 rsp=fffff880092ae068 rbp=fffffa804acc0008
r8=0000000000000001 r9=0000000000000088 r10=0000000000000000
r11=fffff880092adf30 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
[XXXXDriver1]!ScsiWmiFunc2+0x5a:
fffff880014c7ed2 458b5a30 mov r11d,dword ptr [r10+30h] ds:0000000000000030=???
Resetting default scope

LAST_CONTROL_TRANSFER: from fffff80001880be9 to fffff80001881640

STACK_TEXT:
fffff880092add88 fffff80001880be9 : 000000000000000a 0000000000000030 0000000000000002 0000000000000000 : nt!KeBugCheckEx
fffff880092add90 fffff8000187f860 : 3830303063636134 0000000000000000 0000000000000000 0000000000000000 : nt!KiBugCheckDispatch+0x69
fffff880092aded0 fffff880014c7ed2 : fffff880014c8435 fffff8800892a3a0 0000000000000088 ffff000008642f46 : nt!KiPageFault+0x260
fffff880092ae068 fffff880014c8435 : fffff8800892a3a0 0000000000000088 ffff000008642f46 0000000000000088 : [XXXXDriver1]!ScsiWmiFunc2+0x5a [d:\storage\scsiwmi\wmilib.c @ 132]
fffff880092ae070 fffff880014ad977 : 0000000000000001 fffff880014e3db0 0000000000000001 fffffa804acc0008 : [XXXXDriver1]!ScsiWmiFunc1+0x29 [d:\storage\scsiwmi\wmilib.c @ 272]
fffff880092ae0b0 fffff880014c81e1 : fffffa8000000088 fffff8800892a3a0 fffff8a000000002 fffffa804acc0160 : [XXXXDriver1]!FetchWmiQueryInfo+0x257 [c:\stor\xxxxwmi.c @ 647]
fffff880092ae140 fffff880014c84bb : 0000000000000000 0000000000000002 0000000000000001 fffffa804acc0008 : [XXXXDriver1]!ScsiWmiRequest+0x1ed [d:\storage\scsiwmi\wmilib.c @ 522]
fffff880092ae1b0 fffff880014ad4d6 : fffff8800892a390 0000000000000000 0000000000000001 fffff6fc400310c0 : [XXXXDriver1]!DispatchWmi+0x77 [d:\storage\scsiwmi\wmilib.c @ 794]
fffff880092ae220 fffff8800149224f : 0000000000000002 0000000000000000 fffffa804acc6d28 00000000000000fe : [XXXXDriver1]!WmiSrbEx+0xa6 [c:\stor\xxxxwmi.c @ 480]
fffff880092ae270 fffff8800157a269 : fffffa804ab4f101 fffffa804ab400fe 0000000000000000 fffffa804acc3028 : [XXXXDriver1]!MyStartIo+0xea3 [c:\stor\xxxxport.c @ 3104]
fffff880092ae3b0 fffff80001def62f : fffffa80ac391d20 0000000000000000 fffff8800892a0b8 fffff8000198afe9 : storport!RaidpAdapterContinueScatterGather+0x149
fffff880092ae420 fffff8800158fb1b : 0000000000000400 fffffa804ab4f1b0 fffff880092ae600 0000000000000000 : hal!HalBuildScatterGatherList+0x203
fffff880092ae490 fffff880015902b2 : 0000000000000000 fffff8800892a000 fffffa804ab4f1b0 fffff88001597110 : storport!RaidAdapterScatterGatherExecute+0x8b
fffff880092ae4f0 fffff880015902e8 : fffff8800892a010 0000000000000400 0000000000000000 0000000000000400 : storport!RaidAdapterExecuteXrb+0x22
fffff880092ae520 fffff880015d0abf : 0000000000000001 0000000000000000 ffffbb8c053ff4e0 fffff8a000238400 : storport!RaidAdapterRaiseIrqlAndExecuteXrb+0x18
fffff880092ae550 fffff880015d0d49 : fffff8800892a000 fffffa805434c501 fffff880092ae6e8 fffffa804ab4f060 : storport!RaWmiPassToMiniPort+0x27f
fffff880092ae5b0 fffff880015d0ed0 : fffffa805434c510 fffff80000000008 fffff880092ae740 0000000000000042 : storport!RaWmiIrpNormalRequest+0x49
fffff880092ae5e0 fffff880015d0fa7 : fffff880092ae6e8 fffffa805434c510 fffff88001597110 fffffa804ab4f060 : storport!RaWmiDispatchIrp+0x140
fffff880092ae640 fffff88001d87e99 : fffffa805434c510 fffffa805434c510 fffffa805434c510 fffff880092ae810 : storport!RaDriverSystemControlIrp+0x67
fffff880092ae680 fffff88001d87498 : fffffa804ab4fd00 fffffa80000e0082 fffffa80542a4001 fffffa805434c510 : ZZZZScsiFlt!SFiltDeviceExt::SendToNextDriver+0x69 [e:\src\driver\scsibusf\fltdev.cpp @ 467]
fffff880092ae6c0 fffff80001b1b6c4 : fffff88000000000 fffffa804ab4f060 fffffa804ab4f060 fffffa80542a4001 : ZZZZScsiFlt!SFiltDriver::PtDispatchSysCtrl+0xe4 [e:\src\driver\scsibusf\fltdrv.cpp @ 419]
fffff880092ae730 fffff80001c54933 : 0000000000000000 0000000000000000 fffffa805434c540 fffffa80542a4001 : nt!WmipForwardWmiIrp+0x16c
fffff880092ae7b0 fffff80001bc11eb : fffffa804d164500 fffffa804bdac4b0 0000000000000400 0000000000000000 : nt!WmipQuerySetExecuteSI+0x293
fffff880092ae8c0 fffff80001b9bf97 : fffffa804bdac4b0 fffff880092aeca0 fffff880092aeca0 fffffa805434c510 : nt! ?? ::NNGAKEGL::string'+0x319e9 fffff880092aea10 fffff80001b9c7f6 : 0000000000000000 0000000000000988 0000000000000000 0000000000000000 : nt!IopXxxControlFile+0x607 fffff880092aeb40 fffff800018808d3 : 0000000000000000 fffffa804bbfda00 00000000746c6644 fffff880092aec38 : nt!NtDeviceIoControlFile+0x56 fffff880092aebb0 0000000076e9138a : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!KiSystemServiceCopyEnd+0x13 000000000780d928 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 00000000`00000000 : 0x76e9138a

STACK_COMMAND: kb

FOLLOWUP_IP:
[XXXXDriver1]!ScsiWmiFunc2+5a [d:\storage\scsiwmi\wmilib.c @ 132]
fffff880`014c7ed2 458b5a30 mov r11d,dword ptr [r10+30h]

FAULTING_SOURCE_LINE: d:\storage\scsiwmi\wmilib.c

FAULTING_SOURCE_FILE: d:\storage\scsiwmi\wmilib.c

FAULTING_SOURCE_LINE_NUMBER: 132

FAULTING_SOURCE_CODE:
No source found for ‘d:\storage\scsiwmi\wmilib.c’

SYMBOL_STACK_INDEX: 3

SYMBOL_NAME: [XXXXDriver1]!ScsiWmiFunc2+5a

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: [XXXXDriver1]

IMAGE_NAME: [XXXXDriver1].sys

DEBUG_FLR_IMAGE_TIMESTAMP: 4c7e90ef

FAILURE_BUCKET_ID: X64_0xD1_[XXXXDriver1]!ScsiWmiFunc2+5a

BUCKET_ID: X64_0xD1_[XXXXDriver1]!ScsiWmiFunc2+5a

Followup: MachineOwner

7

xxxxx@gmail.com wrote:

Thanks a lot for your inputs everyone… Here’s the output for the “!analyze -v” command.
Thanks for the tip Tim. :slight_smile:

OK, so this shows that you are referencing an offset from a null pointer
early in ScsiWmiFunc2, which had been called very early in
ScsiWmiFunc1. You’re fetching offset 0x30. If that’s an PIRP, then
it’s looking for pIrp->IoStatus.Status. If that’s a PIO_STACK_LOCATION,
then it’s looking for piosl->FileObject.

If this is an IRP, it’s odd to me that you would have called 7 levels
deep in your driver without touching the IRP previously.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.