Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Sept/Oct 2019 Issue of The NT Insider available


Download PDF here: http://insider.osr.com/2019/ntinsider_2019_01.pdf

It’s a particularly BIG issue, too: 40 pages of technical goodness, ranging from WDF to Minifilters. Check it out.
Before Posting...
Please check out the Community Guidelines in the Announcements and Administration Category.

WFP callout driver is crashing; I have the stack trace

OSR_Community_UserOSR_Community_User Member Posts: 110,217
not sure what's going on here...I am blocking HTTP packets and re-injecting from the stream layer

# Child-SP RetAddr Call Site
00 fffff800`f8902f08 fffff800`f6ffb98e nt!DbgBreakPointWithStatus
01 fffff800`f8902f10 fffff800`f6ffb29f nt!KiBugCheckDebugBreak+0x12
02 fffff800`f8902f70 fffff800`f6f6c3a4 nt!KeBugCheck2+0x8ab
03 fffff800`f8903680 fffff800`f6f77de9 nt!KeBugCheckEx+0x104
04 fffff800`f89036c0 fffff800`f6f7663a nt!KiBugCheckDispatch+0x69
05 fffff800`f8903800 fffff801`590d3192 nt!KiPageFault+0x23a
06 fffff800`f8903990 fffff801`590d1e52 tcpip!TcpBeginTcbSend+0x732
07 fffff800`f8903c70 fffff801`590f44ab tcpip!TcpTcbSend+0x226
08 fffff800`f8903fc0 fffff801`590c991c tcpip!TcpFlushDelay+0x20a
09 fffff800`f8904070 fffff801`590c3423 tcpip!TcpPreValidatedReceive+0x3cc
0a fffff800`f8904170 fffff801`590f6e32 tcpip!IpFlcReceivePreValidatedPackets+0x649
0b fffff800`f8904350 fffff800`f6ec5fc3 tcpip!FlReceiveNetBufferListChainCalloutRoutine+0x102
0c fffff800`f8904480 fffff801`590f7076 nt!KeExpandKernelStackAndCalloutInternal+0xf3
0d fffff800`f8904570 fffff801`58eaea53 tcpip!FlReceiveNetBufferListChain+0xb6
0e fffff800`f89045f0 fffff801`58eaee7f ndis!ndisMIndicateNetBufferListsToOpen+0x123
0f fffff800`f89046b0 fffff801`58eaf6b2 ndis!ndisMTopReceiveNetBufferLists+0x22f
10 fffff800`f8904740 fffff801`59da11c4 ndis!NdisMIndicateReceiveNetBufferLists+0x732
11 fffff800`f8904930 fffff801`59da1a9d e1i63x64!RECEIVE::RxIndicateNBLs+0xd4
12 fffff800`f8904970 fffff801`59d94150 e1i63x64!RECEIVE::RxProcessInterrupts+0x19d
13 fffff800`f89049f0 fffff801`59d9457e e1i63x64!INTERRUPT::MsgIntDpcTxRxProcessing+0x1c0
14 fffff800`f8904a60 fffff801`59d93b78 e1i63x64!INTERRUPT::MsgIntMessageInterruptDPC+0x13e
15 fffff800`f8904ac0 fffff801`58eb0e12 e1i63x64!INTERRUPT::MiniportMessageInterruptDPC+0x28
16 fffff800`f8904b00 fffff800`f6e56910 ndis!ndisInterruptDpc+0x1a3
17 fffff800`f8904be0 fffff800`f6e55c57 nt!KiExecuteAllDpcs+0x1b0
18 fffff800`f8904d30 fffff800`f6f6f3d5 nt!KiRetireDpcList+0xd7
19 fffff800`f8904fb0 fffff800`f6f6f1d9 nt!KxRetireDpcList+0x5
1a ffffd000`d7570970 fffff800`f6f712fa nt!KiDispatchInterruptContinue
1b ffffd000`d75709a0 fffff800`f6ed6519 nt!KiDpcInterrupt+0xca
1c ffffd000`d7570b30 fffff800`f6ed5f69 nt!KiSwapThread+0x179
1d ffffd000`d7570bd0 fffff800`f6ed273d nt!KiCommitThreadWait+0x129
1e ffffd000`d7570c50 fffff800`f6f18c10 nt!ExpWorkerThread+0x3ad
1f ffffd000`d7570d00 fffff800`f6f728c6 nt!PspSystemThreadStartup+0x58
20 ffffd000`d7570d60 00000000`00000000 nt!KiStartSystemThread+0x16

Comments

  • Slava_ImameevSlava_Imameev Member Posts: 480
    "!analyze -v" command output would be more helpful and !pte command on the invalid virtual address that CPU tried to access.

    Also, the crash happened at DPC IRQL so be sure you are not injecting something allocated from PagedPool and not locked by MmProbeAndLockPages or smth.
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    hi...

    kd> !analyze -v
    *******************************************************************************
    * *
    * Bugcheck Analysis *
    * *
    *******************************************************************************

    DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
    An attempt was made to access a pageable (or completely invalid) address at an
    interrupt request level (IRQL) that is too high. This is usually
    caused by drivers using improper addresses.
    If kernel debugger is available get stack backtrace.
    Arguments:
    Arg1: 000000000000003c, memory referenced
    Arg2: 0000000000000002, IRQL
    Arg3: 0000000000000001, value 0 = read operation, 1 = write operation
    Arg4: fffff801590d3192, address which referenced memory

    Debugging Details:
    ------------------


    "KERNELBASE.dll" was not found in the image list.
    Debugger will attempt to load "KERNELBASE.dll" at given base 00000000`00000000.

    Please provide the full image name, including the extension (i.e. kernel32.dll)
    for more reliable results.Base address and size overrides can be given as
    .reload <image.ext>=<base>,<size>.
    Unable to add module at 00000000`00000000

    BUGCHECK_P1: 3c

    BUGCHECK_P2: 2

    BUGCHECK_P3: 1

    BUGCHECK_P4: fffff801590d3192

    WRITE_ADDRESS: 000000000000003c

    CURRENT_IRQL: 2

    FAULTING_IP:
    tcpip!TcpBeginTcbSend+732
    fffff801`590d3192 f0ff403c lock inc dword ptr [rax+3Ch]

    CPU_COUNT: 1

    CPU_MHZ: 6a0

    CPU_VENDOR: GenuineIntel

    CPU_FAMILY: 6

    CPU_MODEL: 45

    CPU_STEPPING: 1

    DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT

    BUGCHECK_STR: AV

    PROCESS_NAME: chrome.exe

    ANALYSIS_VERSION: 10.0.10240.9 amd64fre

    DPC_STACK_BASE: FFFFF800F8904FB0

    TRAP_FRAME: 0000000000a0dd0c -- (.trap 0xa0dd0c)
    NOTE: The trap frame does not contain all registers.
    Some register values may be zeroed or incorrect.
    rax=0000000000000000 rbx=0000000000000000 rcx=0000000000000000
    rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
    rip=1a4a5ea100000000 rsp=1a4a23350d82d890 rbp=1a4a23352058bcc9
    r8=0000000000000000 r9=0000000000000000 r10=0000000000000000
    r11=0000000000000000 r12=0000000000000000 r13=0000000000000000
    r14=0000000000000000 r15=0000000000000000
    iopl=3 vip vif nv up ei ng nz na po cy
    819d:0000 ?? ???
    Resetting default scope

    EXCEPTION_RECORD: 00a0dbf800bb2d28 -- (.exr 0xa0dbf800bb2d28)
    Cannot read Exception record @ 00a0dbf800bb2d28

    LAST_CONTROL_TRANSFER: from fffff800f6ffb98e to fffff800f6f72e90

    STACK_TEXT:
    fffff800`f8902f08 fffff800`f6ffb98e : 00000000`00000000 00000000`00000000 fffff800`f8903070 fffff800`f6ef27a4 : nt!DbgBreakPointWithStatus
    fffff800`f8902f10 fffff800`f6ffb29f : 00000000`00000003 fffff800`f8903070 fffff800`f6f7a290 00000000`000000d1 : nt!KiBugCheckDebugBreak+0x12
    fffff800`f8902f70 fffff800`f6f6c3a4 : fffff800`f8903a48 00000000`00000081 00000000`00000007 00000000`00000000 : nt!KeBugCheck2+0x8ab
    fffff800`f8903680 fffff800`f6f77de9 : 00000000`0000000a 00000000`0000003c 00000000`00000002 00000000`00000001 : nt!KeBugCheckEx+0x104
    fffff800`f89036c0 fffff800`f6f7663a : 00000000`00000001 fffff800`f8903a48 00000000`00000000 00000000`00000000 : nt!KiBugCheckDispatch+0x69
    fffff800`f8903800 fffff801`590d3192 : 00000000`fffffffe ffffe000`7d738010 fffffd46`00000000 fffff801`58c05122 : nt!KiPageFault+0x23a
    fffff800`f8903990 fffff801`590d1e52 : ffffe000`7ddebbf0 00000000`00000000 ffffe000`7d096138 00000000`00000001 : tcpip!TcpBeginTcbSend+0x732
    fffff800`f8903c70 fffff801`590f44ab : 00000000`00000001 00000000`00000001 ffffe000`7e99eb10 ffffe000`81075df0 : tcpip!TcpTcbSend+0x226
    fffff800`f8903fc0 fffff801`590c991c : ffffe000`8057262c 00000000`0000e57e 00000000`00000000 00000000`00000000 : tcpip!TcpFlushDelay+0x20a
    fffff800`f8904070 fffff801`590c3423 : ffffe000`7e0b9d80 00000000`00005000 00000000`00004cc2 fffff800`f6ef4cc2 : tcpip!TcpPreValidatedReceive+0x3cc
    fffff800`f8904170 fffff801`590f6e32 : ffffe000`7e7382b0 fffff800`f8904600 00000000`00000006 fffff801`5a8e0006 : tcpip!IpFlcReceivePreValidatedPackets+0x649
    fffff800`f8904350 fffff800`f6ec5fc3 : 00000000`00000003 00000000`00000000 ffffe000`7e0e7e10 fffff800`f88ff000 : tcpip!FlReceiveNetBufferListChainCalloutRoutine+0x102
    fffff800`f8904480 fffff801`590f7076 : fffff801`590f6d30 fffff800`f89045a0 00000000`00000010 00000000`00000801 : nt!KeExpandKernelStackAndCalloutInternal+0xf3
    fffff800`f8904570 fffff801`58eaea53 : 00000000`00000000 fffff800`f8904651 00000000`00000003 fffff801`590d4550 : tcpip!FlReceiveNetBufferListChain+0xb6
    fffff800`f89045f0 fffff801`58eaee7f : 00000000`00000001 00000000`00000000 00000000`00000000 00000000`00000003 : ndis!ndisMIndicateNetBufferListsToOpen+0x123
    fffff800`f89046b0 fffff801`58eaf6b2 : ffffe000`7e9a11a0 00000000`00000001 fffff801`58ebb540 00000000`00000000 : ndis!ndisMTopReceiveNetBufferLists+0x22f
    fffff800`f8904740 fffff801`59da11c4 : ffffe000`80373000 fffff801`59da1efc ffffe000`80373e00 ffffe000`81099c20 : ndis!NdisMIndicateReceiveNetBufferLists+0x732
    fffff800`f8904930 fffff801`59da1a9d : 00000000`00000001 ffffe000`80f39df0 ffffe000`80373000 00000000`00000003 : e1i63x64!RECEIVE::RxIndicateNBLs+0xd4
    fffff800`f8904970 fffff801`59d94150 : 00000000`00000000 ffffe000`7dcb1bf0 00000000`00000000 ffff0001`00000000 : e1i63x64!RECEIVE::RxProcessInterrupts+0x19d
    fffff800`f89049f0 fffff801`59d9457e : ffffe000`7dcb1bf0 ffffe000`80373000 ffff0001`00000000 ffff0001`00000000 : e1i63x64!INTERRUPT::MsgIntDpcTxRxProcessing+0x1c0
    fffff800`f8904a60 fffff801`59d93b78 : fffff800`f8904b79 ffff0001`00000000 00000000`00000000 ffffe000`7e9a11a0 : e1i63x64!INTERRUPT::MsgIntMessageInterruptDPC+0x13e
    fffff800`f8904ac0 fffff801`58eb0e12 : 00000000`00000000 fffff801`596eed08 ffffe000`80ed9402 fffff800`f6e56e17 : e1i63x64!INTERRUPT::MiniportMessageInterruptDPC+0x28
    fffff800`f8904b00 fffff800`f6e56910 : 00000000`00000000 fffff800`f6e1e000 fffff800`f70d2480 ffffe000`7e080f44 : ndis!ndisInterruptDpc+0x1a3
    fffff800`f8904be0 fffff800`f6e55c57 : 00000000`00000000 ffffe000`7e43e080 fffff800`f711b180 00000000`00000000 : nt!KiExecuteAllDpcs+0x1b0
    fffff800`f8904d30 fffff800`f6f6f3d5 : 00000000`00000000 fffff800`f711b180 fffff800`f75fe900 00000000`00bb3c74 : nt!KiRetireDpcList+0xd7
    fffff800`f8904fb0 fffff800`f6f6f1d9 : 00000205`1de0fabc fffff800`f6f71431 00000000`01000010 00000000`00000286 : nt!KxRetireDpcList+0x5
    ffffd000`d8d52bc0 fffff800`f6f71445 : ffffd000`d8d52c80 fffff800`f6f6db87 00000000`00000001 00000000`00000001 : nt!KiDispatchInterruptContinue
    ffffd000`d8d52bf0 fffff800`f6f6db87 : 00000000`00000001 00000000`00000001 00000000`00000001 ffffe000`81408060 : nt!KiDpcInterruptBypass+0x25
    ffffd000`d8d52c00 00000000`6cbc4cc1 : 00a0dbf8`00bb2d28 00a0dd28`00a0dd10 00000000`00a0dd0c 6cbc4abc`00a0db60 : nt!KiInterruptDispatchLBControl+0x197
    00000000`00a0d99c 00a0dbf8`00bb2d28 : 00a0dd28`00a0dd10 00000000`00a0dd0c 6cbc4abc`00a0db60 4c990488`00bb3c74 : chrome_child!ChromeMain+0x250fda
    00000000`00a0d9a4 00a0dd28`00a0dd10 : 00000000`00a0dd0c 6cbc4abc`00a0db60 4c990488`00bb3c74 00bb2d28`00a0dbf8 : 0x00a0dbf8`00bb2d28
    00000000`00a0d9ac 00000000`00a0dd0c : 6cbc4abc`00a0db60 4c990488`00bb3c74 00bb2d28`00a0dbf8 00a0d9c8`6ebbd3cc : 0x00a0dd28`00a0dd10
    00000000`00a0d9b4 6cbc4abc`00a0db60 : 4c990488`00bb3c74 00bb2d28`00a0dbf8 00a0d9c8`6ebbd3cc 00000000`00bb2d28 : 0xa0dd0c
    00000000`00a0d9bc 4c990488`00bb3c74 : 00bb2d28`00a0dbf8 00a0d9c8`6ebbd3cc 00000000`00bb2d28 00000000`00000000 : 0x6cbc4abc`00a0db60
    00000000`00a0d9c4 00bb2d28`00a0dbf8 : 00a0d9c8`6ebbd3cc 00000000`00bb2d28 00000000`00000000 6ebbd3f4`00000000 : 0x4c990488`00bb3c74
    00000000`00a0d9cc 00a0d9c8`6ebbd3cc : 00000000`00bb2d28 00000000`00000000 6ebbd3f4`00000000 00bb2d28`00a0d9c8 : 0x00bb2d28`00a0dbf8
    00000000`00a0d9d4 00000000`00bb2d28 : 00000000`00000000 6ebbd3f4`00000000 00bb2d28`00a0d9c8 00000000`00000000 : 0x00a0d9c8`6ebbd3cc
    00000000`00a0d9dc 00000000`00000000 : 6ebbd3f4`00000000 00bb2d28`00a0d9c8 00000000`00000000 00000000`00000000 : 0xbb2d28


    STACK_COMMAND: kb

    FOLLOWUP_IP:
    e1i63x64!RECEIVE::RxIndicateNBLs+d4
    fffff801`59da11c4 40f6c702 test dil,2

    SYMBOL_STACK_INDEX: 11

    SYMBOL_NAME: e1i63x64!RECEIVE::RxIndicateNBLs+d4

    FOLLOWUP_NAME: MachineOwner

    MODULE_NAME: e1i63x64

    IMAGE_NAME: e1i63x64.sys

    DEBUG_FLR_IMAGE_TIMESTAMP: 51496739

    BUCKET_ID_FUNC_OFFSET: d4

    FAILURE_BUCKET_ID: AV_e1i63x64!RECEIVE::RxIndicateNBLs

    BUCKET_ID: AV_e1i63x64!RECEIVE::RxIndicateNBLs

    PRIMARY_PROBLEM_CLASS: AV_e1i63x64!RECEIVE::RxIndicateNBLs

    ANALYSIS_SOURCE: KM

    FAILURE_ID_HASH_STRING: km:av_e1i63x64!receive::rxindicatenbls

    FAILURE_ID_HASH: {2cf978e1-1c85-263b-b2c7-d8a8f2358ee6}

    Followup: MachineOwner
    ---------



    I am trying to see how to work the !pte, i have never used it
  • Slava_ImameevSlava_Imameev Member Posts: 480
    No need for !pte. The address is 000000000000003c which is always invalid.

    The good news is that both crashes have the same pattern ( TcpBeginTcbSend ) so it would be easy to isolate the culprit code. The damage happened before KiExecuteAllDpcs was called, probably when a packet was injected. Try to play with re-injection to isolate the code that precede crashing, e.g. reinject for a single process ( Chrome e.g. ), introduce a global lock to serialize reinjection( only for debug ).
Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Upcoming OSR Seminars
Writing WDF Drivers 21 Oct 2019 OSR Seminar Space & ONLINE
Internals & Software Drivers 18 Nov 2019 Dulles, VA
Kernel Debugging 30 Mar 2020 OSR Seminar Space
Developing Minifilters 27 Apr 2020 OSR Seminar Space & ONLINE