Kernel mode trap and double fault

ayushmaanshrotriya · March 28, 2024, 10:16am

2: kd> k

Child-SP RetAddr Call Site

00 ffff8b00ad6dcd28 fffff804ef46e129 01 ffff8b00ad6dcd30 fffff804ef467e11 02 ffff8b00ad6dce70 fffff804ef1077c1 03 ffff888865478f80 fffff804ef0506e3 04 ffff8888654790d0 fffff804ef168a16 05 ffff888865479280 fffff804ef074471 06 ffff888865479350 fffff804ef073086 07 ffff8888654794d0 fffff804ef06c8d6 08 ffff888865479570 fffff804ef1934b2 09 ffff888865479650 fffff804ef1df9e2 0a ffff8888654796f0 fffff804ef1a1a22 0b ffff888865479730 fffff804ef16dd4b 0c ffff8888654797b0 fffff804ef15f4fc 0d ffff888865479830 fffff804ef15f5f7 0e ffff888865479890 fffff804ef0126ce 0f ffff8888654798d0 fffff804ef8ff2d1 10 ffff888865479910 fffff804ef8feb1b 11 ffff888865479a00 fffff80482cd43d8 12 (Inline Function) ---------------- 17 ffff888865479b70 0000000000000000 nt!KeBugCheckEx nt!KiBugCheckDispatch+0x69 nt!KiDoubleFaultAbort+0x351 nt!MiClearPfnImageVerified+0x31 nt!MiInsertPageInFreeOrZeroedList+0x313 nt!MiReturnPhysicalPoolPages+0x172 nt!MiClearNonPagedPtes+0x299 nt!MmFreePoolMemory+0x19a nt!RtlpHpSegMgrCommit+0x396 nt!RtlpHpSegPageRangeCommit+0x146 nt!RtlpHpSegLfhVsDecommit+0x52 nt!RtlpHpVsSubsegmentCommitPages+0x122 nt!RtlpHpVsContextFreeInternal+0x61b nt!RtlpHpVsContextFree+0x1ec nt!RtlpHpSegFreeInternal+0x5f nt!RtlpHpFreeHeapInternal+0xda nt!ExFreePoolWithTag+0x7a1 nt!ExFreePool+0xb releasing packet 13 ffff888865479a30 fffff804ef0e957d packet insertion 14 ffff888865479b30 0000000082cd456d nt!RtlDeleteElementGenericTable+0x6d 15 ffff888865479b60 ffffd60400000000 0x82cd456d 16 ffff888865479b68 ffff888865479ca0 0xffffd60400000000
0xffff8888`65479ca0

ayushmaanshrotriya · March 28, 2024, 10:18am

can anyone help with this crash? Looks like a poor memory allocation or some corner case hitted

Peter_Viscarola_OSR · March 28, 2024, 1:05pm

Hmmmm... can you post the output from "!analyze -v", please?

I'd say... pool corruption? Have you enabled Driver Verifier on this driver?

ayushmaanshrotriya · March 29, 2024, 4:38am

2: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

UNEXPECTED_KERNEL_MODE_TRAP (7f)
This means a trap occurred in kernel mode, and it's a trap of a kind
that the kernel isn't allowed to have/catch (bound trap) or that
is always instant death (double fault).  The first number in the
BugCheck params is the number of the trap (8 = double fault, etc)
Consult an Intel x86 family manual to learn more about what these
traps are. Here is a *portion* of those codes:
If kv shows a taskGate
        use .tss on the part before the colon, then kv.
Else if kv shows a trapframe
        use .trap on that value
Else
        .trap on the appropriate frame will show where the trap was taken
        (on x86, this will be the ebp that goes with the procedure KiTrap)
Endif
kb will then show the corrected stack.
Arguments:
Arg1: 0000000000000008, EXCEPTION_DOUBLE_FAULT
Arg2: ffff8b00ad6dce70
Arg3: ffff888865478f80
Arg4: fffff804ef1077c1

Debugging Details:
------------------

*** WARNING: Unable to verify timestamp for DRIVER.SYS

KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.mSec
    Value: 3312

    Key  : Analysis.Elapsed.mSec
    Value: 4506

    Key  : Analysis.IO.Other.Mb
    Value: 0

    Key  : Analysis.IO.Read.Mb
    Value: 0

    Key  : Analysis.IO.Write.Mb
    Value: 0

    Key  : Analysis.Init.CPU.mSec
    Value: 484

    Key  : Analysis.Init.Elapsed.mSec
    Value: 1821

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 91

    Key  : Bugcheck.Code.LegacyAPI
    Value: 0x1000007f

    Key  : Bugcheck.Code.TargetModel
    Value: 0x1000007f

    Key  : Failure.Bucket
    Value: 0x7f_8_WSFILTER!InsertAndScanPacket

    Key  : Failure.Hash
    Value: {4508ce2a-0fea-9358-8532-ca313a392216}

    Key  : Hypervisor.Enlightenments.ValueHex
    Value: 7417df84

    Key  : Hypervisor.Flags.AnyHypervisorPresent
    Value: 1

    Key  : Hypervisor.Flags.ApicEnlightened
    Value: 0

    Key  : Hypervisor.Flags.ApicVirtualizationAvailable
    Value: 1

    Key  : Hypervisor.Flags.AsyncMemoryHint
    Value: 0

    Key  : Hypervisor.Flags.CoreSchedulerRequested
    Value: 0

    Key  : Hypervisor.Flags.CpuManager
    Value: 1

    Key  : Hypervisor.Flags.DeprecateAutoEoi
    Value: 1

    Key  : Hypervisor.Flags.DynamicCpuDisabled
    Value: 1

    Key  : Hypervisor.Flags.Epf
    Value: 0

    Key  : Hypervisor.Flags.ExtendedProcessorMasks
    Value: 1

    Key  : Hypervisor.Flags.HardwareMbecAvailable
    Value: 1

    Key  : Hypervisor.Flags.MaxBankNumber
    Value: 0

    Key  : Hypervisor.Flags.MemoryZeroingControl
    Value: 0

    Key  : Hypervisor.Flags.NoExtendedRangeFlush
    Value: 0

    Key  : Hypervisor.Flags.NoNonArchCoreSharing
    Value: 1

    Key  : Hypervisor.Flags.Phase0InitDone
    Value: 1

    Key  : Hypervisor.Flags.PowerSchedulerQos
    Value: 0

    Key  : Hypervisor.Flags.RootScheduler
    Value: 0

    Key  : Hypervisor.Flags.SynicAvailable
    Value: 1

    Key  : Hypervisor.Flags.UseQpcBias
    Value: 0

    Key  : Hypervisor.Flags.Value
    Value: 55185662

    Key  : Hypervisor.Flags.ValueHex
    Value: 34a10fe

    Key  : Hypervisor.Flags.VpAssistPage
    Value: 1

    Key  : Hypervisor.Flags.VsmAvailable
    Value: 1

    Key  : Hypervisor.RootFlags.AccessStats
    Value: 1

    Key  : Hypervisor.RootFlags.CrashdumpEnlightened
    Value: 1

    Key  : Hypervisor.RootFlags.CreateVirtualProcessor
    Value: 1

    Key  : Hypervisor.RootFlags.DisableHyperthreading
    Value: 0

    Key  : Hypervisor.RootFlags.HostTimelineSync
    Value: 1

    Key  : Hypervisor.RootFlags.HypervisorDebuggingEnabled
    Value: 0

    Key  : Hypervisor.RootFlags.IsHyperV
    Value: 1

    Key  : Hypervisor.RootFlags.LivedumpEnlightened
    Value: 1

    Key  : Hypervisor.RootFlags.MapDeviceInterrupt
    Value: 1

    Key  : Hypervisor.RootFlags.MceEnlightened
    Value: 1

    Key  : Hypervisor.RootFlags.Nested
    Value: 0

    Key  : Hypervisor.RootFlags.StartLogicalProcessor
    Value: 1

    Key  : Hypervisor.RootFlags.Value
    Value: 1015

    Key  : Hypervisor.RootFlags.ValueHex
    Value: 3f7

    Key  : WER.OS.Branch
    Value: ge_release

    Key  : WER.OS.Version
    Value: 10.0.26058.1000


BUGCHECK_CODE:  7f

BUGCHECK_P1: 8

BUGCHECK_P2: ffff8b00ad6dce70

BUGCHECK_P3: ffff888865478f80

BUGCHECK_P4: fffff804ef1077c1

FILE_IN_CAB:  031624-16187-01.dmp

TAG_NOT_DEFINED_202b:  *** Unknown TAG in analysis list 202b


TRAP_FRAME:  ffff8b00ad6dce70 -- (.trap 0xffff8b00ad6dce70)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000000 rbx=0000000000000000 rcx=ffffa20007722cf0
rdx=000000000000000c rsi=0000000000000000 rdi=0000000000000000
rip=fffff804ef1077c1 rsp=ffff888865478f80 rbp=ffff888865479080
 r8=0000000000000002  r9=fffff804efc37cc0 r10=ffffd3e9f4fa7fff
r11=ffff8888654790c8 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0         nv up ei pl zr na po nc
nt!MiClearPfnImageVerified+0x31:
fffff804`ef1077c1 0f11442430      movups  xmmword ptr [rsp+30h],xmm0 ss:0018:ffff8888`65478fb0=????????????????????????????????
Resetting default scope

BLACKBOXBSD: 1 (!blackboxbsd)


BLACKBOXNTFS: 1 (!blackboxntfs)


BLACKBOXPNP: 1 (!blackboxpnp)


BLACKBOXWINLOGON: 1

CUSTOMER_CRASH_COUNT:  1

PROCESS_NAME:  System

STACK_TEXT:  
ffff8b00`ad6dcd28 fffff804`ef46e129     : 00000000`0000007f 00000000`00000008 ffff8b00`ad6dce70 ffff8888`65478f80 : nt!KeBugCheckEx
ffff8b00`ad6dcd30 fffff804`ef467e11     : afafafaf`afafafaf afafafaf`afafafaf afafafaf`afafafaf afafafaf`afafafaf : nt!KiBugCheckDispatch+0x69
ffff8b00`ad6dce70 fffff804`ef1077c1     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiDoubleFaultAbort+0x351
ffff8888`65478f80 fffff804`ef0506e3     : 00009884`1e0a0000 00009888`0000000d 00000000`00000000 00000000`00000000 : nt!MiClearPfnImageVerified+0x31
ffff8888`654790d0 fffff804`ef168a16     : ffffd3e9`00000001 00009888`00000002 00009884`1e0a0000 00009888`0000000d : nt!MiInsertPageInFreeOrZeroedList+0x313
ffff8888`65479280 fffff804`ef074471     : 00000000`00450000 ffffd3eb`00000002 fffff804`00000000 00002a4a`00000002 : nt!MiReturnPhysicalPoolPages+0x172
ffff8888`65479350 fffff804`ef073086     : 00000000`00004000 ffffd604`5d02c000 00000000`00004000 fffff804`00000001 : nt!MiClearNonPagedPtes+0x299
ffff8888`654794d0 fffff804`ef06c8d6     : ffffffff`fffffffe ffff8888`65479601 00000000`00002000 ffffd604`5d02c000 : nt!MmFreePoolMemory+0x19a
ffff8888`65479570 fffff804`ef1934b2     : ffffd604`29a00140 00000000`00000002 ffffffff`fffffffe 00000000`00000009 : nt!RtlpHpSegMgrCommit+0x396
ffff8888`65479650 fffff804`ef1df9e2     : 00009884`00000000 00000000`0000000b 00000000`00000009 00009888`00000002 : nt!RtlpHpSegPageRangeCommit+0x146
ffff8888`654796f0 fffff804`ef1a1a22     : 00000000`00000600 00000000`00000001 ffffd604`29a002c0 ffffd604`5d023000 : nt!RtlpHpSegLfhVsDecommit+0x52
ffff8888`65479730 fffff804`ef16dd4b     : 00000000`00000009 ffffd604`0000000a 00000000`00000000 ffffd604`29a002c0 : nt!RtlpHpVsSubsegmentCommitPages+0x122
ffff8888`654797b0 fffff804`ef15f4fc     : ffffd604`5d02c000 00000000`5d023000 ffffd604`29a00000 00000000`00000000 : nt!RtlpHpVsContextFreeInternal+0x61b
ffff8888`65479830 fffff804`ef15f5f7     : ffffd604`5d02c000 ffffd604`29a00000 ffffd604`29a00140 fffff804`ef8ff2d1 : nt!RtlpHpVsContextFree+0x1ec
ffff8888`65479890 fffff804`ef0126ce     : ffffd604`00002290 ffffd604`5d02c000 00000000`00000000 00000000`00000170 : nt!RtlpHpSegFreeInternal+0x5f
ffff8888`654798d0 fffff804`ef8ff2d1     : 00000000`00002280 ffff8888`65479999 00000000`00000000 01000000`00100000 : nt!RtlpHpFreeHeapInternal+0xda
ffff8888`65479910 fffff804`ef8feb1b     : ffffd604`5d02c000 ffffd604`5bb97348 ffffd604`29a00000 00000000`0000c3f1 : nt!ExFreePoolWithTag+0x7a1
ffff8888`65479a00 fffff804`82cd43d8     : 00000000`00000055 fffff804`ef8ff2d1 00000000`00000000 00000000`00000258 : nt!ExFreePool+0xb
ffff8888`65479a30 fffff804`ef0e957d     : fffff804`82ce8720 00000000`00000000 00000000`0000012c 00000000`0000c3ee : <driver insert packet func>[directory/file.c] 
ffff8888`65479b30 00000000`82cd456d     : ffffd604`00000000 ffff8888`65479ca0 00000000`00000000 00000000`00000258 : nt!RtlDeleteElementGenericTable+0x6d
ffff8888`65479b60 ffffd604`00000000     : ffff8888`65479ca0 00000000`00000000 00000000`00000258 01bb9806`c3ee0000 : 0x82cd456d
ffff8888`65479b68 ffff8888`65479ca0     : 00000000`00000000 00000000`00000258 01bb9806`c3ee0000 00000000`00989680 : 0xffffd604`00000000
ffff8888`65479b70 00000000`00000000     : 00000000`00000258 01bb9806`c3ee0000 00000000`00989680 fffff804`82ce8768 : 0xffff8888`65479ca0


FAULTING_SOURCE_LINE:  directory/file.c

FAULTING_SOURCE_FILE:  directory/file.c

FAULTING_SOURCE_LINE_NUMBER:  2711

FAULTING_SOURCE_CODE:  
No source found for 'directory/file.c'


SYMBOL_NAME:  <driver insert packet function>

MODULE_NAME: <driver name>

IMAGE_NAME: <driver.sys>

STACK_COMMAND:  .cxr; .ecxr ; kb

BUCKET_ID_FUNC_OFFSET:  3f4

FAILURE_BUCKET_ID:  0x7f_8_<driver insert packet function>

OS_VERSION:  10.0.26058.1000

BUILDLAB_STR:  ge_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

FAILURE_ID_HASH:  {4508ce2a-0fea-9358-8532-ca313a392216}

Followup:     MachineOwner
---------

ayushmaanshrotriya · March 29, 2024, 4:41am

driver verifier was not enabled

Scott_Noone_OSR · April 1, 2024, 2:47pm

It's a double fault because you ran out of stack space. But the stack looks messed up at the beginning:

ffff8888`65479a30 fffff804`ef0e957d     : fffff804`82ce8720 00000000`00000000 00000000`0000012c 00000000`0000c3ee : <driver insert packet func>[directory/file.c] 
ffff8888`65479b30 00000000`82cd456d     : ffffd604`00000000 ffff8888`65479ca0 00000000`00000000 00000000`00000258 : nt!RtlDeleteElementGenericTable+0x6d
ffff8888`65479b60 ffffd604`00000000     : ffff8888`65479ca0 00000000`00000000 00000000`00000258 01bb9806`c3ee0000 : 0x82cd456d
ffff8888`65479b68 ffff8888`65479ca0     : 00000000`00000000 00000000`00000258 01bb9806`c3ee0000 00000000`00989680 : 0xffffd604`00000000
ffff8888`65479b70 00000000`00000000     : 00000000`00000258 01bb9806`c3ee0000 00000000`00989680 fffff804`82ce8768 : 0xffff8888`65479ca0

The debugger should be able to walk back from RtlDeleteElementGenericTable but it goes off into the weeds.

Turn Verifier on for your driver. If that doesn't show anything different then turn it on for ntoskrnl.exe in addition to your driver.

ayushmaanshrotriya · April 2, 2024, 9:09am

how to know if I ran out of stack space?
And I cannot reproduce this bug.
I have another dump for the same driver with double fault and slight different stack frame

Scott_Noone_OSR · April 2, 2024, 3:52pm

Mostly a guess because that's the usual cause of a double fault and your stack is all messed up. The faulting instruction is also because of the stack pointer:

Possibly a buffer overrun somewhere unless you're doing weird stuff with the stack (or writing assembly routines). Run Verifier.

Scott_Noone_OSR · April 2, 2024, 3:53pm

Oh, and you can confirm if it's an overflow by running !thread and looking at the stack limits:


12: kd> !thread
THREAD ffffe3856f04a080  Cid 4274.4290  Teb: 00000068b5e02000 Win32Thread: ffffe38570c954b0 RUNNING on processor c
...
Base ffff928d38910000 Limit ffff928d38909000 Call 0000000000000000

Tim_Roberts · April 2, 2024, 10:33pm

Kernel drivers have extremely limited stack space. You can do desk checking of your driver to make sure you're not allocating some large structures on the stack, where "large" means multiple kilobytes.

ayushmaanshrotriya · April 11, 2024, 11:20am

thanks all for replying