Verifier bugcheck C4 with FireEye FeKern.sys

NTDEV
1

Greetings,

I?m looking for a gentle nudge in the right direction. I don?t typically debug kernel components, so my questions may be elementary in nature. Apologies in advance.

I?m trying to identify source of a paged pool leak. I?ve tracked down the tag in question (CMNb) to nt aka ntkrnlmp. Working under assumption that some 3rd party component is invoking NT API incorrectly (not cleaning up), I?ve enabled verifier on ntoskrnl and rebooted.

Sample configuration:
1: kd> !verifier 3

Verify Flags Level 0x00000008

STANDARD FLAGS:
(0x00000000) Automatic Checks
(0x00000001) Special pool
(0x00000002) Force IRQL checking
(0x00000008) Pool tracking
(0x00000010) I/O verification
(0x00000020) Deadlock detection
(0x00000080) DMA checking
(0x00000100) Security checks
(0x00000800) Miscellaneous checks

ADDITIONAL FLAGS:
(0x00000004) Randomized low resources simulation
(0x00000200) Force pending I/O requests
(0x00000400) IRP logging

Indicates flag is enabled

Summary of All Verifier Statistics

RaiseIrqls 0x0
AcquireSpinLocks 0x0
Synch Executions 0x0
Trims 0x0

Pool Allocations Attempted 0x954607
Pool Allocations Succeeded 0x954607
Pool Allocations Succeeded SpecialPool 0xbdcb
Pool Allocations With NO TAG 0x1f5b2
Pool Allocations Failed 0x0

Current paged pool allocations 0x149e9 for 0E6B2800 bytes
Peak paged pool allocations 0x14a40 for 165DA8B0 bytes
Current nonpaged pool allocations 0xdaa4 for 023356C0 bytes
Peak nonpaged pool allocations 0xdb5d for 023D6D70 bytes

Driver Verification List

MODULE: 0xfffffa800c942080 ntoskrnl.exe (Loaded)

Pool Allocation Statistics: ( NonPagedPool / PagedPool )

Current Pool Allocations: ( 0x0000daa4 / 0x000149e9 )
Current Pool Bytes: ( 0x023356c0 / 0x0e6b2800 )
Peak Pool Allocations: ( 0x0000db5d / 0x00014a40 )
Peak Pool Bytes: ( 0x023d6d70 / 0x165da8b0 )
Contiguous Memory Bytes: 0x00000000
Peak Contiguous Memory Bytes: 0x00000000

Upon reboot, verifier detects a problem as soon as login screen comes up, which prevents me from debugging the memory leak (can?t run for a long time and leak memory).
1: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught. This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this driver.
If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA will
be among the most commonly seen crashes.
Arguments:
Arg1: 0000000000000000, caller is trying to allocate zero bytes
Arg2: 0000000000000000, current IRQL
Arg3: 0000000000000000, pool type
Arg4: 0000000000000000, number of bytes

Debugging Details:

BUGCHECK_STR: 0xc4_0

CURRENT_IRQL: 2

DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT

PROCESS_NAME: svchost.exe

ANALYSIS_VERSION: 6.3.9600.16384 (debuggers(dbg).130821-1623) amd64fre

LAST_CONTROL_TRANSFER: from fffff80004363762 to fffff8000426ac70

STACK_TEXT:
fffff88009dcf858 fffff80004363762 : 0000000000000000 fffffa80148afb50 0000000000000065 fffff800042af644 : nt!DbgBreakPointWithStatus
fffff88009dcf860 fffff8000436454e : 0000000000000003 0000000000000000 fffff800042afea0 00000000000000c4 : nt!KiBugCheckDebugBreak+0x12
fffff88009dcf8c0 fffff80004272f44 : 0000000000000130 fffff88009dd1000 fffff88009dd02c8 fffff880017bca01 : nt!KeBugCheck2+0x71e
fffff88009dcff90 fffff800047054ec : 00000000000000c4 0000000000000000 0000000000000000 0000000000000000 : nt!KeBugCheckEx+0x104
fffff88009dcffd0 fffff80004705f2b : 0000000000000003 fffffa800cc48653 0000000000000000 0000000000000130 : nt!VerifierBugCheckIfAppropriate+0x3c
fffff88009dd0010 fffff80004716a58 : 0000000067727453 0000000000000080 0000000000000100 fffff80004715830 : nt!ExAllocatePoolSanityChecks+0xcb
fffff88009dd0050 fffff800043a8037 : fffff88009dd0290 0000000000000000 fffffa8067727453 fffff88009dd0210 : nt!VeAllocatePoolWithTagPriority+0x88
fffff88009dd00c0 fffff80004234061 : fffffa80158598a0 0000000000000000 0000000000000016 0000000000000402 : nt!ExFreePool+0x3b
fffff88009dd01b0 fffff80004262369 : fffff88009dd0210 0000000000000000 fffffa8000000401 fffff88009dd0250 : nt!RtlpUpcaseUnicodeStringPrivate+0x39
fffff88009dd01f0 fffff88008d98e3a : 0000000000000000 fffffa8015b52da0 0000000000000001 0000000000000000 : nt!RtlIsNameInExpression+0x59
fffff88009dd0230 fffff88008d98ec7 : 0000000000000000 fffffa8015d014e8 fffffa8015d015e8 fffffa8015d01401 : FeKern+0xae3a
fffff88009dd0270 fffff88008d93acf : fffffa8013bbba00 fffffa8013bbb930 fffffa8015d014e8 fffffa8015d01538 : FeKern+0xaec7
fffff88009dd02b0 fffff88008d93ea3 : 0000000101060001 0000000000000000 fffff88009dd03e8 fffff880017ae7e7 : FeKern+0x5acf
fffff88009dd0310 fffff880017ad288 : 0000000000001635 0000000000000000 fffffa801116fa38 fffffa801116fa38 : FeKern+0x5ea3
fffff88009dd03a0 fffff880017abd1b : fffffa80148f8010 fffffa8013bbb9d0 fffffa80158598a0 fffffa8015859ac0 : fltmgr!FltpPerformPostCallbacks+0x368
fffff88009dd0470 fffff880017cb2b9 : fffffa801116f800 fffffa8015a36010 fffffa801116f800 fffffa8015ae3040 : fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x39b
fffff88009dd0500 fffff80004721880 : 0000000000000000 0000000000000000 fffffa8000000040 0000000000000000 : fltmgr!FltpCreate+0x2a9
fffff88009dd05b0 fffff800045772bb : 0000000000000005 0000000000000040 fffffa8013b05550 fffffa8013b055e8 : nt!IovCallDriver+0xa0
fffff88009dd0610 fffff80004572dde : fffffa800e42cc60 0000000000000000 fffffa8014c2f670 0000000000000001 : nt!IopParseDevice+0x14e2
fffff88009dd0770 fffff800045738c6 : 0000000000000000 fffff88009dd08f0 fffff8a000000040 fffffa800d294420 : nt!ObpLookupObjectName+0x784
fffff88009dd0870 fffff800045756bc : 0000000003ebf9f0 0000000000000000 fffff8a06e566d01 fffff8a0135c53a0 : nt!ObOpenObjectByName+0x306
fffff88009dd0940 fffff8000455e7a8 : 0000000003ebf9b8 00000000000f01ff 0000000003ebf970 0000000003ebf960 : nt!IopCreateFile+0x2bc
fffff88009dd09e0 fffff800042720d3 : 0000000000000fa8 0000000000000000 0000000000000000 0000098000000003 : nt!NtOpenFile+0x58
fffff88009dd0a70 0000000077cdc06a : 000007feecdca7b1 0000000000000000 0000000077b7a54a 0000000000000000 : nt!KiSystemServiceCopyEnd+0x13
0000000003ebf8f8 000007feecdca7b1 : 0000000000000000 0000000077b7a54a 0000000000000000 0000000000000000 : ntdll!ZwOpenFile+0xa
0000000003ebf900 000007feecdca603 : 0000000000000fa8 000007feecdead40 0000000000000000 0000000000000000 : srvsvc!SsNotifyRdrOfGuid+0x7d
0000000003ebf9b0 000007feecdc4cfc : 000007feecdea890 0000000000000f70 0000000000000f54 0000000000000f54 : srvsvc!SsCheckRegistry+0x2da
0000000003ebfa20 000007feecdc3b31 : 0000000000000001 0000000000000001 0000000000000001 000007feecdf2180 : srvsvc!SsInitialize+0x37c
0000000003ebfcb0 00000000ff651344 : 000007feecdc8040 0000000000000000 000007feecdc8040 0000000000000000 : srvsvc!ServiceMain+0x2fc
0000000003ebfdc0 000007fefe39a82d : 0000000000000001 0000000002cd32f8 0000000000000000 000007feecdc8040 : svchost!ServiceStarter+0x1e8
0000000003ebfe50 0000000077b859cd : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : sechost!ScSvcctrlThreadA+0x25
0000000003ebfe80 0000000077cba561 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : kernel32!BaseThreadInitThunk+0xd
0000000003ebfeb0 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : ntdll!RtlUserThreadStart+0x1d

STACK_COMMAND: kb

FOLLOWUP_IP:
FeKern+ae3a
fffff880`08d98e3a 448ac8 mov r9b,al

SYMBOL_STACK_INDEX: a

SYMBOL_NAME: FeKern+ae3a

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: FeKern

IMAGE_NAME: FeKern.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 58bed113

FAILURE_BUCKET_ID: X64_0xc4_0_VRFK_FeKern+ae3a

BUCKET_ID: X64_0xc4_0_VRFK_FeKern+ae3a

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:x64_0xc4_0_vrfk_fekern+ae3a

FAILURE_ID_HASH: {d745cf9d-c314-24ea-0a62-99d021dbb5b9}

Followup: MachineOwner

The offending driver is a FireEye Realtime Driver ? part of security solution deployed to the target machine.
1: kd> lmvm FeKern
start end module name
fffff88008d8e000 fffff88008dcb000 FeKern (no symbols)
Loaded symbol image file: FeKern.sys
Image path: \SystemRoot\System32\drivers\FeKern.sys
Image name: FeKern.sys
Timestamp: Tue Mar 07 09:26:11 2017 (58BED113)
CheckSum: 00045980
ImageSize: 0003D000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

Questions for you:
1 - Am I on the right path with the verifier settings targeting ntoskrnl?
2 - The bugcheck stack looks odd: ExFreePool calling VeAllocatePoolWithTagPriority ? is that normal / expected?
3 - Does it look like a bug in FeKern.sys because it invokes RtlIsNameInExpression with NULL for the Expression parameter?

Thank you,
JT

2

Verifier will only detect pool leaks on driver unload. So, you need to
enable Verifier on the suspect driver(s) and then try to unload the drivers
(a reboot does not work). The drivers may not be unloadable, in which case
there’s not much you can do.

Also, unfortunately this tag appears to be an internal Registry tag that is
used ALL the time. You can see this for yourself by setting the PoolHitTag
breakpoint:

ed nt!PoolHitTag ‘bNMC’

Looks like it’s primarily around opening/creating Registry keys. That
*could* mean this is a handle leak, in which case Verifier won’t do you much
good either.

Running Verifier on the kernel image is a good idea for tracking down API
usage problems. Though in this case I think you’ve just stumbled onto
something unrelated with respect to the RtlIsNameInExpression error. And the
call stack is a bit weird but probably just an optimization artifact as the
rest of the call stack looks fine.

If this were my problem, I’d probably just go low tech. Try to come up with
a way to reproduce the leak and then start disabling things and retrying
until the problem goes away.

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntdev…

Greetings,

I?m looking for a gentle nudge in the right direction. I don?t typically
debug kernel components, so my questions may be elementary in nature.
Apologies in advance.

I?m trying to identify source of a paged pool leak. I?ve tracked down the
tag in question (CMNb) to nt aka ntkrnlmp. Working under assumption that
some 3rd party component is invoking NT API incorrectly (not cleaning up),
I?ve enabled verifier on ntoskrnl and rebooted.

Sample configuration:
1: kd> !verifier 3

Verify Flags Level 0x00000008

STANDARD FLAGS:
(0x00000000) Automatic Checks
(0x00000001) Special pool
(0x00000002) Force IRQL checking
(0x00000008) Pool tracking
(0x00000010) I/O verification
(0x00000020) Deadlock detection
(0x00000080) DMA checking
(0x00000100) Security checks
(0x00000800) Miscellaneous checks

ADDITIONAL FLAGS:
(0x00000004) Randomized low resources simulation
(0x00000200) Force pending I/O requests
(0x00000400) IRP logging

Indicates flag is enabled

Summary of All Verifier Statistics

RaiseIrqls 0x0
AcquireSpinLocks 0x0
Synch Executions 0x0
Trims 0x0

Pool Allocations Attempted 0x954607
Pool Allocations Succeeded 0x954607
Pool Allocations Succeeded SpecialPool 0xbdcb
Pool Allocations With NO TAG 0x1f5b2
Pool Allocations Failed 0x0

Current paged pool allocations 0x149e9 for 0E6B2800 bytes
Peak paged pool allocations 0x14a40 for 165DA8B0 bytes
Current nonpaged pool allocations 0xdaa4 for 023356C0 bytes
Peak nonpaged pool allocations 0xdb5d for 023D6D70 bytes

Driver Verification List

MODULE: 0xfffffa800c942080 ntoskrnl.exe (Loaded)

Pool Allocation Statistics: ( NonPagedPool / PagedPool )

Current Pool Allocations: ( 0x0000daa4 / 0x000149e9 )
Current Pool Bytes: ( 0x023356c0 / 0x0e6b2800 )
Peak Pool Allocations: ( 0x0000db5d / 0x00014a40 )
Peak Pool Bytes: ( 0x023d6d70 / 0x165da8b0 )
Contiguous Memory Bytes: 0x00000000
Peak Contiguous Memory Bytes: 0x00000000

Upon reboot, verifier detects a problem as soon as login screen comes up,
which prevents me from debugging the memory leak (can?t run for a long time
and leak memory).
1: kd> !analyze -v
*******************************************************************************
*
*
* Bugcheck Analysis
*
*
*
*******************************************************************************

DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught. This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this
driver.
If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA
will
be among the most commonly seen crashes.
Arguments:
Arg1: 0000000000000000, caller is trying to allocate zero bytes
Arg2: 0000000000000000, current IRQL
Arg3: 0000000000000000, pool type
Arg4: 0000000000000000, number of bytes

Debugging Details:

BUGCHECK_STR: 0xc4_0

CURRENT_IRQL: 2

DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT

PROCESS_NAME: svchost.exe

ANALYSIS_VERSION: 6.3.9600.16384 (debuggers(dbg).130821-1623) amd64fre

LAST_CONTROL_TRANSFER: from fffff80004363762 to fffff8000426ac70

STACK_TEXT:
fffff88009dcf858 fffff80004363762 : 0000000000000000 fffffa80148afb50
0000000000000065 fffff800042af644 : nt!DbgBreakPointWithStatus
fffff88009dcf860 fffff8000436454e : 0000000000000003 0000000000000000
fffff800042afea0 00000000000000c4 : nt!KiBugCheckDebugBreak+0x12
fffff88009dcf8c0 fffff80004272f44 : 0000000000000130 fffff88009dd1000
fffff88009dd02c8 fffff880017bca01 : nt!KeBugCheck2+0x71e
fffff88009dcff90 fffff800047054ec : 00000000000000c4 0000000000000000
0000000000000000 0000000000000000 : nt!KeBugCheckEx+0x104
fffff88009dcffd0 fffff80004705f2b : 0000000000000003 fffffa800cc48653
0000000000000000 0000000000000130 : nt!VerifierBugCheckIfAppropriate+0x3c
fffff88009dd0010 fffff80004716a58 : 0000000067727453 0000000000000080
0000000000000100 fffff80004715830 : nt!ExAllocatePoolSanityChecks+0xcb
fffff88009dd0050 fffff800043a8037 : fffff88009dd0290 0000000000000000
fffffa8067727453 fffff88009dd0210 : nt!VeAllocatePoolWithTagPriority+0x88
fffff88009dd00c0 fffff80004234061 : fffffa80158598a0 0000000000000000
0000000000000016 0000000000000402 : nt!ExFreePool+0x3b
fffff88009dd01b0 fffff80004262369 : fffff88009dd0210 0000000000000000
fffffa8000000401 fffff88009dd0250 : nt!RtlpUpcaseUnicodeStringPrivate+0x39
fffff88009dd01f0 fffff88008d98e3a : 0000000000000000 fffffa8015b52da0
0000000000000001 0000000000000000 : nt!RtlIsNameInExpression+0x59
fffff88009dd0230 fffff88008d98ec7 : 0000000000000000 fffffa8015d014e8
fffffa8015d015e8 fffffa8015d01401 : FeKern+0xae3a
fffff88009dd0270 fffff88008d93acf : fffffa8013bbba00 fffffa8013bbb930
fffffa8015d014e8 fffffa8015d01538 : FeKern+0xaec7
fffff88009dd02b0 fffff88008d93ea3 : 0000000101060001 0000000000000000
fffff88009dd03e8 fffff880017ae7e7 : FeKern+0x5acf
fffff88009dd0310 fffff880017ad288 : 0000000000001635 0000000000000000
fffffa801116fa38 fffffa801116fa38 : FeKern+0x5ea3
fffff88009dd03a0 fffff880017abd1b : fffffa80148f8010 fffffa8013bbb9d0
fffffa80158598a0 fffffa8015859ac0 : fltmgr!FltpPerformPostCallbacks+0x368
fffff88009dd0470 fffff880017cb2b9 : fffffa801116f800 fffffa8015a36010
fffffa801116f800 fffffa8015ae3040 :
fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0x39b
fffff88009dd0500 fffff80004721880 : 0000000000000000 0000000000000000
fffffa8000000040 0000000000000000 : fltmgr!FltpCreate+0x2a9
fffff88009dd05b0 fffff800045772bb : 0000000000000005 0000000000000040
fffffa8013b05550 fffffa8013b055e8 : nt!IovCallDriver+0xa0
fffff88009dd0610 fffff80004572dde : fffffa800e42cc60 0000000000000000
fffffa8014c2f670 0000000000000001 : nt!IopParseDevice+0x14e2
fffff88009dd0770 fffff800045738c6 : 0000000000000000 fffff88009dd08f0
fffff8a000000040 fffffa800d294420 : nt!ObpLookupObjectName+0x784
fffff88009dd0870 fffff800045756bc : 0000000003ebf9f0 0000000000000000
fffff8a06e566d01 fffff8a0135c53a0 : nt!ObOpenObjectByName+0x306
fffff88009dd0940 fffff8000455e7a8 : 0000000003ebf9b8 00000000000f01ff
0000000003ebf970 0000000003ebf960 : nt!IopCreateFile+0x2bc
fffff88009dd09e0 fffff800042720d3 : 0000000000000fa8 0000000000000000
0000000000000000 0000098000000003 : nt!NtOpenFile+0x58
fffff88009dd0a70 0000000077cdc06a : 000007feecdca7b1 0000000000000000
0000000077b7a54a 0000000000000000 : nt!KiSystemServiceCopyEnd+0x13
0000000003ebf8f8 000007feecdca7b1 : 0000000000000000 0000000077b7a54a
0000000000000000 0000000000000000 : ntdll!ZwOpenFile+0xa
0000000003ebf900 000007feecdca603 : 0000000000000fa8 000007feecdead40
0000000000000000 0000000000000000 : srvsvc!SsNotifyRdrOfGuid+0x7d
0000000003ebf9b0 000007feecdc4cfc : 000007feecdea890 0000000000000f70
0000000000000f54 0000000000000f54 : srvsvc!SsCheckRegistry+0x2da
0000000003ebfa20 000007feecdc3b31 : 0000000000000001 0000000000000001
0000000000000001 000007feecdf2180 : srvsvc!SsInitialize+0x37c
0000000003ebfcb0 00000000ff651344 : 000007feecdc8040 0000000000000000
000007feecdc8040 0000000000000000 : srvsvc!ServiceMain+0x2fc
0000000003ebfdc0 000007fefe39a82d : 0000000000000001 0000000002cd32f8
0000000000000000 000007feecdc8040 : svchost!ServiceStarter+0x1e8
0000000003ebfe50 0000000077b859cd : 0000000000000000 0000000000000000
0000000000000000 0000000000000000 : sechost!ScSvcctrlThreadA+0x25
0000000003ebfe80 0000000077cba561 : 0000000000000000 0000000000000000
0000000000000000 0000000000000000 : kernel32!BaseThreadInitThunk+0xd
0000000003ebfeb0 0000000000000000 : 0000000000000000 0000000000000000
0000000000000000 0000000000000000 : ntdll!RtlUserThreadStart+0x1d

STACK_COMMAND: kb

FOLLOWUP_IP:
FeKern+ae3a
fffff880`08d98e3a 448ac8 mov r9b,al

SYMBOL_STACK_INDEX: a

SYMBOL_NAME: FeKern+ae3a

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: FeKern

IMAGE_NAME: FeKern.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 58bed113

FAILURE_BUCKET_ID: X64_0xc4_0_VRFK_FeKern+ae3a

BUCKET_ID: X64_0xc4_0_VRFK_FeKern+ae3a

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:x64_0xc4_0_vrfk_fekern+ae3a

FAILURE_ID_HASH: {d745cf9d-c314-24ea-0a62-99d021dbb5b9}

Followup: MachineOwner

The offending driver is a FireEye Realtime Driver ? part of security
solution deployed to the target machine.
1: kd> lmvm FeKern
start end module name
fffff88008d8e000 fffff88008dcb000 FeKern (no symbols)
Loaded symbol image file: FeKern.sys
Image path: \SystemRoot\System32\drivers\FeKern.sys
Image name: FeKern.sys
Timestamp: Tue Mar 07 09:26:11 2017 (58BED113)
CheckSum: 00045980
ImageSize: 0003D000
Translations: 0000.04b0 0000.04e4 0409.04b0 0409.04e4

Questions for you:
1 - Am I on the right path with the verifier settings targeting ntoskrnl?
2 - The bugcheck stack looks odd: ExFreePool calling
VeAllocatePoolWithTagPriority ? is that normal / expected?
3 - Does it look like a bug in FeKern.sys because it invokes
RtlIsNameInExpression with NULL for the Expression parameter?

Thank you,
JT

3

Hello Scott,

Thank you very much for your feedback. I will use the breakpoint you mentioned and see if I can determine a pattern. I was planning on using “!verifier 80” to get call stacks for recent allocations for the tag in question.

Interesting thing about the leak is that I don’t see a handle leak, but there are millions of outstanding allocations after 6 to 8 hours of running. The leaked objects are small (0x20 bytes) and they resemble something like a GUID.
0: kd> !poolused 4
…
Sorting by Paged Pool Consumed

NonPaged Paged
Tag Allocs Used Allocs Used

CMNb 0 0 15,037,835 722,889,008 Configuration Manager Name Tag , Binary: nt!cm

Unfortunately, custom access control list is blocking me from disabling 3rd party security applications & drivers on target machine.

I’ve reported the suspected bug in FeKern.sys, and hopefully it will make its way up to the vendor.

Thank you,
JT