After a recent update to an isolation filter, a customer on a Win10 x64 is reporting numerous BSOD’s with different codes each time (KMODE_EXCEPTION_NOT_HANDLED, PAGE_FAULT_IN_NONPAGED_AREA, CUSTOM_ERROR, etc). I managed to get a dump of one of those crashes (PAGE_FAULT) and it points to KiDeliverApc. In fact, the conditions pretty much mirror the error described here: http://www.osronline.com/showThread.cfm?link=118677 except for OS and architecture. But the setup is the same.

KiDeliverApc
mov rbx,qword ptr gs:[188h] <-CurrentThread
mov qword ptr [rsp+90h],rdi
cmp word ptr [rbx+1E6h],0
mov r14,qword ptr [rbx+90h]
mov r15,qword ptr [rbx+0B8h]
mov qword ptr [rbx+90h],rsi
mov byte ptr [rbx+0C1h],0
jne nt!KiDeliverApc+0x153
lock or dword ptr [rsp],0
lea rdi,[rbx+98h] <-CurrentThread.ApcState
xor r9d,r9d
mov ecx,2
cmp qword ptr [rdi],rdi <-Bugcheck

The actual stack trace is

00 ffff9c80941bb358 fffff803391b6fb4 nt!KeBugCheckEx
01 ffff9c80941bb360 fffff803390a22d6 nt!MiSystemFault+0x116e84
02 ffff9c80941bb400 fffff8033918ad72 nt!MmAccessFault+0xae6
03 ffff9c80941bb5f0 fffff8033904a335 nt!KiPageFault+0x132
04 ffff9c80941bb780 fffff8033904bd75 nt!KiDeliverApc+0x75
05 ffff9c80941bb810 fffff803394d3bf9 nt!KiCheckForKernelApcDelivery+0x25
06 ffff9c80941bb840 fffff803394ba6d5 nt!ObpCallPreOperationCallbacks+0x1c9
07 ffff9c80941bb8c0 fffff803394d1b5e nt!ObpCreateHandle+0x9d5
08 ffff9c80941bbb00 fffff803394501e5 nt!ObOpenObjectByPointer+0x11e
09 ffff9c80941bbd60 fffff8033944fe55 nt!PspCreateObjectHandle+0x3d
0a ffff9c80941bbdb0 fffff803394379a3 nt!PspInsertThread+0x429
0b ffff9c80941bbe70 fffff8033918c413 nt!NtCreateUserProcess+0x953
0c ffff9c80941bca90 00007ffbb1e06b94 nt!KiSystemServiceCopyEnd+0x13
0d 000000756987e078 00007ffbaede4a61 ntdll!NtCreateUserProcess+0x14
0e 000000756987e080 00007ffbaede7c46 KERNELBASE!CreateProcessInternalW+0xd21
0f 000000756987ec50 00007ffbaf2bba83 KERNELBASE!CreateProcessW+0x66
10 000000756987ecc0 00007ff740b39e6f KERNEL32!CreateProcessWStub+0x53

But again, because the customer is getting different error codes each time none of this probably means much. I can’t find any reference to my driver in the current stack (dps @rsp) and my driver isn’t listed as the offending driver that caused the crash.

One of the stranger things is that the !chkimg command is reporting a couple thousand changes but the majority of them are 2 bytes with similar patterns. I’m guessing this could of just been a Windows update/hotfix?

CHKIMG_EXTENSION: !chkimg -lo 50 -d !nt
fffff803390166bc-fffff803390166c0 5 bytes - nt!MiLockDownWorkingSet+54
[d0 be 7d fb f6:40 9b 36 6d da]
fffff803390166da-fffff803390166db 2 bytes - nt!MiLockDownWorkingSet+72 (+0x1e)
[80 f6:00 da]
fffff80339016701-fffff80339016702 2 bytes - nt!MiLockDownWorkingSet+99 (+0x27)
[80 fa:00 f1]
fffff803390167f4-fffff803390167f5 2 bytes - nt!MiCreateForkWsles+28 (+0xf3)
[80 f6:00 da]
fffff80339016908-fffff80339016909 2 bytes - nt!MiCreateForkWsle+60 (+0x114)
[80 f6:00 da]
fffff8033901694e-fffff8033901694f 2 bytes - nt!MiCreateForkWsle+a6 (+0x46)
[80 fa:00 f1]
fffff80339016b83-fffff80339016b84 2 bytes - nt!MiCloneVads+1ab (+0x235)
[80 f6:00 da]
fffff80339016d6c-fffff80339016d6d 2 bytes - nt!MiCloneVads+394 (+0x1e9)
[80 f6:00 da]
fffff8033901703e-fffff8033901703f 2 bytes - nt!MiBuildForkPte+b6 (+0x2d2)
[80 f6:00 da]
fffff80339017052-fffff80339017053 2 bytes - nt!MiBuildForkPte+ca (+0x14)
[80 fa:00 f1]
fffff8033901716a-fffff8033901716b 2 bytes - nt!MiBuildForkPte+1e2 (+0x118)
[80 fa:00 f1]
fffff803390175fb-fffff803390175fc 2 bytes - nt!MiUpdateForkMaps+83 (+0x491)
[80 f6:00 da]
fffff803390176ec-fffff803390176ed 2 bytes - nt!MiUpdateForkMaps+174 (+0xf1)
[80 fa:00 f1]
fffff80339017735-fffff80339017736 2 bytes - nt!MiUpdateForkMaps+1bd (+0x49)
[80 fa:00 f1]
fffff80339017843-fffff80339017844 2 bytes - nt!MiBuildForkPageTable+33 (+0x10e)
[80 fa:00 f1]
fffff80339017a04-fffff80339017a05 2 bytes - nt!MiDoneWithThisPageGetAnother+9c (+0x1c1)
[80 fa:00 f1]

The update I pushed didn’t have substantial changes. All that was added was another linked list to keep track of some new state information and I reused a lot of code that has been working without any issues for a long time. In the link above, Slava mentioned it could be a synchronization issue but I’m not sure why this would just appear now especially when the changes had nothing to do with pending operations.

Nothing obvious jumping out at me based on the information provided. If you
put the crash dump someplace I can take a look and see if I see anything
interesting.

Absent that, a few things:

1. Post the full !analyze -v output. It helps in understanding the
   corruption better

2. Did you decode the APC to see what the callback is? I’m interested in
   which operation we’re talking about (i.e. I/O completion, thread
   termination, etc.)

3. The !chkimg thing is a red herring due to recent O/S changes, you can
   ignore it.

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntfsd…

After a recent update to an isolation filter, a customer on a Win10 x64 is
reporting numerous BSOD’s with different codes each time
(KMODE_EXCEPTION_NOT_HANDLED, PAGE_FAULT_IN_NONPAGED_AREA, CUSTOM_ERROR,
etc). I managed to get a dump of one of those crashes (PAGE_FAULT) and it
points to KiDeliverApc. In fact, the conditions pretty much mirror the
error described here: http://www.osronline.com/showThread.cfm?link=118677
except for OS and architecture. But the setup is the same.

KiDeliverApc
mov rbx,qword ptr gs:[188h] <-CurrentThread
mov qword ptr [rsp+90h],rdi
cmp word ptr [rbx+1E6h],0
mov r14,qword ptr [rbx+90h]
mov r15,qword ptr [rbx+0B8h]
mov qword ptr [rbx+90h],rsi
mov byte ptr [rbx+0C1h],0
jne nt!KiDeliverApc+0x153
lock or dword ptr [rsp],0
lea rdi,[rbx+98h] <-CurrentThread.ApcState
xor r9d,r9d
mov ecx,2
cmp qword ptr [rdi],rdi <-Bugcheck

The actual stack trace is

00 ffff9c80941bb358 fffff803391b6fb4 nt!KeBugCheckEx
01 ffff9c80941bb360 fffff803390a22d6 nt!MiSystemFault+0x116e84
02 ffff9c80941bb400 fffff8033918ad72 nt!MmAccessFault+0xae6
03 ffff9c80941bb5f0 fffff8033904a335 nt!KiPageFault+0x132
04 ffff9c80941bb780 fffff8033904bd75 nt!KiDeliverApc+0x75
05 ffff9c80941bb810 fffff803394d3bf9 nt!KiCheckForKernelApcDelivery+0x25
06 ffff9c80941bb840 fffff803394ba6d5 nt!ObpCallPreOperationCallbacks+0x1c9
07 ffff9c80941bb8c0 fffff803394d1b5e nt!ObpCreateHandle+0x9d5
08 ffff9c80941bbb00 fffff803394501e5 nt!ObOpenObjectByPointer+0x11e
09 ffff9c80941bbd60 fffff8033944fe55 nt!PspCreateObjectHandle+0x3d
0a ffff9c80941bbdb0 fffff803394379a3 nt!PspInsertThread+0x429
0b ffff9c80941bbe70 fffff8033918c413 nt!NtCreateUserProcess+0x953
0c ffff9c80941bca90 00007ffbb1e06b94 nt!KiSystemServiceCopyEnd+0x13
0d 000000756987e078 00007ffbaede4a61 ntdll!NtCreateUserProcess+0x14
0e 000000756987e080 00007ffbaede7c46
KERNELBASE!CreateProcessInternalW+0xd21
0f 000000756987ec50 00007ffbaf2bba83 KERNELBASE!CreateProcessW+0x66
10 000000756987ecc0 00007ff740b39e6f KERNEL32!CreateProcessWStub+0x53

But again, because the customer is getting different error codes each time
none of this probably means much. I can’t find any reference to my driver
in the current stack (dps @rsp) and my driver isn’t listed as the offending
driver that caused the crash.

One of the stranger things is that the !chkimg command is reporting a couple
thousand changes but the majority of them are 2 bytes with similar patterns.
I’m guessing this could of just been a Windows update/hotfix?

CHKIMG_EXTENSION: !chkimg -lo 50 -d !nt
fffff803390166bc-fffff803390166c0 5 bytes - nt!MiLockDownWorkingSet+54
[d0 be 7d fb f6:40 9b 36 6d da]
fffff803390166da-fffff803390166db 2 bytes - nt!MiLockDownWorkingSet+72
(+0x1e)
[80 f6:00 da]
fffff80339016701-fffff80339016702 2 bytes - nt!MiLockDownWorkingSet+99
(+0x27)
[80 fa:00 f1]
fffff803390167f4-fffff803390167f5 2 bytes - nt!MiCreateForkWsles+28
(+0xf3)
[80 f6:00 da]
fffff80339016908-fffff80339016909 2 bytes - nt!MiCreateForkWsle+60
(+0x114)
[80 f6:00 da]
fffff8033901694e-fffff8033901694f 2 bytes - nt!MiCreateForkWsle+a6
(+0x46)
[80 fa:00 f1]
fffff80339016b83-fffff80339016b84 2 bytes - nt!MiCloneVads+1ab (+0x235)
[80 f6:00 da]
fffff80339016d6c-fffff80339016d6d 2 bytes - nt!MiCloneVads+394 (+0x1e9)
[80 f6:00 da]
fffff8033901703e-fffff8033901703f 2 bytes - nt!MiBuildForkPte+b6
(+0x2d2)
[80 f6:00 da]
fffff80339017052-fffff80339017053 2 bytes - nt!MiBuildForkPte+ca
(+0x14)
[80 fa:00 f1]
fffff8033901716a-fffff8033901716b 2 bytes - nt!MiBuildForkPte+1e2
(+0x118)
[80 fa:00 f1]
fffff803390175fb-fffff803390175fc 2 bytes - nt!MiUpdateForkMaps+83
(+0x491)
[80 f6:00 da]
fffff803390176ec-fffff803390176ed 2 bytes - nt!MiUpdateForkMaps+174
(+0xf1)
[80 fa:00 f1]
fffff80339017735-fffff80339017736 2 bytes - nt!MiUpdateForkMaps+1bd
(+0x49)
[80 fa:00 f1]
fffff80339017843-fffff80339017844 2 bytes - nt!MiBuildForkPageTable+33
(+0x10e)
[80 fa:00 f1]
fffff80339017a04-fffff80339017a05 2 bytes -
nt!MiDoneWithThisPageGetAnother+9c (+0x1c1)
[80 fa:00 f1]

The update I pushed didn’t have substantial changes. All that was added was
another linked list to keep track of some new state information and I reused
a lot of code that has been working without any issues for a long time. In
the link above, Slava mentioned it could be a synchronization issue but I’m
not sure why this would just appear now especially when the changes had
nothing to do with pending operations.

Thanks Scott. Let me check to make sure it’s OK to share the dump. It’d be great to get your help with it.

Couple more strange things I noticed with this. When I decode the APC, I’m either doing it wrong (probably) or there is something really screwy, because the values look bogus (type and size are all wrong). And while this crash occurred in services.exe context, when I run the !apc command, services.exe is not listed as having any APCs.

In the meantime, here’s some more info.

4: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced. This cannot be protected by try-except.
Typically the address is just plain bad or it is pointing at freed memory.
Arguments:
Arg1: ffffc80500000000, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffff8033904a335, If non-zero, the instruction address which referenced the bad memory
address.
Arg4: 0000000000000002, (reserved)

Debugging Details:

DUMP_CLASS: 1

DUMP_QUALIFIER: 402

BUILD_VERSION_STRING: 15063.0.amd64fre.rs2_release.170317-1834

SYSTEM_MANUFACTURER: Dell Inc.

SYSTEM_PRODUCT_NAME: Dell System XPS L502X

SYSTEM_SKU: System SKUNumber

BIOS_VENDOR: Dell Inc.

BIOS_VERSION: A12

BIOS_DATE: 09/07/2012

BASEBOARD_MANUFACTURER: Dell Inc.

BASEBOARD_PRODUCT: 0NJT03

BASEBOARD_VERSION: A00

DUMP_TYPE: 0

BUGCHECK_P1: ffffc80500000000

BUGCHECK_P2: 0

BUGCHECK_P3: fffff8033904a335

BUGCHECK_P4: 2

READ_ADDRESS: ffffc80500000000

FAULTING_IP:
nt!KiDeliverApc+75
fffff803`3904a335 48393f cmp qword ptr [rdi],rdi

MM_INTERNAL_CODE: 2

CPU_COUNT: 8

CPU_MHZ: 7cb

CPU_VENDOR: GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 2a

CPU_STEPPING: 7

CPU_MICROCODE: 6,2a,7,0 (F,M,S,R) SIG: 29’00000000 (cache) 29’00000000 (init)

DEFAULT_BUCKET_ID: CODE_CORRUPTION

BUGCHECK_STR: AV

PROCESS_NAME: services.exe

CURRENT_IRQL: 1

ANALYSIS_SESSION_HOST: LAPTOPWIN10

ANALYSIS_SESSION_TIME: 07-19-2017 18:35:17.0796

ANALYSIS_VERSION: 10.0.10586.567 amd64fre

TRAP_FRAME: ffff9c80941bb5f0 – (.trap 0xffff9c80941bb5f0)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=ffffc805fc976ec0 rbx=0000000000000000 rcx=0000000000000002
rdx=ffffc805fc976ec0 rsi=0000000000000000 rdi=0000000000000000
rip=fffff8033904a335 rsp=ffff9c80941bb780 rbp=ffff9c80941bb7d0
r8=ffffc805fc986a50 r9=0000000000000000 r10=ffffc805f6337960
r11=ffffc805fc474d30 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
nt!KiDeliverApc+0x75:
fffff8033904a335 48393f cmp qword ptr [rdi],rdi ds:0000000000000000=???
Resetting default scope

LAST_CONTROL_TRANSFER: from fffff803391b6fb4 to fffff803391814c0

STACK_TEXT:
ffff9c80941bb358 fffff803391b6fb4 : 0000000000000050 ffffc80500000000 0000000000000000 ffff9c80941bb5f0 : nt!KeBugCheckEx
ffff9c80941bb360 fffff803390a22d6 : 0000000000000000 ffffc80500000000 ffff9c80941bb5f0 ffffc805fbe42640 : nt!MiSystemFault+0x116e84
ffff9c80941bb400 fffff8033918ad72 : ffffc805fc976050 0000000000000000 0000000000000000 fffff80300001800 : nt!MmAccessFault+0xae6
ffff9c80941bb5f0 fffff8033904a335 : ffffc805fc474da8 ffffc805f622f390 ffffc805fc979000 0000000000000000 : nt!KiPageFault+0x132
ffff9c80941bb780 fffff8033904bd75 : 0000000000000000 0000000000000000 0000000000000000 ffff9c80941bba00 : nt!KiDeliverApc+0x75
ffff9c80941bb810 fffff803394d3bf9 : ffff9c80941bba50 0000000000000000 ffff9c8000000000 ffff9c80941bba50 : nt!KiCheckForKernelApcDelivery+0x25
ffff9c80941bb840 fffff803394ba6d5 : ffffc805f622f390 ffff9c80941bb9e0 ffff9c80941bc3c0 ffff9c80941bb9e0 : nt!ObpCallPreOperationCallbacks+0x1c9
ffff9c80941bb8c0 fffff803394d1b5e : ffff9c80941bc3c0 0000000000000000 0000000000000000 0000000000000b54 : nt!ObpCreateHandle+0x9d5
ffff9c80941bbb00 fffff803394501e5 : ffffc805fc976080 fffff80339453488 fffff803393970e8 fffff803393972c8 : nt!ObOpenObjectByPointer+0x11e
ffff9c80941bbd60 fffff8033944fe55 : ffff9c80941bc550 0000000000000000 0000000000000000 0000000000000001 : nt!PspCreateObjectHandle+0x3d
ffff9c80941bbdb0 fffff803394379a3 : ffffc805fc979080 ffffc80500000000 ffff9c80941bc010 ffff9c80941bbef8 : nt!PspInsertThread+0x429
ffff9c80941bbe70 fffff8033918c413 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!NtCreateUserProcess+0x953
ffff9c80941bca90 00007ffbb1e06b94 : 00007ffbaede4a61 0000000000000003 000000756987e2b0 0000000000000001 : nt!KiSystemServiceCopyEnd+0x13
000000756987e078 00007ffbaede4a61 : 0000000000000003 000000756987e2b0 0000000000000001 0000000000000000 : ntdll!NtCreateUserProcess+0x14
000000756987e080 00007ffbaede7c46 : 0000025586ee8210 00007ffb00000042 0000025586870000 0000000000000070 : KERNELBASE!CreateProcessInternalW+0xd21
000000756987ec50 00007ffbaf2bba83 : 0000000000000037 0000000000000000 0000000000000550 0000000000000000 : KERNELBASE!CreateProcessW+0x66
000000756987ecc0 00007ff740b39e6f : 0000000000000070 000002558696fdd0 00007ff740b93630 0000025586ee74d0 : KERNEL32!CreateProcessWStub+0x53
000000756987ed20 00007ff740b3acff : 000002558696fdd0 000000756987f240 0000000000000000 0000025586ee74d0 : services!CWin32ServiceRecord::LogonAndStartImage+0x97b
000000756987f140 00007ff740b3dd5a : 0000000000000000 0000025586ed1ac0 0000000000000018 000002558696fea0 : services!CWin32ServiceRecord::StartInternal+0x1ef
000000756987f3f0 00007ff740b3ec40 : 0000000000000000 0000025586ed1ac0 000000756987f4a0 00007ff740ba4000 : services!CServiceRecord::Start+0xa2
000000756987f450 00007ff740b3e892 : 0000025586916ba0 0000025500000000 0000000000000000 0000025500000000 : services!ScStartMarkedServicesInServiceSet+0x19c
000000756987f4e0 00007ff740b3d524 : 000002558696fdd0 000000756987f5f8 00007ff700000010 0000000000000000 : services!ScStartServicesInStartList+0x272
000000756987f5c0 00007ff740b56007 : 0000000000000000 0000000000000000 0000000000000000 0000025586e391f8 : services!ScStartServiceAndDependencies+0x1cc
000000756987f690 00007ffbb1da3021 : 0000025586e39130 0000000000000000 000000756987f8a8 000000007ffe0386 : services!ScStartServiceCallback+0x37
000000756987f6d0 00007ffbb1da1989 : 0000025586934500 0000000000000000 00007ffbb1da2ef0 0000000000000000 : ntdll!TppWorkpExecuteCallback+0x131
000000756987f720 00007ffbaf2b2774 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : ntdll!TppWorkerThread+0x6c9
000000756987fa30 00007ffbb1dd0d51 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : KERNEL32!BaseThreadInitThunk+0x14
000000756987fa60 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : ntdll!RtlUserThreadStart+0x21

STACK_COMMAND: kb

MODULE_NAME: memory_corruption

IMAGE_NAME: memory_corruption

FOLLOWUP_NAME: memory_corruption

DEBUG_FLR_IMAGE_TIMESTAMP: 0

MEMORY_CORRUPTOR: LARGE

FAILURE_BUCKET_ID: MEMORY_CORRUPTION_LARGE

BUCKET_ID: MEMORY_CORRUPTION_LARGE

PRIMARY_PROBLEM_CLASS: MEMORY_CORRUPTION_LARGE

TARGET_TIME: 2017-07-17T11:54:38.000Z

OSBUILD: 15063

OSSERVICEPACK: 0

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK: 272

PRODUCT_TYPE: 1

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

OSEDITION: Windows 10 WinNt TerminalServer SingleUserTS

OS_LOCALE:

USER_LCID: 0

OSBUILD_TIMESTAMP: 2017-07-07 02:06:35

BUILDDATESTAMP_STR: 170317-1834

BUILDLAB_STR: rs2_release

BUILDOSVER_STR: 10.0.15063.0.amd64fre.rs2_release.170317-1834

ANALYSIS_SESSION_ELAPSED_TIME: 1b2f

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:memory_corruption_large

FAILURE_ID_HASH: {e29154ac-69a4-0eb8-172a-a860f73c0a3c}

Followup: memory_corruption

4: kd> !process -1 0
PROCESS ffffc805fbe42640
SessionId: 0 Cid: 03ec Peb: 75694c8000 ParentCid: 0328
DirBase: 12ad83000 ObjectTable: ffffb0893f71ad80 HandleCount: 336.
Image: services.exe

4: kd> !thread -1 0
THREAD ffffc805fc0fa4c0 Cid 03ec.02cc Teb: 00000075694d3000 Win32Thread: 0000000000000000 RUNNING on processor 4

4: kd> dt ffffc805fc0fa4c0 _KTHREAD
ntdll!_KTHREAD
+0x000 Header : _DISPATCHER_HEADER
+0x018 SListFaultAddress : (null)
+0x020 QuantumTarget : 0x871b273
+0x028 InitialStack : 0xffff9c80941bcc90 Void +0x030 StackLimit : 0xffff9c80941b7000 Void
+0x038 StackBase : 0xffff9c80941bd000 Void +0x040 ThreadLock : 0 +0x048 CycleTime : 0x594ade7 +0x050 CurrentRunTime : 0x3fd6ad +0x054 ExpectedRunTime : 0x993c +0x058 KernelStack : 0xffff9c80941bab20 Void
+0x060 StateSaveArea : 0xffff9c80941bccc0 _XSAVE_FORMAT +0x068 SchedulingGroup : (null) +0x070 WaitRegister : _KWAIT_STATUS_REGISTER +0x071 Running : 0x1 '' +0x072 Alerted : [2] "" +0x074 AutoBoostActive : 0y1 +0x074 ReadyTransition : 0y0 +0x074 WaitNext : 0y0 +0x074 SystemAffinityActive : 0y0 +0x074 Alertable : 0y0 +0x074 UserStackWalkActive : 0y0 +0x074 ApcInterruptRequest : 0y0 +0x074 QuantumEndMigrate : 0y0 +0x074 UmsDirectedSwitchEnable : 0y0 +0x074 TimerActive : 0y0 +0x074 SystemThread : 0y0 +0x074 ProcessDetachActive : 0y0 +0x074 CalloutActive : 0y0 +0x074 ScbReadyQueue : 0y0 +0x074 ApcQueueable : 0y1 +0x074 ReservedStackInUse : 0y0 +0x074 UmsPerformingSyscall : 0y0 +0x074 TimerSuspended : 0y0 +0x074 SuspendedWaitMode : 0y0 +0x074 SuspendSchedulerApcWait : 0y0 +0x074 Reserved : 0y000000000000 (0) +0x074 MiscFlags : 0n16385 +0x078 AutoAlignment : 0y0 +0x078 DisableBoost : 0y0 +0x078 BamEppImportant : 0y0 +0x078 AlertedByThreadId : 0y0 +0x078 QuantumDonation : 0y1 +0x078 EnableStackSwap : 0y1 +0x078 GuiThread : 0y0 +0x078 DisableQuantum : 0y0 +0x078 ChargeOnlySchedulingGroup : 0y0 +0x078 DeferPreemption : 0y0 +0x078 QueueDeferPreemption : 0y0 +0x078 ForceDeferSchedule : 0y0 +0x078 SharedReadyQueueAffinity : 0y1 +0x078 FreezeCount : 0y0 +0x078 TerminationApcRequest : 0y0 +0x078 AutoBoostEntriesExhausted : 0y0 +0x078 KernelStackResident : 0y1 +0x078 TerminateRequestReason : 0y00 +0x078 ProcessStackCountDecremented : 0y0 +0x078 RestrictedGuiThread : 0y0 +0x078 ThreadFlagsSpare : 0y000 +0x078 EtwStackTraceApcInserted : 0y00000000 (0) +0x078 ThreadFlags : 0n69680 +0x07c Tag : 0 '' +0x07d SystemHeteroCpuPolicy : 0 '' +0x07e UserHeteroCpuPolicy : 0y0001000 (0x8) +0x07e ExplicitSystemHeteroCpuPolicy : 0y0 +0x07f Spare0 : 0 '' +0x080 SystemCallNumber : 0xc0 +0x084 ReadyTime : 0 +0x088 FirstArgument : 0x000000756987e160 Void
+0x090 TrapFrame : (null)
+0x098 ApcState : _KAPC_STATE
+0x098 ApcStateFill : [43] “X???”
+0x0c3 Priority : 9 ‘’
+0x0c4 UserIdealProcessor : 2
+0x0c8 WaitStatus : 0n0
+0x0d0 WaitBlockList : 0xffffc805fc0fa600 _KWAIT_BLOCK +0x0d8 WaitListEntry : _LIST_ENTRY [0x0000000000000000 - 0xfffff80337ca6860] +0x0d8 SwapListEntry : _SINGLE_LIST_ENTRY +0x0e8 Queue : 0xffffc805fa1ecf80 _DISPATCHER_HEADER
+0x0f0 Teb : 0x00000075694d3000 Void +0x0f8 RelativeTimerBias : 0 +0x100 Timer : _KTIMER +0x140 WaitBlock : [4] _KWAIT_BLOCK +0x140 WaitBlockFill4 : [20] " .f???" +0x154 ContextSwitches : 0x1b4 +0x140 WaitBlockFill5 : [68] " .f???" +0x184 State : 0x2 '' +0x185 Spare13 : 0 '' +0x186 WaitIrql : 0 '' +0x187 WaitMode : 0 '' +0x140 WaitBlockFill6 : [116] " .f???" +0x1b4 WaitTime : 0xcf2 +0x140 WaitBlockFill7 : [164] " .f???" +0x1e4 KernelApcDisable : 0n-2 +0x1e6 SpecialApcDisable : 0n0 +0x1e4 CombinedApcDisable : 0xfffe +0x140 WaitBlockFill8 : [40] " .f???" +0x168 ThreadCounters : (null) +0x140 WaitBlockFill9 : [88] " .f???" +0x198 XStateSave : (null) +0x140 WaitBlockFill10 : [136] " .f???" +0x1c8 Win32Thread : (null) +0x140 WaitBlockFill11 : [176] " .f???" +0x1f0 Ucb : (null) +0x1f8 Uch : (null) +0x200 Spare21 : (null) +0x208 QueueListEntry : _LIST_ENTRY [0xffffc805fc0fe288 - 0xffffc805fc0f5488] +0x218 NextProcessor : 4 +0x218 NextProcessorNumber : 0y0000000000000000000000000000100 (0x4) +0x218 SharedReadyQueue : 0y0 +0x21c QueuePriority : 0n0 +0x220 Process : 0xffffc805fbe42640 _KPROCESS
+0x228 UserAffinity : _GROUP_AFFINITY
+0x228 UserAffinityFill : [10] “???”
+0x232 PreviousMode : 1 ‘’
+0x233 BasePriority : 8 ‘’
+0x234 PriorityDecrement : 0 ‘’
+0x234 ForegroundBoost : 0y0000
+0x234 UnusualBoost : 0y0000
+0x235 Preempted : 0 ‘’
+0x236 AdjustReason : 0 ‘’
+0x237 AdjustIncrement : 1 ‘’
+0x238 AffinityVersion : 0x24
+0x240 Affinity : _GROUP_AFFINITY
+0x240 AffinityFill : [10] “???”
+0x24a ApcStateIndex : 0 ‘’
+0x24b WaitBlockCount : 0x1 ‘’
+0x24c IdealProcessor : 2
+0x250 NpxState : 5
+0x258 SavedApcState : _KAPC_STATE
+0x258 SavedApcStateFill : [43] “???”
+0x283 WaitReason : 0x6 ‘’
+0x284 SuspendCount : 0 ‘’
+0x285 Saturation : 0 ‘’
+0x286 SListFaultCount : 0
+0x288 SchedulerApc : _KAPC
+0x288 SchedulerApcFill0 : [1] “???”
+0x289 ResourceIndex : 0x1 ‘’
+0x288 SchedulerApcFill1 : [3] “???”
+0x28b QuantumReset : 0x6 ‘’
+0x288 SchedulerApcFill2 : [4] “???”
+0x28c KernelTime : 4
+0x288 SchedulerApcFill3 : [64] “???”
+0x2c8 WaitPrcb : (null)
+0x288 SchedulerApcFill4 : [72] “???”
+0x2d0 LegoData : (null)
+0x288 SchedulerApcFill5 : [83] “???”
+0x2db CallbackNestingLevel : 0 ‘’
+0x2dc UserTime : 0
+0x2e0 SuspendEvent : _KEVENT
+0x2f8 ThreadListEntry : _LIST_ENTRY [0xffffc805fc0fe378 - 0xffffc805fc0f5578]
+0x308 MutantListHead : _LIST_ENTRY [0xffffc805fc0fa7c8 - 0xffffc805fc0fa7c8]
+0x318 AbEntrySummary : 0x3f ‘?’
+0x319 AbWaitEntryCount : 0 ‘’
+0x31a AbAllocationRegionCount : 0 ‘’
+0x31b Spare20 : 0 ‘’
+0x31c SecureThreadCookie : 0
+0x320 LockEntries : [6] _KLOCK_ENTRY
+0x560 PropagateBoostsEntry : _SINGLE_LIST_ENTRY
+0x568 IoSelfBoostsEntry : _SINGLE_LIST_ENTRY
+0x570 PriorityFloorCounts : [16] “”
+0x580 PriorityFloorSummary : 0
+0x584 AbCompletedIoBoostCount : 0n0
+0x588 AbCompletedIoQoSBoostCount : 0n0
+0x58c KeReferenceCount : 0n0
+0x58e AbOrphanedEntrySummary : 0 ‘’
+0x58f AbOwnedEntryCount : 0 ‘’
+0x590 ForegroundLossTime : 0
+0x598 GlobalForegroundListEntry : _LIST_ENTRY [0x0000000000000001 - 0x0000000000000000]
+0x598 ForegroundDpcStackListEntry : _SINGLE_LIST_ENTRY
+0x5a0 InGlobalForegroundList : 0
+0x5a8 ReadOperationCount : 0n0
+0x5b0 WriteOperationCount : 0n1
+0x5b8 OtherOperationCount : 0n364
+0x5c0 ReadTransferCount : 0n0
+0x5c8 WriteTransferCount : 0n1080
+0x5d0 OtherTransferCount : 0n2102
+0x5d8 QueuedScb : (null)
+0x5e0 ThreadTimerDelay : 0
+0x5e4 Spare22 : 0n0

4: kd> dt ffffc805fc0fa4c0+0x98 _KAPC_STATE
ntdll!_KAPC_STATE
+0x000 ApcListHead : [2] _LIST_ENTRY [0xffffc805fc0fa558 - 0xffffc805fc0fa558]
+0x020 Process : 0xffffc805`fbe42640 _KPROCESS
+0x028 InProgressFlags : 0 ‘’
+0x028 KernelApcInProgress : 0y0
+0x028 SpecialApcInProgress : 0y0
+0x029 KernelApcPending : 0 ‘’
+0x02a UserApcPending : 0 ‘’

4: kd> dt 0xffffc805fc0fa558-0x10 _KAPC ntdll!_KAPC +0x000 Type : 0x60 '’
+0x001 SpareByte0 : 0xe1 ‘’
+0x002 Size : 0x87 ‘’
+0x003 SpareByte1 : 0x69 ‘i’
+0x004 SpareLong0 : 0x75
+0x008 Thread : (null)
+0x010 ApcListEntry : _LIST_ENTRY [0xffffc805fc0fa558 - 0xffffc805fc0fa558]
+0x020 KernelRoutine : 0xffffc805fc0fa568 void +ffffc805fc0fa568 +0x028 RundownRoutine : 0xffffc805fc0fa568 void +ffffc805fc0fa568
+0x030 NormalRoutine : 0xffffc805fbe42640 void +ffffc805fbe42640 +0x020 Reserved : [3] 0xffffc805fc0fa568 Void
+0x038 NormalContext : 0x0000000209000000 Void +0x040 SystemArgument1 : (null) +0x048 SystemArgument2 : 0xffffc805fc0fa600 Void
+0x050 ApcStateIndex : 0 ‘’
+0x051 ApcMode : 0 ‘’
+0x052 Inserted : 0 ‘’

4: kd> !apc
*** Enumerating APCs in all processes
Process ffffc805f62b4040 System
Thread ffffc805f6304040 Thread ffffc805f6335040
Thread ffffc805fa220700 Thread ffffc805faf6a700
Process ffffc805fb0e7080 csrss.exe
Thread ffffc805fa7fd080 Thread ffffc805fafcf080
Process ffffc805fbd68080 csrss.exe
Thread ffffc805fbd64500 Thread ffffc805fbd61080
Thread ffffc805fbdff080 Thread ffffc805fbe07080
Thread ffffc805fbe6c080 Thread ffffc805fbda5080
Thread ffffc805fbe43080 Thread ffffc805fbe3b080
Thread ffffc805fbe32080 Thread ffffc805fbe2f080
Thread ffffc805fbe2b080 Thread ffffc805fbe40080
Thread ffffc805fc10c080 Thread ffffc805fc113700
Thread ffffc805fc12d080 Thread ffffc805fc131080
Thread ffffc805fc12c080 Thread ffffc805fc132080
Thread ffffc805fc144080 Thread ffffc805fc146080
Thread ffffc805fc1bb080 Thread ffffc805fc1bd3c0
Thread ffffc805fb635080 Thread ffffc805fb622080
Thread ffffc805fb6ad080 Thread ffffc805fb6ab080
Thread ffffc805fb6a1080 Thread ffffc805fa425700
Thread ffffc805fb74a6c0 Thread ffffc805fb73c080
Thread ffffc805fb741080 Thread ffffc805fbd295c0
Thread ffffc805fb76d080 Thread ffffc805fb763080
Thread ffffc805fb79b080 Thread
Thread ffffc805fb79a680 Thread ffffc805fb7ff080
Thread ffffc805fc459080 Thread ffffc805fc4553c0
Thread ffffc805fc446080 Thread ffffc805fc44a580
Thread ffffc805fb75c080 Thread ffffc805fb7d0080
Thread ffffc805fb66f540 Thread ffffc805fb7cd080
Thread ffffc805fb7f24c0 Thread ffffc805fb7e7080
Thread ffffc805fb7ed080 Thread ffffc805fc46b2c0
Thread ffffc805fb7d8080 Thread ffffc805fb7d6080
Thread ffffc805fc49b080 Thread ffffc805fc599080
Thread ffffc805fc55f080 Thread ffffc805fc389080
Thread ffffc805fc5d9080 Thread ffffc805fc594080
Thread ffffc805fc5cb080 Thread ffffc805fb8e8080
Thread ffffc805fc44d080 Thread ffffc805fc2f73c0
Thread ffffc805fc329080 Thread ffffc805fc334700
Thread ffffc805fc348080 Thread ffffc805fc34c3c0
Thread ffffc805fc33f080 Thread ffffc805fc384080
Thread ffffc805fc381080 Thread ffffc805fc399080
Thread ffffc805fc396080 Thread ffffc805fa6781c0
Thread ffffc805fc3a0080 Thread ffffc805fc620080
Thread ffffc805fc611080 Thread ffffc805fc60a700
Thread ffffc805fc650400 Thread ffffc805fc649080
Thread ffffc805fc63a600 Thread ffffc805fc66c080
Thread ffffc805fc6a5080 Thread ffffc805fc694080
Thread ffffc805fc7dc080 Thread ffffc805fc7d7080
Thread ffffc805fc7da400 Thread ffffc805fc7d4080
Thread ffffc805fc6e7700 Thread ffffc805fc6c0600
Thread ffffc805fc76a140 Thread ffffc805fc65b080
Thread ffffc805fc979080 ApcStateIndex 0 ApcListHead ffffc805fc979118 [KERNEL]
KAPC @ ffffc805fc979308
Type 12
KernelRoutine fffff8033914e850 nt!EmpCheckErrataList+0
RundownRoutine fffff8033914e850 nt!EmpCheckErrataList+0
Process ffffc805fc977080 svchost.exe
Thread ffffc805fc978080Process ffffc805fc97a080 svchost.exe
Thread ffffc805fc97b080 ApcStateIndex 0 ApcListHead ffffc805fc97b118 [KERNEL]
KAPC @ ffffc805fc97b308
Type 12
KernelRoutine fffff8033914e850 nt!EmpCheckErrataList+0
RundownRoutine fffff8033914e850 nt!EmpCheckErrataList+0

There are two APC queues and it looks like you’re looking at an empty one:

4: kd> dt ffffc805fc0fa4c0+0x98 _KAPC_STATE
ntdll!_KAPC_STATE
+0x000 ApcListHead : [2] _LIST_ENTRY [ 0xffffc805fc0fa558 - 0xffffc805fc0fa558 ]

Note that ffffc805fc0fa4c0+0x98 == 0xffffc805`fc0fa558

Try adding a /b at the end of the dt command and seeing what the other list
looks like.

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntfsd…

Thanks Scott. Let me check to make sure it’s OK to share the dump. It’d be
great to get your help with it.

Couple more strange things I noticed with this. When I decode the APC, I’m
either doing it wrong (probably) or there is something really screwy,
because the values look bogus (type and size are all wrong). And while this
crash occurred in services.exe context, when I run the !apc command,
services.exe is not listed as having any APCs.

In the meantime, here’s some more info.

4: kd> !analyze -v
*******************************************************************************
*
*
* Bugcheck Analysis
*
*
*
*******************************************************************************

PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced. This cannot be protected by
try-except.
Typically the address is just plain bad or it is pointing at freed memory.
Arguments:
Arg1: ffffc80500000000, memory referenced.
Arg2: 0000000000000000, value 0 = read operation, 1 = write operation.
Arg3: fffff8033904a335, If non-zero, the instruction address which
referenced the bad memory
address.
Arg4: 0000000000000002, (reserved)

Debugging Details:

DUMP_CLASS: 1

DUMP_QUALIFIER: 402

BUILD_VERSION_STRING: 15063.0.amd64fre.rs2_release.170317-1834

SYSTEM_MANUFACTURER: Dell Inc.

SYSTEM_PRODUCT_NAME: Dell System XPS L502X

SYSTEM_SKU: System SKUNumber

BIOS_VENDOR: Dell Inc.

BIOS_VERSION: A12

BIOS_DATE: 09/07/2012

BASEBOARD_MANUFACTURER: Dell Inc.

BASEBOARD_PRODUCT: 0NJT03

BASEBOARD_VERSION: A00

DUMP_TYPE: 0

BUGCHECK_P1: ffffc80500000000

BUGCHECK_P2: 0

BUGCHECK_P3: fffff8033904a335

BUGCHECK_P4: 2

READ_ADDRESS: ffffc80500000000

FAULTING_IP:
nt!KiDeliverApc+75
fffff803`3904a335 48393f cmp qword ptr [rdi],rdi

MM_INTERNAL_CODE: 2

CPU_COUNT: 8

CPU_MHZ: 7cb

CPU_VENDOR: GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 2a

CPU_STEPPING: 7

CPU_MICROCODE: 6,2a,7,0 (F,M,S,R) SIG: 29’00000000 (cache) 29’00000000
(init)

DEFAULT_BUCKET_ID: CODE_CORRUPTION

BUGCHECK_STR: AV

PROCESS_NAME: services.exe

CURRENT_IRQL: 1

ANALYSIS_SESSION_HOST: LAPTOPWIN10

ANALYSIS_SESSION_TIME: 07-19-2017 18:35:17.0796

ANALYSIS_VERSION: 10.0.10586.567 amd64fre

TRAP_FRAME: ffff9c80941bb5f0 – (.trap 0xffff9c80941bb5f0)
NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=ffffc805fc976ec0 rbx=0000000000000000 rcx=0000000000000002
rdx=ffffc805fc976ec0 rsi=0000000000000000 rdi=0000000000000000
rip=fffff8033904a335 rsp=ffff9c80941bb780 rbp=ffff9c80941bb7d0
r8=ffffc805fc986a50 r9=0000000000000000 r10=ffffc805f6337960
r11=ffffc805fc474d30 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
nt!KiDeliverApc+0x75:
fffff8033904a335 48393f cmp qword ptr [rdi],rdi ds:0000000000000000=???
Resetting default scope

LAST_CONTROL_TRANSFER: from fffff803391b6fb4 to fffff803391814c0

STACK_TEXT:
ffff9c80941bb358 fffff803391b6fb4 : 0000000000000050 ffffc80500000000
0000000000000000 ffff9c80941bb5f0 : nt!KeBugCheckEx
ffff9c80941bb360 fffff803390a22d6 : 0000000000000000 ffffc80500000000
ffff9c80941bb5f0 ffffc805fbe42640 : nt!MiSystemFault+0x116e84
ffff9c80941bb400 fffff8033918ad72 : ffffc805fc976050 0000000000000000
0000000000000000 fffff80300001800 : nt!MmAccessFault+0xae6
ffff9c80941bb5f0 fffff8033904a335 : ffffc805fc474da8 ffffc805f622f390
ffffc805fc979000 0000000000000000 : nt!KiPageFault+0x132
ffff9c80941bb780 fffff8033904bd75 : 0000000000000000 0000000000000000
0000000000000000 ffff9c80941bba00 : nt!KiDeliverApc+0x75
ffff9c80941bb810 fffff803394d3bf9 : ffff9c80941bba50 0000000000000000
ffff9c8000000000 ffff9c80941bba50 : nt!KiCheckForKernelApcDelivery+0x25
ffff9c80941bb840 fffff803394ba6d5 : ffffc805f622f390 ffff9c80941bb9e0
ffff9c80941bc3c0 ffff9c80941bb9e0 : nt!ObpCallPreOperationCallbacks+0x1c9
ffff9c80941bb8c0 fffff803394d1b5e : ffff9c80941bc3c0 0000000000000000
0000000000000000 0000000000000b54 : nt!ObpCreateHandle+0x9d5
ffff9c80941bbb00 fffff803394501e5 : ffffc805fc976080 fffff80339453488
fffff803393970e8 fffff803393972c8 : nt!ObOpenObjectByPointer+0x11e
ffff9c80941bbd60 fffff8033944fe55 : ffff9c80941bc550 0000000000000000
0000000000000000 0000000000000001 : nt!PspCreateObjectHandle+0x3d
ffff9c80941bbdb0 fffff803394379a3 : ffffc805fc979080 ffffc80500000000
ffff9c80941bc010 ffff9c80941bbef8 : nt!PspInsertThread+0x429
ffff9c80941bbe70 fffff8033918c413 : 0000000000000000 0000000000000000
0000000000000000 0000000000000000 : nt!NtCreateUserProcess+0x953
ffff9c80941bca90 00007ffbb1e06b94 : 00007ffbaede4a61 0000000000000003
000000756987e2b0 0000000000000001 : nt!KiSystemServiceCopyEnd+0x13
000000756987e078 00007ffbaede4a61 : 0000000000000003 000000756987e2b0
0000000000000001 0000000000000000 : ntdll!NtCreateUserProcess+0x14
000000756987e080 00007ffbaede7c46 : 0000025586ee8210 00007ffb00000042
0000025586870000 0000000000000070 :
KERNELBASE!CreateProcessInternalW+0xd21
000000756987ec50 00007ffbaf2bba83 : 0000000000000037 0000000000000000
0000000000000550 0000000000000000 : KERNELBASE!CreateProcessW+0x66
000000756987ecc0 00007ff740b39e6f : 0000000000000070 000002558696fdd0
00007ff740b93630 0000025586ee74d0 : KERNEL32!CreateProcessWStub+0x53
000000756987ed20 00007ff740b3acff : 000002558696fdd0 000000756987f240
0000000000000000 0000025586ee74d0 :
services!CWin32ServiceRecord::LogonAndStartImage+0x97b
000000756987f140 00007ff740b3dd5a : 0000000000000000 0000025586ed1ac0
0000000000000018 000002558696fea0 :
services!CWin32ServiceRecord::StartInternal+0x1ef
000000756987f3f0 00007ff740b3ec40 : 0000000000000000 0000025586ed1ac0
000000756987f4a0 00007ff740ba4000 : services!CServiceRecord::Start+0xa2
000000756987f450 00007ff740b3e892 : 0000025586916ba0 0000025500000000
0000000000000000 0000025500000000 :
services!ScStartMarkedServicesInServiceSet+0x19c
000000756987f4e0 00007ff740b3d524 : 000002558696fdd0 000000756987f5f8
00007ff700000010 0000000000000000 :
services!ScStartServicesInStartList+0x272
000000756987f5c0 00007ff740b56007 : 0000000000000000 0000000000000000
0000000000000000 0000025586e391f8 :
services!ScStartServiceAndDependencies+0x1cc
000000756987f690 00007ffbb1da3021 : 0000025586e39130 0000000000000000
000000756987f8a8 000000007ffe0386 : services!ScStartServiceCallback+0x37
000000756987f6d0 00007ffbb1da1989 : 0000025586934500 0000000000000000
00007ffbb1da2ef0 0000000000000000 : ntdll!TppWorkpExecuteCallback+0x131
000000756987f720 00007ffbaf2b2774 : 0000000000000000 0000000000000000
0000000000000000 0000000000000000 : ntdll!TppWorkerThread+0x6c9
000000756987fa30 00007ffbb1dd0d51 : 0000000000000000 0000000000000000
0000000000000000 0000000000000000 : KERNEL32!BaseThreadInitThunk+0x14
000000756987fa60 0000000000000000 : 0000000000000000 0000000000000000
0000000000000000 0000000000000000 : ntdll!RtlUserThreadStart+0x21

STACK_COMMAND: kb

MODULE_NAME: memory_corruption

IMAGE_NAME: memory_corruption

FOLLOWUP_NAME: memory_corruption

DEBUG_FLR_IMAGE_TIMESTAMP: 0

MEMORY_CORRUPTOR: LARGE

FAILURE_BUCKET_ID: MEMORY_CORRUPTION_LARGE

BUCKET_ID: MEMORY_CORRUPTION_LARGE

PRIMARY_PROBLEM_CLASS: MEMORY_CORRUPTION_LARGE

TARGET_TIME: 2017-07-17T11:54:38.000Z

OSBUILD: 15063

OSSERVICEPACK: 0

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK: 272

PRODUCT_TYPE: 1

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

OSEDITION: Windows 10 WinNt TerminalServer SingleUserTS

OS_LOCALE:

USER_LCID: 0

OSBUILD_TIMESTAMP: 2017-07-07 02:06:35

BUILDDATESTAMP_STR: 170317-1834

BUILDLAB_STR: rs2_release

BUILDOSVER_STR: 10.0.15063.0.amd64fre.rs2_release.170317-1834

ANALYSIS_SESSION_ELAPSED_TIME: 1b2f

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:memory_corruption_large

FAILURE_ID_HASH: {e29154ac-69a4-0eb8-172a-a860f73c0a3c}

Followup: memory_corruption

4: kd> !process -1 0
PROCESS ffffc805fbe42640
SessionId: 0 Cid: 03ec Peb: 75694c8000 ParentCid: 0328
DirBase: 12ad83000 ObjectTable: ffffb0893f71ad80 HandleCount: 336.
Image: services.exe

4: kd> !thread -1 0
THREAD ffffc805fc0fa4c0 Cid 03ec.02cc Teb: 00000075694d3000 Win32Thread:
0000000000000000 RUNNING on processor 4

4: kd> dt ffffc805fc0fa4c0 _KTHREAD
ntdll!_KTHREAD
+0x000 Header : _DISPATCHER_HEADER
+0x018 SListFaultAddress : (null)
+0x020 QuantumTarget : 0x871b273
+0x028 InitialStack : 0xffff9c80941bcc90 Void +0x030 StackLimit : 0xffff9c80941b7000 Void
+0x038 StackBase : 0xffff9c80941bd000 Void +0x040 ThreadLock : 0 +0x048 CycleTime : 0x594ade7 +0x050 CurrentRunTime : 0x3fd6ad +0x054 ExpectedRunTime : 0x993c +0x058 KernelStack : 0xffff9c80941bab20 Void
+0x060 StateSaveArea : 0xffff9c80941bccc0 _XSAVE_FORMAT +0x068 SchedulingGroup : (null) +0x070 WaitRegister : _KWAIT_STATUS_REGISTER +0x071 Running : 0x1 '' +0x072 Alerted : [2] "" +0x074 AutoBoostActive : 0y1 +0x074 ReadyTransition : 0y0 +0x074 WaitNext : 0y0 +0x074 SystemAffinityActive : 0y0 +0x074 Alertable : 0y0 +0x074 UserStackWalkActive : 0y0 +0x074 ApcInterruptRequest : 0y0 +0x074 QuantumEndMigrate : 0y0 +0x074 UmsDirectedSwitchEnable : 0y0 +0x074 TimerActive : 0y0 +0x074 SystemThread : 0y0 +0x074 ProcessDetachActive : 0y0 +0x074 CalloutActive : 0y0 +0x074 ScbReadyQueue : 0y0 +0x074 ApcQueueable : 0y1 +0x074 ReservedStackInUse : 0y0 +0x074 UmsPerformingSyscall : 0y0 +0x074 TimerSuspended : 0y0 +0x074 SuspendedWaitMode : 0y0 +0x074 SuspendSchedulerApcWait : 0y0 +0x074 Reserved : 0y000000000000 (0) +0x074 MiscFlags : 0n16385 +0x078 AutoAlignment : 0y0 +0x078 DisableBoost : 0y0 +0x078 BamEppImportant : 0y0 +0x078 AlertedByThreadId : 0y0 +0x078 QuantumDonation : 0y1 +0x078 EnableStackSwap : 0y1 +0x078 GuiThread : 0y0 +0x078 DisableQuantum : 0y0 +0x078 ChargeOnlySchedulingGroup : 0y0 +0x078 DeferPreemption : 0y0 +0x078 QueueDeferPreemption : 0y0 +0x078 ForceDeferSchedule : 0y0 +0x078 SharedReadyQueueAffinity : 0y1 +0x078 FreezeCount : 0y0 +0x078 TerminationApcRequest : 0y0 +0x078 AutoBoostEntriesExhausted : 0y0 +0x078 KernelStackResident : 0y1 +0x078 TerminateRequestReason : 0y00 +0x078 ProcessStackCountDecremented : 0y0 +0x078 RestrictedGuiThread : 0y0 +0x078 ThreadFlagsSpare : 0y000 +0x078 EtwStackTraceApcInserted : 0y00000000 (0) +0x078 ThreadFlags : 0n69680 +0x07c Tag : 0 '' +0x07d SystemHeteroCpuPolicy : 0 '' +0x07e UserHeteroCpuPolicy : 0y0001000 (0x8) +0x07e ExplicitSystemHeteroCpuPolicy : 0y0 +0x07f Spare0 : 0 '' +0x080 SystemCallNumber : 0xc0 +0x084 ReadyTime : 0 +0x088 FirstArgument : 0x000000756987e160 Void
+0x090 TrapFrame : (null)
+0x098 ApcState : _KAPC_STATE
+0x098 ApcStateFill : [43] “X???”
+0x0c3 Priority : 9 ‘’
+0x0c4 UserIdealProcessor : 2
+0x0c8 WaitStatus : 0n0
+0x0d0 WaitBlockList : 0xffffc805fc0fa600 _KWAIT_BLOCK +0x0d8 WaitListEntry : _LIST_ENTRY [ 0x0000000000000000 -
0xfffff80337ca6860 ] +0x0d8 SwapListEntry : _SINGLE_LIST_ENTRY +0x0e8 Queue : 0xffffc805fa1ecf80 _DISPATCHER_HEADER
+0x0f0 Teb : 0x00000075694d3000 Void +0x0f8 RelativeTimerBias : 0 +0x100 Timer : _KTIMER +0x140 WaitBlock : [4] _KWAIT_BLOCK +0x140 WaitBlockFill4 : [20] " .f???" +0x154 ContextSwitches : 0x1b4 +0x140 WaitBlockFill5 : [68] " .f???" +0x184 State : 0x2 '' +0x185 Spare13 : 0 '' +0x186 WaitIrql : 0 '' +0x187 WaitMode : 0 '' +0x140 WaitBlockFill6 : [116] " .f???" +0x1b4 WaitTime : 0xcf2 +0x140 WaitBlockFill7 : [164] " .f???" +0x1e4 KernelApcDisable : 0n-2 +0x1e6 SpecialApcDisable : 0n0 +0x1e4 CombinedApcDisable : 0xfffe +0x140 WaitBlockFill8 : [40] " .f???" +0x168 ThreadCounters : (null) +0x140 WaitBlockFill9 : [88] " .f???" +0x198 XStateSave : (null) +0x140 WaitBlockFill10 : [136] " .f???" +0x1c8 Win32Thread : (null) +0x140 WaitBlockFill11 : [176] " .f???" +0x1f0 Ucb : (null) +0x1f8 Uch : (null) +0x200 Spare21 : (null) +0x208 QueueListEntry : _LIST_ENTRY [ 0xffffc805fc0fe288 -
0xffffc805fc0f5488 ] +0x218 NextProcessor : 4 +0x218 NextProcessorNumber : 0y0000000000000000000000000000100 (0x4) +0x218 SharedReadyQueue : 0y0 +0x21c QueuePriority : 0n0 +0x220 Process : 0xffffc805fbe42640 _KPROCESS
+0x228 UserAffinity : _GROUP_AFFINITY
+0x228 UserAffinityFill : [10] “???”
+0x232 PreviousMode : 1 ‘’
+0x233 BasePriority : 8 ‘’
+0x234 PriorityDecrement : 0 ‘’
+0x234 ForegroundBoost : 0y0000
+0x234 UnusualBoost : 0y0000
+0x235 Preempted : 0 ‘’
+0x236 AdjustReason : 0 ‘’
+0x237 AdjustIncrement : 1 ‘’
+0x238 AffinityVersion : 0x24
+0x240 Affinity : _GROUP_AFFINITY
+0x240 AffinityFill : [10] “???”
+0x24a ApcStateIndex : 0 ‘’
+0x24b WaitBlockCount : 0x1 ‘’
+0x24c IdealProcessor : 2
+0x250 NpxState : 5
+0x258 SavedApcState : _KAPC_STATE
+0x258 SavedApcStateFill : [43] “???”
+0x283 WaitReason : 0x6 ‘’
+0x284 SuspendCount : 0 ‘’
+0x285 Saturation : 0 ‘’
+0x286 SListFaultCount : 0
+0x288 SchedulerApc : _KAPC
+0x288 SchedulerApcFill0 : [1] “???”
+0x289 ResourceIndex : 0x1 ‘’
+0x288 SchedulerApcFill1 : [3] “???”
+0x28b QuantumReset : 0x6 ‘’
+0x288 SchedulerApcFill2 : [4] “???”
+0x28c KernelTime : 4
+0x288 SchedulerApcFill3 : [64] “???”
+0x2c8 WaitPrcb : (null)
+0x288 SchedulerApcFill4 : [72] “???”
+0x2d0 LegoData : (null)
+0x288 SchedulerApcFill5 : [83] “???”
+0x2db CallbackNestingLevel : 0 ‘’
+0x2dc UserTime : 0
+0x2e0 SuspendEvent : _KEVENT
+0x2f8 ThreadListEntry : _LIST_ENTRY [ 0xffffc805fc0fe378 - 0xffffc805fc0f5578 ]
+0x308 MutantListHead : _LIST_ENTRY [ 0xffffc805fc0fa7c8 - 0xffffc805fc0fa7c8 ]
+0x318 AbEntrySummary : 0x3f ‘?’
+0x319 AbWaitEntryCount : 0 ‘’
+0x31a AbAllocationRegionCount : 0 ‘’
+0x31b Spare20 : 0 ‘’
+0x31c SecureThreadCookie : 0
+0x320 LockEntries : [6] _KLOCK_ENTRY
+0x560 PropagateBoostsEntry : _SINGLE_LIST_ENTRY
+0x568 IoSelfBoostsEntry : _SINGLE_LIST_ENTRY
+0x570 PriorityFloorCounts : [16] “”
+0x580 PriorityFloorSummary : 0
+0x584 AbCompletedIoBoostCount : 0n0
+0x588 AbCompletedIoQoSBoostCount : 0n0
+0x58c KeReferenceCount : 0n0
+0x58e AbOrphanedEntrySummary : 0 ‘’
+0x58f AbOwnedEntryCount : 0 ‘’
+0x590 ForegroundLossTime : 0
+0x598 GlobalForegroundListEntry : _LIST_ENTRY [ 0x0000000000000001 - 0x0000000000000000 ]
+0x598 ForegroundDpcStackListEntry : _SINGLE_LIST_ENTRY
+0x5a0 InGlobalForegroundList : 0
+0x5a8 ReadOperationCount : 0n0
+0x5b0 WriteOperationCount : 0n1
+0x5b8 OtherOperationCount : 0n364
+0x5c0 ReadTransferCount : 0n0
+0x5c8 WriteTransferCount : 0n1080
+0x5d0 OtherTransferCount : 0n2102
+0x5d8 QueuedScb : (null)
+0x5e0 ThreadTimerDelay : 0
+0x5e4 Spare22 : 0n0

4: kd> dt ffffc805fc0fa4c0+0x98 _KAPC_STATE
ntdll!_KAPC_STATE
+0x000 ApcListHead : [2] _LIST_ENTRY [ 0xffffc805fc0fa558 - 0xffffc805fc0fa558 ]
+0x020 Process : 0xffffc805`fbe42640 _KPROCESS
+0x028 InProgressFlags : 0 ‘’
+0x028 KernelApcInProgress : 0y0
+0x028 SpecialApcInProgress : 0y0
+0x029 KernelApcPending : 0 ‘’
+0x02a UserApcPending : 0 ‘’

4: kd> dt 0xffffc805fc0fa558-0x10 _KAPC ntdll!_KAPC +0x000 Type : 0x60 '’
+0x001 SpareByte0 : 0xe1 ‘’
+0x002 Size : 0x87 ‘’
+0x003 SpareByte1 : 0x69 ‘i’
+0x004 SpareLong0 : 0x75
+0x008 Thread : (null)
+0x010 ApcListEntry : _LIST_ENTRY [ 0xffffc805fc0fa558 - 0xffffc805fc0fa558 ]
+0x020 KernelRoutine : 0xffffc805fc0fa568 void +ffffc805fc0fa568 +0x028 RundownRoutine : 0xffffc805fc0fa568 void +ffffc805fc0fa568
+0x030 NormalRoutine : 0xffffc805fbe42640 void +ffffc805fbe42640 +0x020 Reserved : [3] 0xffffc805fc0fa568 Void
+0x038 NormalContext : 0x0000000209000000 Void +0x040 SystemArgument1 : (null) +0x048 SystemArgument2 : 0xffffc805fc0fa600 Void
+0x050 ApcStateIndex : 0 ‘’
+0x051 ApcMode : 0 ‘’
+0x052 Inserted : 0 ‘’

4: kd> !apc
*** Enumerating APCs in all processes
Process ffffc805f62b4040 System
Thread ffffc805f6304040 Thread ffffc805f6335040
Thread ffffc805fa220700 Thread ffffc805faf6a700
Process ffffc805fb0e7080 csrss.exe
Thread ffffc805fa7fd080 Thread ffffc805fafcf080
Process ffffc805fbd68080 csrss.exe
Thread ffffc805fbd64500 Thread ffffc805fbd61080
Thread ffffc805fbdff080 Thread ffffc805fbe07080
Thread ffffc805fbe6c080 Thread ffffc805fbda5080
Thread ffffc805fbe43080 Thread ffffc805fbe3b080
Thread ffffc805fbe32080 Thread ffffc805fbe2f080
Thread ffffc805fbe2b080 Thread ffffc805fbe40080
Thread ffffc805fc10c080 Thread ffffc805fc113700
Thread ffffc805fc12d080 Thread ffffc805fc131080
Thread ffffc805fc12c080 Thread ffffc805fc132080
Thread ffffc805fc144080 Thread ffffc805fc146080
Thread ffffc805fc1bb080 Thread ffffc805fc1bd3c0
Thread ffffc805fb635080 Thread ffffc805fb622080
Thread ffffc805fb6ad080 Thread ffffc805fb6ab080
Thread ffffc805fb6a1080 Thread ffffc805fa425700
Thread ffffc805fb74a6c0 Thread ffffc805fb73c080
Thread ffffc805fb741080 Thread ffffc805fbd295c0
Thread ffffc805fb76d080 Thread ffffc805fb763080
Thread ffffc805fb79b080 Thread
Thread ffffc805fb79a680 Thread ffffc805fb7ff080
Thread ffffc805fc459080 Thread ffffc805fc4553c0
Thread ffffc805fc446080 Thread ffffc805fc44a580
Thread ffffc805fb75c080 Thread ffffc805fb7d0080
Thread ffffc805fb66f540 Thread ffffc805fb7cd080
Thread ffffc805fb7f24c0 Thread ffffc805fb7e7080
Thread ffffc805fb7ed080 Thread ffffc805fc46b2c0
Thread ffffc805fb7d8080 Thread ffffc805fb7d6080
Thread ffffc805fc49b080 Thread ffffc805fc599080
Thread ffffc805fc55f080 Thread ffffc805fc389080
Thread ffffc805fc5d9080 Thread ffffc805fc594080
Thread ffffc805fc5cb080 Thread ffffc805fb8e8080
Thread ffffc805fc44d080 Thread ffffc805fc2f73c0
Thread ffffc805fc329080 Thread ffffc805fc334700
Thread ffffc805fc348080 Thread ffffc805fc34c3c0
Thread ffffc805fc33f080 Thread ffffc805fc384080
Thread ffffc805fc381080 Thread ffffc805fc399080
Thread ffffc805fc396080 Thread ffffc805fa6781c0
Thread ffffc805fc3a0080 Thread ffffc805fc620080
Thread ffffc805fc611080 Thread ffffc805fc60a700
Thread ffffc805fc650400 Thread ffffc805fc649080
Thread ffffc805fc63a600 Thread ffffc805fc66c080
Thread ffffc805fc6a5080 Thread ffffc805fc694080
Thread ffffc805fc7dc080 Thread ffffc805fc7d7080
Thread ffffc805fc7da400 Thread ffffc805fc7d4080
Thread ffffc805fc6e7700 Thread ffffc805fc6c0600
Thread ffffc805fc76a140 Thread ffffc805fc65b080
Thread ffffc805fc979080 ApcStateIndex 0 ApcListHead ffffc805fc979118
[KERNEL]
KAPC @ ffffc805fc979308
Type 12
KernelRoutine fffff8033914e850 nt!EmpCheckErrataList+0
RundownRoutine fffff8033914e850 nt!EmpCheckErrataList+0
Process ffffc805fc977080 svchost.exe
Thread ffffc805fc978080Process ffffc805fc97a080 svchost.exe
Thread ffffc805fc97b080 ApcStateIndex 0 ApcListHead ffffc805fc97b118
[KERNEL]
KAPC @ ffffc805fc97b308
Type 12
KernelRoutine fffff8033914e850 nt!EmpCheckErrataList+0
RundownRoutine fffff8033914e850 nt!EmpCheckErrataList+0

Thanks for pointing that out Scott. Here are the various APC structures from the KTHREAD. It doesn’t look like the second values are any more valid than the first (just +0x10 more).

+0x098 ApcState : _KAPC_STATE
+0x000 ApcListHead :
[00] _LIST_ENTRY [0xffffc805fc0fa558 - 0xffffc805fc0fa558]
+0x000 Flink : 0xffffc805fc0fa558 +0x008 Blink : 0xffffc805fc0fa558
[01] [0xffffc805fc0fa568 - 0xffffc805fc0fa568]
+0x000 Flink : 0xffffc805fc0fa568 +0x008 Blink : 0xffffc805fc0fa568
+0x020 Process : 0xffffc805`fbe42640
+0x028 InProgressFlags : 0 ‘’
+0x028 KernelApcInProgress : 0y0
+0x028 SpecialApcInProgress : 0y0
+0x029 KernelApcPending : 0 ‘’
+0x02a UserApcPending : 0 ‘’

+0x288 SchedulerApc : _KAPC
+0x000 Type : 0x12 ‘’
+0x001 SpareByte0 : 0x1 ‘’
+0x002 Size : 0x58 ‘X’
+0x003 SpareByte1 : 0x6 ‘’
+0x004 SpareLong0 : 4
+0x008 Thread : 0xffffc805fc0fa4c0 +0x010 ApcListEntry : _LIST_ENTRY [0x0000000000000000 - 0x0000000000000000] +0x000 Flink : (null) +0x008 Blink : (null) +0x020 KernelRoutine : 0xfffff8033914e850
+0x028 RundownRoutine : 0xfffff8033914e850 +0x030 NormalRoutine : 0xfffff8033902fe00
+0x020 Reserved :
[00] 0xfffff8033914e850 [01] 0xfffff8033914e850
[02] 0xfffff8033902fe00 +0x038 NormalContext : 0xffffc805fc0fa4c0
+0x040 SystemArgument1 : (null)
+0x048 SystemArgument2 : (null)
+0x050 ApcStateIndex : 0 ‘’
+0x051 ApcMode : 0 ‘’
+0x052 Inserted : 0 ‘’

+0x258 SavedApcState : _KAPC_STATE
+0x000 ApcListHead :
[00] _LIST_ENTRY [0xffffc805fc0fa718 - 0xffffc805fc0fa718]
+0x000 Flink : 0xffffc805fc0fa718 +0x008 Blink : 0xffffc805fc0fa718
[01] [0xffffc805fc0fa728 - 0xffffc805fc0fa728]
+0x000 Flink : 0xffffc805fc0fa728 +0x008 Blink : 0xffffc805fc0fa728
+0x020 Process : (null)
+0x028 InProgressFlags : 0 ‘’
+0x028 KernelApcInProgress : 0y0
+0x028 SpecialApcInProgress : 0y0
+0x029 KernelApcPending : 0 ‘’
+0x02a UserApcPending : 0 ‘’

I’m not seeing where the bad address of ffffc80500000000 came from. Either
we looped or something weird happened. Interesting that it’s the thread
address with the low 32-bits shaved off…

What’s in RBX in the KiDeliverApc frame? (k ; .frame /c is>)

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntfsd…

Thanks for pointing that out Scott. Here are the various APC structures
from the KTHREAD. It doesn’t look like the second values are any more valid
than the first (just +0x10 more).

+0x098 ApcState : _KAPC_STATE
+0x000 ApcListHead :
[00] _LIST_ENTRY [0xffffc805fc0fa558 - 0xffffc805fc0fa558]
+0x000 Flink : 0xffffc805fc0fa558<br> +0x008 Blink : 0xffffc805fc0fa558
[01] [0xffffc805fc0fa568 - 0xffffc805fc0fa568]
+0x000 Flink : 0xffffc805fc0fa568<br> +0x008 Blink : 0xffffc805fc0fa568
+0x020 Process : 0xffffc805fbe42640<br> +0x028 InProgressFlags : 0 ''<br> +0x028 KernelApcInProgress : 0y0<br> +0x028 SpecialApcInProgress : 0y0<br> +0x029 KernelApcPending : 0 ''<br> +0x02a UserApcPending : 0 ''<br><br>+0x288 SchedulerApc : _KAPC<br> +0x000 Type : 0x12 ''<br> +0x001 SpareByte0 : 0x1 ''<br> +0x002 Size : 0x58 'X'<br> +0x003 SpareByte1 : 0x6 ''<br> +0x004 SpareLong0 : 4<br> +0x008 Thread : 0xffffc805fc0fa4c0
+0x010 ApcListEntry : _LIST_ENTRY [0x0000000000000000 - <br>0x0000000000000000]
+0x000 Flink : (null)
+0x008 Blink : (null)
+0x020 KernelRoutine : 0xfffff8033914e850<br> +0x028 RundownRoutine : 0xfffff8033914e850
+0x030 NormalRoutine : 0xfffff8033902fe00<br> +0x020 Reserved :<br> [00] 0xfffff8033914e850
[01] 0xfffff8033914e850<br> [02] 0xfffff8033902fe00
+0x038 NormalContext : 0xffffc805fc0fa4c0<br> +0x040 SystemArgument1 : (null)<br> +0x048 SystemArgument2 : (null)<br> +0x050 ApcStateIndex : 0 ''<br> +0x051 ApcMode : 0 ''<br> +0x052 Inserted : 0 ''<br><br>+0x258 SavedApcState : _KAPC_STATE<br> +0x000 ApcListHead :<br> [00] _LIST_ENTRY [0xffffc805fc0fa718 - 0xffffc805fc0fa718]<br> +0x000 Flink : 0xffffc805fc0fa718
+0x008 Blink : 0xffffc805fc0fa718<br> [01] [0xffffc805fc0fa728 - 0xffffc805fc0fa728]<br> +0x000 Flink : 0xffffc805fc0fa728
+0x008 Blink : 0xffffc805`fc0fa728
+0x020 Process : (null)
+0x028 InProgressFlags : 0 ‘’
+0x028 KernelApcInProgress : 0y0
+0x028 SpecialApcInProgress : 0y0
+0x029 KernelApcPending : 0 ‘’
+0x02a UserApcPending : 0 ‘’

Yes, in many cases this indicates an asynchronous stack or memory corruption.

Zeroing the lower half is consistent with Irp->UserIosb.Status = STATUS_SUCCESS when UserIosb points to a registers frame pushed on a thread stack during interrupt processing, e.g. to deliver APC by invoking KiApcInterrupt through LAPIC.

It might be that in the past the damage was at an address below thread’s RSP value so it was benign. The added code has changed timing and thread managed to progress further before completion APC being inserted for it.

For this particular case when a crash happened in KiDeliverApc the RDI register might have been changed when being restored by a completion APC routine invoked by KiDeliverApc.

Below is my speculation on how the RDI register was changed

KiDeliverApc()
{

//
// RDI == &Thread->ApcState.ApcListHead[KernelMode]
//
// IsListEmpty is translated to cmp and je machine instructions
//
// cmp qword ptr [rdi],rdi
// je nt!KiDeliverApc+Offset_Outside_While
//
while( ! IsListEmpty( &Thread->ApcState.ApcListHead[KernelMode] )
{
//
// RDI is an invariant in the cycle and points to &Thread->ApcState.ApcListHead[KernelMode]
//
ApcEntry = RemoveHeadList( &Thread->ApcState.ApcListHead[KernelMode] );
Apc = CONTAINING_RECORD(ApcEntry, KAPC, ApcListEntry);
CompletionRoutine = Apc->KernelRoutine;
…
call CompletionRoutine
{
//
// inside CompletionRoutine
//
…
push …
push RDI // after push RSP == Irp->UserIosb
push …
push …
…
Irp->UserIosb.Status = STATUS_SUCCESS // this changes the saved RDI
…
pop …
pop …
pop RDI // the changed RDI is restored from the stack
pop …
ret
}
//
// go to while with RDI having its lower half being zeroed
//
}

}

Thanks Slava.

I got another dump from the same system and this one is stranger than the first. Any thoughts on this one? Could these be a sign of hardware failure? In the meantime I’ve asked the client to disable my software for a few days to see if another crash occurs. Fingers crossed that one does because at least I won’t have to keep chasing ghosts.

4: kd> !analyze -v
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the bugcheck
Arg2: fffff80182a46357, Address of the instruction which caused the bugcheck
Arg3: ffffbd000e408300, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.

Debugging Details:

DUMP_CLASS: 1

DUMP_QUALIFIER: 402

BUILD_VERSION_STRING: 15063.0.amd64fre.rs2_release.170317-1834

SYSTEM_MANUFACTURER: Dell Inc.

SYSTEM_PRODUCT_NAME: Dell System XPS L502X

SYSTEM_SKU: System SKUNumber

BIOS_VENDOR: Dell Inc.

BIOS_VERSION: A12

BIOS_DATE: 09/07/2012

BASEBOARD_MANUFACTURER: Dell Inc.

BASEBOARD_PRODUCT: 0NJT03

BASEBOARD_VERSION: A00

DUMP_TYPE: 0

BUGCHECK_P1: c0000005

BUGCHECK_P2: fffff80182a46357

BUGCHECK_P3: ffffbd000e408300

BUGCHECK_P4: 0

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

FAULTING_IP:
nt!PspCallProcessNotifyRoutines+14f
fffff801`82a46357 488b45a8 mov rax,qword ptr [rbp-58h]

CONTEXT: ffffbd000e408300 – (.cxr 0xffffbd000e408300)
rax=0000000000000000 rbx=ffffbd000e408d20 rcx=fffff8018298a4c8
rdx=ffffd1885a6740b0 rsi=ffffd1885a7f6400 rdi=0000000000000040
rip=fffff80182a46357 rsp=ffffbd000e408cf0 rbp=0000000000000438
r8=ffffd1885a6740bf r9=ffffd1885a6740b0 r10=7ffffffffffffffc
r11=ffffbd000e408ce0 r12=fffff8018298a4c8 r13=0000000000000000
r14=0000000000000000 r15=0000000000000001
iopl=0 nv up ei pl zr na po nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010246
nt!PspCallProcessNotifyRoutines+0x14f:
fffff80182a46357 488b45a8 mov rax,qword ptr [rbp-58h] ss:0018:00000000000003e0=???
Resetting default scope

CPU_COUNT: 8

CPU_MHZ: 7cb

CPU_VENDOR: GenuineIntel

CPU_FAMILY: 6

CPU_MODEL: 2a

CPU_STEPPING: 7

CPU_MICROCODE: 6,2a,7,0 (F,M,S,R) SIG: 29’00000000 (cache) 29’00000000 (init)

DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT

BUGCHECK_STR: 0x3B

PROCESS_NAME: services.exe

CURRENT_IRQL: 0

LAST_CONTROL_TRANSFER: from fffff80182a43006 to fffff80182a46357

STACK_TEXT:
ffffbd000e408cf0 fffff80182a43006 : ffffffff00000000 ffffbd000e409550 ffffbd000e409b01 ffffd1885a7819d0 : nt!PspCallProcessNotifyRoutines+0x14f
ffffbd000e408db0 fffff80182a2a9a3 : ffffd1885a7f8080 ffffd18800000000 ffffbd000e409010 ffffbd000e408ef8 : nt!PspInsertThread+0x5da
ffffbd000e408e70 fffff8018277f413 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!NtCreateUserProcess+0x953
ffffbd000e409a90 00007ff964d56b94 : 00007ff961e14a61 0000000000000004 000000c37dafe050 0000000000000001 : nt!KiSystemServiceCopyEnd+0x13
000000c37dafde18 00007ff961e14a61 : 0000000000000004 000000c37dafe050 0000000000000001 0000000000000000 : ntdll!NtCreateUserProcess+0x14
000000c37dafde20 00007ff961e43233 : 0000023625b05628 0000023625846520 0000000000000002 00007ff964cbb1d9 : KERNELBASE!CreateProcessInternalW+0xd21
000000c37dafe9f0 00007ff9645dc09f : 0000023600000001 00000000000004c0 0000023600000004 0000023600000030 : KERNELBASE!CreateProcessAsUserW+0x63
000000c37dafea60 00007ff6d7fb9a36 : 000000c37dafec68 000000c37dafeba8 0000000000000000 0000000000000000 : KERNEL32!CreateProcessAsUserWStub+0x5f
000000c37dafead0 00007ff6d7fbacff : 00000236254374c0 000000c37dafeff0 0000000000000000 00000236258a8050 : services!CWin32ServiceRecord::LogonAndStartImage+0x542
000000c37dafeef0 00007ff6d7fbdd5a : 0000000000000000 0000023625846280 0000000000000018 0000023625437590 : services!CWin32ServiceRecord::StartInternal+0x1ef
000000c37daff1a0 00007ff6d7fbec40 : 0000000000000000 0000023625846280 000000c37daff250 00007ff6d8024000 : services!CServiceRecord::Start+0xa2
000000c37daff200 00007ff6d7fbe892 : 0000023625416aa0 0000023600000000 0000000000000000 0000023600000000 : services!ScStartMarkedServicesInServiceSet+0x19c
000000c37daff290 00007ff6d7fbd524 : 00000236254374c0 000000c37daff3a8 00007ff600000010 0000000000000000 : services!ScStartServicesInStartList+0x272
000000c37daff370 00007ff6d7fd6007 : 0000000000000000 00007ff900000000 0000000000000000 00000236258386b8 : services!ScStartServiceAndDependencies+0x1cc
000000c37daff440 00007ff964cf3021 : 00000236258385f0 0000000000000000 000000c37daff658 000000007ffe0386 : services!ScStartServiceCallback+0x37
000000c37daff480 00007ff964cf1989 : 00000236258a15a0 0000000000000000 00007ff964cf2ef0 0000000000000000 : ntdll!TppWorkpExecuteCallback+0x131
000000c37daff4d0 00007ff9645d2774 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : ntdll!TppWorkerThread+0x6c9
000000c37daff7e0 00007ff964d20d51 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : KERNEL32!BaseThreadInitThunk+0x14
000000c37daff810 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : ntdll!RtlUserThreadStart+0x21

THREAD_SHA1_HASH_MOD_FUNC: 84460b6870b9160c6408e8981a801397ec66e1ac

THREAD_SHA1_HASH_MOD_FUNC_OFFSET: 64dff8121a57faf795617508fa8197b43e594dc0

THREAD_SHA1_HASH_MOD: 874a7bc38ab4ed8610eb8a61d70245c818118c94

FOLLOWUP_IP:
nt!PspCallProcessNotifyRoutines+14f
fffff801`82a46357 488b45a8 mov rax,qword ptr [rbp-58h]

FAULT_INSTR_CODE: a8458b48

SYMBOL_STACK_INDEX: 0

SYMBOL_NAME: nt!PspCallProcessNotifyRoutines+14f

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: nt

IMAGE_NAME: ntkrnlmp.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 595f24eb

STACK_COMMAND: .cxr 0xffffbd000e408300 ; kb

BUCKET_ID_FUNC_OFFSET: 14f

FAILURE_BUCKET_ID: 0x3B_nt!PspCallProcessNotifyRoutines

BUCKET_ID: 0x3B_nt!PspCallProcessNotifyRoutines

PRIMARY_PROBLEM_CLASS: 0x3B_nt!PspCallProcessNotifyRoutines

TARGET_TIME: 2017-07-22T11:56:13.000Z

OSBUILD: 15063

OSSERVICEPACK: 0

SERVICEPACK_NUMBER: 0

OS_REVISION: 0

SUITE_MASK: 272

PRODUCT_TYPE: 1

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

OSEDITION: Windows 10 WinNt TerminalServer SingleUserTS

OS_LOCALE:

USER_LCID: 0

OSBUILD_TIMESTAMP: 2017-07-07 02:06:35

BUILDDATESTAMP_STR: 170317-1834

BUILDLAB_STR: rs2_release

BUILDOSVER_STR: 10.0.15063.0.amd64fre.rs2_release.170317-1834

ANALYSIS_SESSION_ELAPSED_TIME: 5f7

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:0x3b_nt!pspcallprocessnotifyroutines

FAILURE_ID_HASH: {fb8f0529-bf66-dc2b-aef9-2bd5de7dec71}

4: kd> k

Child-SP RetAddr Call Site

00 ffffbd000e407a28 fffff8018277f8a9 nt!KeBugCheckEx
01 ffffbd000e407a30 fffff8018277f03c nt!KiBugCheckDispatch+0x69
02 ffffbd000e407b70 fffff8018277a99d nt!KiSystemServiceHandler+0x7c
03 ffffbd000e407bb0 fffff80182633d94 nt!RtlpExecuteHandlerForException+0xd
04 ffffbd000e407be0 fffff80182632b36 nt!RtlDispatchException+0x404
05 ffffbd000e4082d0 fffff8018277f98e nt!KiDispatchException+0x1f6
06 ffffbd000e408980 fffff8018277de57 nt!KiExceptionDispatch+0xce
07 ffffbd000e408b60 fffff80182a46357 nt!KiPageFault+0x217
08 ffffbd000e408cf0 fffff80182a43006 nt!PspCallProcessNotifyRoutines+0x14f
09 ffffbd000e408db0 fffff80182a2a9a3 nt!PspInsertThread+0x5da
0a ffffbd000e408e70 fffff8018277f413 nt!NtCreateUserProcess+0x953
0b ffffbd000e409a90 00007ff964d56b94 nt!KiSystemServiceCopyEnd+0x13
0c 000000c37dafde18 00007ff961e14a61 ntdll!NtCreateUserProcess+0x14
0d 000000c37dafde20 00007ff961e43233 KERNELBASE!CreateProcessInternalW+0xd21
0e 000000c37dafe9f0 00007ff9645dc09f KERNELBASE!CreateProcessAsUserW+0x63
0f 000000c37dafea60 00007ff6d7fb9a36 KERNEL32!CreateProcessAsUserWStub+0x5f
10 000000c37dafead0 00007ff6d7fbacff services!CWin32ServiceRecord::LogonAndStartImage+0x542
11 000000c37dafeef0 00007ff6d7fbdd5a services!CWin32ServiceRecord::StartInternal+0x1ef
12 000000c37daff1a0 00007ff6d7fbec40 services!CServiceRecord::Start+0xa2
13 000000c37daff200 00007ff6d7fbe892 services!ScStartMarkedServicesInServiceSet+0x19c
14 000000c37daff290 00007ff6d7fbd524 services!ScStartServicesInStartList+0x272
15 000000c37daff370 00007ff6d7fd6007 services!ScStartServiceAndDependencies+0x1cc
16 000000c37daff440 00007ff964cf3021 services!ScStartServiceCallback+0x37
17 000000c37daff480 00007ff964cf1989 ntdll!TppWorkpExecuteCallback+0x131
18 000000c37daff4d0 00007ff9645d2774 ntdll!TppWorkerThread+0x6c9
19 000000c37daff7e0 00007ff964d20d51 KERNEL32!BaseThreadInitThunk+0x14
1a 000000c37daff810 0000000000000000 ntdll!RtlUserThreadStart+0x21

4: kd> .frame /c 8
08 ffffbd000e408cf0 fffff80182a43006 nt!PspCallProcessNotifyRoutines+0x14f
rax=0000000000000000 rbx=ffffbd000e408d20 rcx=fffff8018298a4c8
rdx=ffffd1885a6740b0 rsi=ffffd1885a7f6400 rdi=0000000000000040
rip=fffff80182a46357 rsp=ffffbd000e408cf0 rbp=0000000000000438
r8=ffffd1885a6740bf r9=ffffd1885a6740b0 r10=7ffffffffffffffc
r11=ffffbd000e408ce0 r12=fffff8018298a4c8 r13=0000000000000000
r14=0000000000000000 r15=0000000000000001
iopl=0 nv up ei ng nz na po nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00000286
nt!PspCallProcessNotifyRoutines+0x14f:
fffff80182a46357 488b45a8 mov rax,qword ptr [rbp-58h] ss:0018:00000000000003e0=???

If there is a hardware failure ( like memory or memory controller ), the user should have experienced user mode application crashes and crashes in the idle thread contexts where the system spends most of the time. Though the failure might be isolated in a particular physical address range which is never used to map virtual addresses of a user application.

In the last dump the RBP register was changed. RBP is a pointer to a call frame and its usage is pretty consistent, in a function prologue

push rbp
mov rbp, rsp

in a function epilogue

pop rbp

The damage is consistent with the stack being changed and then a modified value is restored from the stack to RBP.

rbp=0000000000000438

This might be a number of bytes transferred which is put to Irp->UserIosb.Information.

1. How re you allocating the memory for your linked list? Are you linking
   together extension, or allocating.
2. Are there any KeWaitForXxx() in your code?
3. It almost sounds like the kernel stack is being swapped out.
4. Maybe try KeEnterCriticalRegion()/KeLeaveCriticalRegion()?

On Tue, Jul 25, 2017 at 1:59 AM xxxxx@hotmail.com
wrote:

> If there is a hardware failure ( like memory or memory controller ), the
> user should have experienced user mode application crashes and crashes in
> the idle thread contexts where the system spends most of the time. Though
> the failure might be isolated in a particular physical address range which
> is never used to map virtual addresses of a user application.
>
> In the last dump the RBP register was changed. RBP is a pointer to a call
> frame and its usage is pretty consistent, in a function prologue
>
> push rbp
> mov rbp, rsp
>
> in a function epilogue
>
> pop rbp
>
> The damage is consistent with the stack being changed and then a modified
> value is restored from the stack to RBP.
>
> rbp=0000000000000438
>
> This might be a number of bytes transferred which is put to
> Irp->UserIosb.Information.
>
> —
> NTFSD is sponsored by OSR
>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

Thanks again Slava,

I’m having a bit of trouble following your conceptual example. I guess I don’t understand how Irp->UserIosb.Status could point to RDI in the first place.

Thanks Jamey,

1. Everything is allocated.
2. Yes. There are a number of KeWaits. These mainly occur when I pend an operation and have to notify user mode.
3. What are you suggesting I should wrap with KeEnter/LeaveCriticalRegion?

The idea is as follows.

There are a number of nested function calls on the stack ( F1 calls F2 etc )

F1
F2
F3

Each function has a stack frame allocated for local variables, saved caller registers and for spilling registers. For example F3 does the following

F3()
{
IO_STATUS_BLOCK IoStatusBlock;

Irp = IoAllocateIrp();

Irp->UserIosb = &IoStatusBlock;
RC = IoCallDriver();
return RC;
}

IoStatusBlock is allocated on the stack. If RC == STATUS_PENDING an APC queued to complete the Irp in the thread context can be executed after F3 returned to its caller F2 which already called another function F4. This happens because F3 fails to wait for IRP completion if RC == STATUS_PENDING.

F1
F2
F4

As a result the APC is executed with Irp->UserIosb pointing to a stack frame for F4. The stack frame is used to save and restore registers. The APC uses Irp->UserIosb to report the completion status to a caller. But Irp->UserIosb points to a stack frame that doesn’t belong to a caller F3 anymore.

In this manner APC can overwrite values for registers pushed on the stack.