vmss2core complete memory dmp

[Malcolm_McCaffery]

I was asked to look at issue with Citrix Server. Unfortunately it was not configured to save memory dumps, or event logs (They were cleaned on reboot) So I only have VMWare Snapshot (.vmsn) I converted it to a complete memory .dmp using vmss2core

https://labs.vmware.com/flings/vmss2core

This is the first time I’ve attempted to analyze such a dump file.

The System Logs for 4 other Citrix servers using same “image” went back further and identified these crashes.

WerFault - 87,808 crashes with error:

The instruction at 0x00000023 referenced memory at 0x00000023.

Which matches this description:

After a server runs for several days, attempts to launch a 32-bit application can fail. The application exits unexpectedly and the following error message appears:

“The instruction at 0x00000023 referenced memory at 0x00000023. The memory could not be read.” (http://support.citrix.com/article/CTX134195)

I think this is also likely cause in this case. If possible I wanted to try and confirm from the dmp file.

note: I am normally looking at application usermode dmp files, not system crashes

However when I try to look at !memusage

I see

• 4,839 “Invalid PTE Frames” -> is there a way I can trace if this is related to the Citrix component? Could vmss2core being tarnishing the dmp file? (If anyone has tried this before)
• When I output !vm all/most of the processes running as user show 0Kb, is this normal in complete memory dump:

0: kd> !vm

*** Virtual Memory Usage ***
Physical Memory: 8388483 ( 33553932 Kb)
Page File: ??\D:\pagefile.sys
Current: 8388608 Kb Free Space: 8332780 Kb
Minimum: 8388608 Kb Maximum: 8388608 Kb
unable to get nt!MmSystemLockPagesCount
Available Pages: 6567814 ( 26271256 Kb)
ResAvail Pages: 7481208 ( 29924832 Kb)
Locked IO Pages: 0 ( 0 Kb)
Free System PTEs: 33426319 ( 133705276 Kb)
Modified Pages: 1946 ( 7784 Kb)
Modified PF Pages: 1946 ( 7784 Kb)
NonPagedPool Usage: 744304 ( 2977216 Kb)
NonPagedPool Max: 6271998 ( 25087992 Kb)
PagedPool 0 Usage: 94578 ( 378312 Kb)
PagedPool 1 Usage: 36103 ( 144412 Kb)
PagedPool 2 Usage: 30465 ( 121860 Kb)
PagedPool 3 Usage: 30294 ( 121176 Kb)
PagedPool 4 Usage: 30344 ( 121376 Kb)
PagedPool Usage: 221784 ( 887136 Kb)
PagedPool Maximum: 33554432 ( 134217728 Kb)
Session Commit: 21367 ( 85468 Kb)
Shared Commit: 18854 ( 75416 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 23626 ( 94504 Kb)
PagedPool Commit: 221844 ( 887376 Kb)
Driver Commit: 3776 ( 15104 Kb)
Committed pages: 1737468 ( 6949872 Kb)
Commit limit: 10485171 ( 41940684 Kb)

Total Private: 588242 ( 2352968 Kb)
0708 svchost.exe 114688 ( 458752 Kb)
166c SavService.exe 52789 ( 211156 Kb)
0a54 spoolsv.exe 33495 ( 133980 Kb)
0ee0 XTE.exe 16537 ( 66148 Kb)
1a04 CcmExec.exe 16215 ( 64860 Kb)
af7c SearchIndexer. 15360 ( 61440 Kb)
1b8c ImaSrv.exe 14416 ( 57664 Kb)
044c lsass.exe 13127 ( 52508 Kb)
0728 EmSystem.exe 11944 ( 47776 Kb)
1ccc WmiPrvSE.exe 11074 ( 44296 Kb)
72b8 PresentationFo 10406 ( 41624 Kb)
0d1c HCAService.exe 9709 ( 38836 Kb)
063c svchost.exe 9000 ( 36000 Kb)
16f4 SemsService.ex 8733 ( 34932 Kb)
252c aws_orb.exe 8520 ( 34080 Kb)
1f38 aws_agtgate.ex 8018 ( 32072 Kb)
03ec CmRcService.ex 8003 ( 32012 Kb)
0620 svchost.exe 7983 ( 31932 Kb)
0cc0 swi_service.ex 7971 ( 31884 Kb)
06dc svchost.exe 7904 ( 31616 Kb)
1610 AMAgent.exe 7711 ( 30844 Kb)
0558 svchost.exe 7641 ( 30564 Kb)
14e8 WmiPrvSE.exe 7435 ( 29740 Kb)
05c4 svchost.exe 7116 ( 28464 Kb)
1420 svchost.exe 6946 ( 27784 Kb)
1660 dllhost.exe 6593 ( 26372 Kb)
0ac0 PmAgent.exe 6539 ( 26156 Kb)
0bd8 caiWinA3.exe 5629 ( 22516 Kb)
123c MSOIDSVC.EXE 5560 ( 22240 Kb)
0ae4 AppVClient.exe 5205 ( 20820 Kb)
0d68 CpSvc.exe 4634 ( 18536 Kb)
043c services.exe 4605 ( 18420 Kb)
0cf4 CitrixCseEngin 4558 ( 18232 Kb)
2854 aws_sadmin.exe 4506 ( 18024 Kb)
2694 OSPPSVC.EXE 4341 ( 17364 Kb)
0754 svchost.exe 4075 ( 16300 Kb)
0454 lsm.exe 4030 ( 16120 Kb)
1874 sftlist.exe 3962 ( 15848 Kb)
16b0 vmtoolsd.exe 3956 ( 15824 Kb)
094c svchost.exe 3867 ( 15468 Kb)
0e10 CtxSvcHost.exe 3432 ( 13728 Kb)
14b0 ManagementAgen 3429 ( 13716 Kb)
1adc WmiPrvSE.exe 3281 ( 13124 Kb)
0920 svchost.exe 3241 ( 12964 Kb)
04c8 svchost.exe 3232 ( 12928 Kb)
05b4 LogonUI.exe 3220 ( 12880 Kb)
1c44 dllhost.exe 2869 ( 11476 Kb)
a564 svchost.exe 2802 ( 11208 Kb)
09b0 BNDevice.exe 2670 ( 10680 Kb)
2b04 AMAgentAssist. 2603 ( 10412 Kb)
0da0 CtxSvcHost.exe 2465 ( 9860 Kb)
20e4 WmiPrvSE.exe 2339 ( 9356 Kb)
1ca4 msdtc.exe 2315 ( 9260 Kb)
d7e4 WmiPrvSE.exe 2268 ( 9072 Kb)
12ec svchost.exe 2254 ( 9016 Kb)
184c RadeSvc.exe 2253 ( 9012 Kb)
1a14 caiLogA2.exe 2159 ( 8636 Kb)
0e9c CtxSvcHost.exe 2092 ( 8368 Kb)
13ac awservices.exe 2086 ( 8344 Kb)
1944 VMUpgradeHelpe 2042 ( 8168 Kb)
12b0 svchost.exe 2007 ( 8028 Kb)
0d48 XTE.exe 1950 ( 7800 Kb)
1330 UI0Detect.exe 1919 ( 7676 Kb)
0e58 CtxSvcHost.exe 1865 ( 7460 Kb)
1200 CtxSvcHost.exe 1855 ( 7420 Kb)
068c EmCoreService. 1802 ( 7208 Kb)
0f28 IMAAdvanceSrv. 1736 ( 6944 Kb)
11e8 mfcom.exe 1692 ( 6768 Kb)
45d0 svchost.exe 1666 ( 6664 Kb)
01cc wininit.exe 1648 ( 6592 Kb)
1290 MSOIDSVCM.EXE 1584 ( 6336 Kb)
4ce4 msiexec.exe 1550 ( 6200 Kb)
24a8 ALsvc.exe 1531 ( 6124 Kb)
0530 conhost.exe 1430 ( 5720 Kb)
11a0 LogWatNT.exe 1325 ( 5300 Kb)
18c0 SAVAdminServic 1283 ( 5132 Kb)
0cb4 encsvc.exe 1266 ( 5064 Kb)
01d8 winlogon.exe 1227 ( 4908 Kb)
12d0 RadeHlprSvc.ex 1180 ( 4720 Kb)
13bc swc_service.ex 1153 ( 4612 Kb)
11e0 lic98Service.e 1147 ( 4588 Kb)
0234 csrss.exe 1147 ( 4588 Kb)
0c88 CdfSvc.exe 1061 ( 4244 Kb)
148c sftvsa.exe 1006 ( 4024 Kb)
1264 ntmulti.exe 975 ( 3900 Kb)
1404 PmAgentAssist. 953 ( 3812 Kb)
0510 nslsvice.exe 940 ( 3760 Kb)
0528 nsl.exe 821 ( 3284 Kb)
03c8 csrss.exe 481 ( 1924 Kb)
03a4 smss.exe 158 ( 632 Kb)
0004 System 31 ( 124 Kb)
1000c iexplore.exe 0 ( 0 Kb)
fff8 iexplore.exe 0 ( 0 Kb)
ffd0 userinit.exe 0 ( 0 Kb)
ff48 sftlp.exe 0 ( 0 Kb)
fd50 iexplore.exe 0 ( 0 Kb)
fc84 prowc.exe 0 ( 0 Kb)
fbd8 iexplore.exe 0 ( 0 Kb)
fb8c rundll32.exe 0 ( 0 Kb)
f7d8 userinit.exe 0 ( 0 Kb)
f73c CPWSave.exe 0 ( 0 Kb)
f514 iexplore.exe 0 ( 0 Kb)
f494 iexplore.exe 0 ( 0 Kb)
f3e8 userinit.exe 0 ( 0 Kb)
f3dc userinit.exe 0 ( 0 Kb)
f3d4 CPWSave.exe 0 ( 0 Kb)
f3b0 AppVStreamingU 0 ( 0 Kb)
f394 CPWSave.exe 0 ( 0 Kb)
f34c CPWSave.exe 0 ( 0 Kb)
f320 iexplore.exe 0 ( 0 Kb)
f31c AppVStreamingU 0 ( 0 Kb)
f300 iexplore.exe 0 ( 0 Kb)
f2fc iexplore.exe 0 ( 0 Kb)
f2ec conhost.exe 0 ( 0 Kb)
f2c8 iexplore.exe 0 ( 0 Kb)
f274 winlogon.exe 0 ( 0 Kb)
f22c userinit.exe 0 ( 0 Kb)
f1e4 CPWSave.exe 0 ( 0 Kb)
f1d4 iexplore.exe 0 ( 0 Kb)
f0fc iexplore.exe 0 ( 0 Kb)
f0e8 wfshell.exe 0 ( 0 Kb)
f0ac wfshell.exe 0 ( 0 Kb)
f004 AppVStreamingU 0 ( 0 Kb)
efdc CPWSave.exe 0 ( 0 Kb)
efd4 AppVStreamingU 0 ( 0 Kb)
efcc userinit.exe 0 ( 0 Kb)
etc

When we run kv- I suspect this is related to when VMware is creating the suspended state?

0: kd> kv
Child-SP RetAddr : Args to Child : Call Site
fffff8000151aad0 fffff800016d1a93 : 0000000000000000 fffff800016483c0 fffff80000000301 fffffa8052b2fcd0 : hal!HalpRtcClockInterrupt+0x2a
fffff8000151ab00 fffff880036119c2 : fffff800016de109 0000000000369e99 fffffa80199f3ca8 fffff80001860cc0 : nt!KiInterruptDispatchNoLock+0x163 (TrapFrame @ fffff8000151ab00) fffff8000151ac98 fffff800016de109 : 0000000000369e99 fffffa80199f3ca8 fffff80001860cc0 0000000000000001 : intelppm!C1Halt+0x2 fffff8000151aca0 fffff800016cd21c : fffff80001852e80 fffff80000000000 0000000000000000 fffff88000eb8800 : nt!PoIdle+0x52a fffff8000151ad80 0000000000000000 : fffff8000151b000 fffff80001515000 fffff8000151ad40 00000000`00000000 : nt!KiIdleLoop+0x2c

Many of the user process have “no active threads”

PROCESS fffffa804f06b060
SessionId: 4 Cid: 2d04 Peb: 7fffffd8000 ParentCid: 28ac
DirBase: 392f36000 ObjectTable: 00000000 HandleCount: 0.
Image: explorer.exe
VadRoot 0000000000000000 Vads 0 Clone 0 Private 1. Modified 338397. Locked 0.
DeviceMap fffff8a00b3519d0
Token fffff8a01451b050
ElapsedTime 10 Days 22:01:10.232
UserTime 00:00:23.640
KernelTime 00:00:52.125
QuotaPoolUsage[PagedPool] 0
QuotaPoolUsage[NonPagedPool] 0
Working Set Sizes (now,min,max) (5, 50, 345) (20KB, 200KB, 1380KB)
PeakWorkingSetSize 15693
VirtualSize 442 Mb
PeakVirtualSize 485 Mb
PageFaultCount 502925
MemoryPriority BACKGROUND
BasePriority 10
CommitCharge 0

No active threads

0: kd> .process fffffa804f06b060
Implicit process is now fffffa80`4f06b060
0: kd> !peb
PEB at 000007fffffd8000
error 1 InitTypeRead( nt!_PEB at 000007fffffd8000)…

I notice PEB in 738 cases is this address

7efdf000

and in 238 cases is

7fffffdf000

Is this expected, or not in complete memory dmp?

[Scott_Noone_OSR]

• 4,839 “Invalid PTE Frames” -> is there a way I can trace if this is
  related to the Citrix component? Could vmss2core being tarnishing the dmp
  file? (If anyone has tried this before)

I’ve never used vmss2core to debug anything, so I can’t speak to any
specific peculiarities with those dumps. I wouldn’t be surprised though if
this was an artifact.

• When I output !vm all/most of the processes running as user show 0Kb, is
  this normal in complete memory dump:

These are terminated processes that someone still has a handle or reference
to. Doesn’t necessarily point to a problem, though if these never go away
you’re leaking memory. Having lots of these might be “normal” for Citrix
systems, impossible to say.

0: kd> .process fffffa804f06b060
Implicit process is now fffffa80`4f06b060
0: kd> !peb
PEB at 000007fffffd8000
error 1 InitTypeRead( nt!_PEB at 000007fffffd8000)…

The process is terminated, so the PEB is gone.

While it doesn’t matter in this case, if you want to look at user mode state
in a dump .process itself is not sufficient. You need “.process /r /P” to
make sure that user mode related addresses are translated correctly.

I notice PEB in 738 cases is this address

7efdf000

and in 238 cases is

7fffffdf000

Unrelated to anything. The first ones were probably 32-bit processes and the
others 64-bit.

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@windbg…

I was asked to look at issue with Citrix Server. Unfortunately it was not
configured to save memory dumps, or event logs (They were cleaned on reboot)
So I only have VMWare Snapshot (.vmsn) I converted it to a complete memory
.dmp using vmss2core

https://labs.vmware.com/flings/vmss2core

This is the first time I’ve attempted to analyze such a dump file.

The System Logs for 4 other Citrix servers using same “image” went back
further and identified these crashes.

WerFault - 87,808 crashes with error:

The instruction at 0x00000023 referenced memory at 0x00000023.

Which matches this description:

After a server runs for several days, attempts to launch a 32-bit
application can fail. The application exits unexpectedly and the following
error message appears:

“The instruction at 0x00000023 referenced memory at 0x00000023. The memory
could not be read.” (http://support.citrix.com/article/CTX134195)

I think this is also likely cause in this case. If possible I wanted to try
and confirm from the dmp file.

note: I am normally looking at application usermode dmp files, not system
crashes

However when I try to look at !memusage

I see

• 4,839 “Invalid PTE Frames” -> is there a way I can trace if this is
  related to the Citrix component? Could vmss2core being tarnishing the dmp
  file? (If anyone has tried this before)
• When I output !vm all/most of the processes running as user show 0Kb, is
  this normal in complete memory dump:

0: kd> !vm

*** Virtual Memory Usage ***
Physical Memory: 8388483 ( 33553932 Kb)
Page File: ??\D:\pagefile.sys
Current: 8388608 Kb Free Space: 8332780 Kb
Minimum: 8388608 Kb Maximum: 8388608 Kb
unable to get nt!MmSystemLockPagesCount
Available Pages: 6567814 ( 26271256 Kb)
ResAvail Pages: 7481208 ( 29924832 Kb)
Locked IO Pages: 0 ( 0 Kb)
Free System PTEs: 33426319 ( 133705276 Kb)
Modified Pages: 1946 ( 7784 Kb)
Modified PF Pages: 1946 ( 7784 Kb)
NonPagedPool Usage: 744304 ( 2977216 Kb)
NonPagedPool Max: 6271998 ( 25087992 Kb)
PagedPool 0 Usage: 94578 ( 378312 Kb)
PagedPool 1 Usage: 36103 ( 144412 Kb)
PagedPool 2 Usage: 30465 ( 121860 Kb)
PagedPool 3 Usage: 30294 ( 121176 Kb)
PagedPool 4 Usage: 30344 ( 121376 Kb)
PagedPool Usage: 221784 ( 887136 Kb)
PagedPool Maximum: 33554432 ( 134217728 Kb)
Session Commit: 21367 ( 85468 Kb)
Shared Commit: 18854 ( 75416 Kb)
Special Pool: 0 ( 0 Kb)
Shared Process: 23626 ( 94504 Kb)
PagedPool Commit: 221844 ( 887376 Kb)
Driver Commit: 3776 ( 15104 Kb)
Committed pages: 1737468 ( 6949872 Kb)
Commit limit: 10485171 ( 41940684 Kb)

Total Private: 588242 ( 2352968 Kb)
0708 svchost.exe 114688 ( 458752 Kb)
166c SavService.exe 52789 ( 211156 Kb)
0a54 spoolsv.exe 33495 ( 133980 Kb)
0ee0 XTE.exe 16537 ( 66148 Kb)
1a04 CcmExec.exe 16215 ( 64860 Kb)
af7c SearchIndexer. 15360 ( 61440 Kb)
1b8c ImaSrv.exe 14416 ( 57664 Kb)
044c lsass.exe 13127 ( 52508 Kb)
0728 EmSystem.exe 11944 ( 47776 Kb)
1ccc WmiPrvSE.exe 11074 ( 44296 Kb)
72b8 PresentationFo 10406 ( 41624 Kb)
0d1c HCAService.exe 9709 ( 38836 Kb)
063c svchost.exe 9000 ( 36000 Kb)
16f4 SemsService.ex 8733 ( 34932 Kb)
252c aws_orb.exe 8520 ( 34080 Kb)
1f38 aws_agtgate.ex 8018 ( 32072 Kb)
03ec CmRcService.ex 8003 ( 32012 Kb)
0620 svchost.exe 7983 ( 31932 Kb)
0cc0 swi_service.ex 7971 ( 31884 Kb)
06dc svchost.exe 7904 ( 31616 Kb)
1610 AMAgent.exe 7711 ( 30844 Kb)
0558 svchost.exe 7641 ( 30564 Kb)
14e8 WmiPrvSE.exe 7435 ( 29740 Kb)
05c4 svchost.exe 7116 ( 28464 Kb)
1420 svchost.exe 6946 ( 27784 Kb)
1660 dllhost.exe 6593 ( 26372 Kb)
0ac0 PmAgent.exe 6539 ( 26156 Kb)
0bd8 caiWinA3.exe 5629 ( 22516 Kb)
123c MSOIDSVC.EXE 5560 ( 22240 Kb)
0ae4 AppVClient.exe 5205 ( 20820 Kb)
0d68 CpSvc.exe 4634 ( 18536 Kb)
043c services.exe 4605 ( 18420 Kb)
0cf4 CitrixCseEngin 4558 ( 18232 Kb)
2854 aws_sadmin.exe 4506 ( 18024 Kb)
2694 OSPPSVC.EXE 4341 ( 17364 Kb)
0754 svchost.exe 4075 ( 16300 Kb)
0454 lsm.exe 4030 ( 16120 Kb)
1874 sftlist.exe 3962 ( 15848 Kb)
16b0 vmtoolsd.exe 3956 ( 15824 Kb)
094c svchost.exe 3867 ( 15468 Kb)
0e10 CtxSvcHost.exe 3432 ( 13728 Kb)
14b0 ManagementAgen 3429 ( 13716 Kb)
1adc WmiPrvSE.exe 3281 ( 13124 Kb)
0920 svchost.exe 3241 ( 12964 Kb)
04c8 svchost.exe 3232 ( 12928 Kb)
05b4 LogonUI.exe 3220 ( 12880 Kb)
1c44 dllhost.exe 2869 ( 11476 Kb)
a564 svchost.exe 2802 ( 11208 Kb)
09b0 BNDevice.exe 2670 ( 10680 Kb)
2b04 AMAgentAssist. 2603 ( 10412 Kb)
0da0 CtxSvcHost.exe 2465 ( 9860 Kb)
20e4 WmiPrvSE.exe 2339 ( 9356 Kb)
1ca4 msdtc.exe 2315 ( 9260 Kb)
d7e4 WmiPrvSE.exe 2268 ( 9072 Kb)
12ec svchost.exe 2254 ( 9016 Kb)
184c RadeSvc.exe 2253 ( 9012 Kb)
1a14 caiLogA2.exe 2159 ( 8636 Kb)
0e9c CtxSvcHost.exe 2092 ( 8368 Kb)
13ac awservices.exe 2086 ( 8344 Kb)
1944 VMUpgradeHelpe 2042 ( 8168 Kb)
12b0 svchost.exe 2007 ( 8028 Kb)
0d48 XTE.exe 1950 ( 7800 Kb)
1330 UI0Detect.exe 1919 ( 7676 Kb)
0e58 CtxSvcHost.exe 1865 ( 7460 Kb)
1200 CtxSvcHost.exe 1855 ( 7420 Kb)
068c EmCoreService. 1802 ( 7208 Kb)
0f28 IMAAdvanceSrv. 1736 ( 6944 Kb)
11e8 mfcom.exe 1692 ( 6768 Kb)
45d0 svchost.exe 1666 ( 6664 Kb)
01cc wininit.exe 1648 ( 6592 Kb)
1290 MSOIDSVCM.EXE 1584 ( 6336 Kb)
4ce4 msiexec.exe 1550 ( 6200 Kb)
24a8 ALsvc.exe 1531 ( 6124 Kb)
0530 conhost.exe 1430 ( 5720 Kb)
11a0 LogWatNT.exe 1325 ( 5300 Kb)
18c0 SAVAdminServic 1283 ( 5132 Kb)
0cb4 encsvc.exe 1266 ( 5064 Kb)
01d8 winlogon.exe 1227 ( 4908 Kb)
12d0 RadeHlprSvc.ex 1180 ( 4720 Kb)
13bc swc_service.ex 1153 ( 4612 Kb)
11e0 lic98Service.e 1147 ( 4588 Kb)
0234 csrss.exe 1147 ( 4588 Kb)
0c88 CdfSvc.exe 1061 ( 4244 Kb)
148c sftvsa.exe 1006 ( 4024 Kb)
1264 ntmulti.exe 975 ( 3900 Kb)
1404 PmAgentAssist. 953 ( 3812 Kb)
0510 nslsvice.exe 940 ( 3760 Kb)
0528 nsl.exe 821 ( 3284 Kb)
03c8 csrss.exe 481 ( 1924 Kb)
03a4 smss.exe 158 ( 632 Kb)
0004 System 31 ( 124 Kb)
1000c iexplore.exe 0 ( 0 Kb)
fff8 iexplore.exe 0 ( 0 Kb)
ffd0 userinit.exe 0 ( 0 Kb)
ff48 sftlp.exe 0 ( 0 Kb)
fd50 iexplore.exe 0 ( 0 Kb)
fc84 prowc.exe 0 ( 0 Kb)
fbd8 iexplore.exe 0 ( 0 Kb)
fb8c rundll32.exe 0 ( 0 Kb)
f7d8 userinit.exe 0 ( 0 Kb)
f73c CPWSave.exe 0 ( 0 Kb)
f514 iexplore.exe 0 ( 0 Kb)
f494 iexplore.exe 0 ( 0 Kb)
f3e8 userinit.exe 0 ( 0 Kb)
f3dc userinit.exe 0 ( 0 Kb)
f3d4 CPWSave.exe 0 ( 0 Kb)
f3b0 AppVStreamingU 0 ( 0 Kb)
f394 CPWSave.exe 0 ( 0 Kb)
f34c CPWSave.exe 0 ( 0 Kb)
f320 iexplore.exe 0 ( 0 Kb)
f31c AppVStreamingU 0 ( 0 Kb)
f300 iexplore.exe 0 ( 0 Kb)
f2fc iexplore.exe 0 ( 0 Kb)
f2ec conhost.exe 0 ( 0 Kb)
f2c8 iexplore.exe 0 ( 0 Kb)
f274 winlogon.exe 0 ( 0 Kb)
f22c userinit.exe 0 ( 0 Kb)
f1e4 CPWSave.exe 0 ( 0 Kb)
f1d4 iexplore.exe 0 ( 0 Kb)
f0fc iexplore.exe 0 ( 0 Kb)
f0e8 wfshell.exe 0 ( 0 Kb)
f0ac wfshell.exe 0 ( 0 Kb)
f004 AppVStreamingU 0 ( 0 Kb)
efdc CPWSave.exe 0 ( 0 Kb)
efd4 AppVStreamingU 0 ( 0 Kb)
efcc userinit.exe 0 ( 0 Kb)
etc

When we run kv- I suspect this is related to when VMware is creating the
suspended state?

0: kd> kv
Child-SP RetAddr : Args to Child
: Call Site
fffff8000151aad0 fffff800016d1a93 : 0000000000000000 fffff800016483c0
fffff80000000301 fffffa8052b2fcd0 : hal!HalpRtcClockInterrupt+0x2a
fffff8000151ab00 fffff880036119c2 : fffff800016de109 0000000000369e99
fffffa80199f3ca8 fffff80001860cc0 : nt!KiInterruptDispatchNoLock+0x163
(TrapFrame @ fffff8000151ab00) fffff8000151ac98 fffff800016de109 : 0000000000369e99 fffffa80199f3ca8 fffff80001860cc0 0000000000000001 : intelppm!C1Halt+0x2 fffff8000151aca0 fffff800016cd21c : fffff80001852e80 fffff80000000000 0000000000000000 fffff88000eb8800 : nt!PoIdle+0x52a fffff8000151ad80 0000000000000000 : fffff8000151b000 fffff80001515000 fffff8000151ad40 00000000`00000000 : nt!KiIdleLoop+0x2c

Many of the user process have “no active threads”

PROCESS fffffa804f06b060
SessionId: 4 Cid: 2d04 Peb: 7fffffd8000 ParentCid: 28ac
DirBase: 392f36000 ObjectTable: 00000000 HandleCount: 0.
Image: explorer.exe
VadRoot 0000000000000000 Vads 0 Clone 0 Private 1. Modified 338397.
Locked 0.
DeviceMap fffff8a00b3519d0
Token fffff8a01451b050
ElapsedTime 10 Days 22:01:10.232
UserTime 00:00:23.640
KernelTime 00:00:52.125
QuotaPoolUsage[PagedPool] 0
QuotaPoolUsage[NonPagedPool] 0
Working Set Sizes (now,min,max) (5, 50, 345) (20KB, 200KB, 1380KB)
PeakWorkingSetSize 15693
VirtualSize 442 Mb
PeakVirtualSize 485 Mb
PageFaultCount 502925
MemoryPriority BACKGROUND
BasePriority 10
CommitCharge 0

No active threads

0: kd> .process fffffa804f06b060
Implicit process is now fffffa80`4f06b060
0: kd> !peb
PEB at 000007fffffd8000
error 1 InitTypeRead( nt!_PEB at 000007fffffd8000)…

I notice PEB in 738 cases is this address

7efdf000

and in 238 cases is

7fffffdf000

Is this expected, or not in complete memory dmp?