VMware server freeze

byty114

Hello,
I have a server that is freezing. No reaction to any kind of commands even on the console.
I made a snapshot of the VM with memory. I transform it in a dump.
What are useful commands (!locks......) to find out why it was feezed. I am assuming there is a lock.
Also how do I read the result of the commands?

Windows Server 2019 OS

Scott_Noone_(OSR)

For a livelock (i.e. CPUs are spinning at 100% and nothing else can get done):

!running -ti
!ready f

For a deadlock (i.e. threads are sleeping and nothing is getting done):

!locks - ERESOURCEs only
!mex.uniquestacks (Mex available here: https://www.microsoft.com/en-us/download/details.aspx?id=53304)
!stacks 2
!process 0 f

byty114

Thank you for the commands. I will try to investigate. the fact is that the CPU was not at 100%. It was just frozen, also the memory went low under 1GB when you were checking the VM. Basically, nothing was running any longer, just stanning still.

0: kd> !ready f
Processor 0: No threads in READY state
Processor 1: No threads in READY state
Processor 2: No threads in READY state
Processor 3: No threads in READY state
Processor 4: No threads in READY state
Processor 5: No threads in READY state
Processor 6: No threads in READY state
Processor 7: No threads in READY state

0: kd> !running -ti

System Processors: (00000000000000ff)
Idle Processors: (00000000000000eb)

   Prcbs             Current         (pri) Next            (pri) Idle

0 fffff802423c1180 fffff8024243d980 ( 0) fffff8024243d980 ................

# Child-SP RetAddr Call Site
00 fffff80244331840 fffff8024210839c nt!PpmIdleGuestExecute+0x15
01 fffff80244331880 fffff8024210752a nt!PpmIdleExecuteTransition+0xcbc
02 fffff80244331b00 fffff802421df67c nt!PoIdle+0x33a
03 fffff80244331c60 0000000000000000 nt!KiIdleLoop+0x2c

1 ffffe7806057c180 ffffe7806058d000 ( 0) ffffe7806058d000 ................

# Child-SP RetAddr Call Site
00 ffffe7805ffe2840 fffff8024210839c nt!PpmIdleGuestExecute+0x15
01 ffffe7805ffe2880 fffff8024210752a nt!PpmIdleExecuteTransition+0xcbc
02 ffffe7805ffe2b00 fffff802421df67c nt!PoIdle+0x33a
03 ffffe7805ffe2c60 0000000000000000 nt!KiIdleLoop+0x2c

2 ffffe78060240180 ffff91035c963080 (15) ffffe78060251000 ................

# Child-SP RetAddr Call Site
00 ffffe7806e324a50 ffffc069adc5b36d win32kbase!SetHandleFlag+0x26
01 ffffe7806e324a80 ffffc069adc5b2fc win32kfull+0x5b36d
02 ffffe7806e324ad0 fffff802421ecc03 win32kfull+0x5b2fc
03 ffffe7806e324b00 00007ffe44b220e4 nt!KiSystemServiceCopyEnd+0x13
04 000000d832f7f678 00007ffe2bd3c138 0x00007ffe44b220e4 05 000000d832f7f680 000000000000000e 0x00007ffe2bd3c138
06 000000d832f7f688 00007ffe2bd67b3c 0xe
07 000000d832f7f690 ffffffffffffffff 0x00007ffe2bd67b3c 08 000000d832f7f698 000000d832f7f828 0xffffffffffffffff
09 000000d832f7f6a0 000000d832f7f6f0 0x000000d832f7f828 0a 000000d832f7f6a8 00000000000004a8 0x000000d832f7f6f0
0b 000000d832f7f6b0 000000000002001a 0x4a8
0c 000000d832f7f6b8 000000000002001a 0x2001a
0d 000000d832f7f6c0 000000000000004a 0x2001a
0e 000000d832f7f6c8 00007ffe47ba15a5 0x4a
0f 000000d832f7f6d0 000001e430367a10 0x00007ffe47ba15a5 10 000000d832f7f6d8 000000d832f7f800 0x000001e430367a10
11 000000d832f7f6e0 0000000000010018 0x000000d832f7f800 12 000000d832f7f6e8 000001e431287510 0x10018 13 000000d832f7f6f0 000000d800000012 0x000001e431287510
14 000000d832f7f6f8 00007ffe00000012 0x000000d800000012 15 000000d832f7f700 0000000000000000 0x00007ffe00000012

3 ffffe780602ca180 ffffe780602db000 ( 0) ffffe780602db000 ................

# Child-SP RetAddr Call Site
00 ffffe780602ff840 fffff8024210839c nt!PpmIdleGuestExecute+0x15
01 ffffe780602ff880 fffff8024210752a nt!PpmIdleExecuteTransition+0xcbc
02 ffffe780602ffb00 fffff802421df67c nt!PoIdle+0x33a
03 ffffe780602ffc60 0000000000000000 nt!KiIdleLoop+0x2c

4 ffffe78060357180 ffff9103b29e85c0 ( 8) ffffe78060368000 ................

# Child-SP RetAddr Call Site
00 0000000000000000 0000000000000000 0x0

5 ffffe780603e5180 ffffe780603f6000 ( 0) ffffe780603f6000 ................

# Child-SP RetAddr Call Site
00 ffffe7806061b840 fffff8024210839c nt!PpmIdleGuestExecute+0x15
01 ffffe7806061b880 fffff8024210752a nt!PpmIdleExecuteTransition+0xcbc
02 ffffe7806061bb00 fffff802421df67c nt!PoIdle+0x33a
03 ffffe7806061bc60 0000000000000000 nt!KiIdleLoop+0x2c

6 ffffe78060680180 ffffe78060691000 ( 0) ffffe78060691000 ................

# Child-SP RetAddr Call Site
00 ffffe78060679840 fffff8024210839c nt!PpmIdleGuestExecute+0x15
01 ffffe78060679880 fffff8024210752a nt!PpmIdleExecuteTransition+0xcbc
02 ffffe78060679b00 fffff802421df67c nt!PoIdle+0x33a
03 ffffe78060679c60 0000000000000000 nt!KiIdleLoop+0x2c

7 ffffe78060701180 ffffe78060712000 ( 0) ffffe78060712000 ................

# Child-SP RetAddr Call Site
00 ffffe780607369e0 fffff802421df6aa nt!KiRetireDpcList+0x163
01 ffffe78060736c60 0000000000000000 nt!KiIdleLoop+0x5a

byty114

I think this is relevant, still trying to understand

kd> !running
Process PID Thread Id Pri Base Pri Next CPU CSwitches User Kernel State Time Reason
================== ==== ================ ==== === ======== ======== ========= ====== ================ ======= ========== ==============
Idle 0 fffff8024243d980 0 0 0 0 507176964 0 27d.15:51:08.766 Running 30m:32.718 WrCalloutStack
Idle 0 ffffe7806058d000 0 0 0 1 538894446 0 27d.14:38:25.344 Running 31m:47.187 WrCalloutStack
winlogon.exe 3840 ffff91035c963080 9530 15 15 2 257963 0 16ms Running 46ms DelayExecution
Idle 0 ffffe780602db000 0 0 0 3 473101167 0 27d.19:03:01.094 Running 27m:44.968 WrCalloutStack
msedgewebview2.exe 4b78 ffff9103b29e85c0 597c 8 8 4 35189 6s.234 1s.016 Running 390ms UserRequest
Idle 0 ffffe780603f6000 0 0 0 5 540638435 0 27d.17:08:24.500 Running 30m:36.703 WrCalloutStack
Idle 0 ffffe78060691000 0 0 0 6 463232415 0 27d.16:48:22.844 Running 30m:16.718 WrCalloutStack
Idle 0 ffffe78060712000 0 0 0 7 641617303 0 27d.06:59:53.141 Running 33m:45.000 WrCalloutStack

Tim_Roberts

One thing I don't think you know, and which is very difficult to tell with a Windows guest, is whether the VM is frozen, or the UI is frozen. Can you ping the network interface?

byty114

Good question. for this one sadly i do not remember. The above VM`s was getting stuck, and also some others, and they had kind of the same application on them Citrix in general when you logged on to the system mainly with RDP. If i get a new one i will get back, as also the Citrix team did some changes at VMware.