System hang, many thread stalled in CcCanIWrite

Hello,

I have a computer, where many applications are freezing. I asked the user to take a full memory dump.

Inside the dump, I can see that many threads are stalled in CcCanIWrite. I ran !defwrites, here is the ouput :

0: kd> !defwrites
*** Cache Write Throttle Analysis ***

GetUlongFromAddress: unable to read from 0000000000000000
GetUlongFromAddress: unable to read from 0000000000000000
CcTotalDirtyPages:                     0 (       0 Kb)
CcDirtyPageThreshold:                  0 (       0 Kb)
AvailablePages:                   583532 ( 2334128 Kb)
ThrottleTop:                         450 (    1800 Kb)
ThrottleBottom:                       80 (     320 Kb)
ModifiedPages:                     78315 (  313260 Kb)

CcTotalDirtyPages >= CcDirtyPageThreshold, writes throttled

Check these thread(s): CcWriteBehind(LazyWriter)
Check critical workqueue for the lazy writer, !exqueue 16
Unable to get address of CcDeferredWrites

It says that CcDirtyPageThreshold is 0.

I’m wondering, if this information given by windbg is reliable ? Os does

GetUlongFromAddress: unable to read from 0000000000000000

shows that this value could not be read ?

If so, is it normal that it is 0 ? Why would it be 0 ?

Could this be the reason of all those stalled threads ?

Thank you

!defwrites and !exqueue have been broken for a long time due to both public PDB issues and also internal changed to the Cc.

You need to look at the Cc lazy writer threads and figure out why they’re stuck. They’ll all be in the system process:

!process 0 1F System

!vm command tells me that there are 78315 modified pages (+/- 300Mo)

Inside the Registry process there is a thread that tries to Write \Windows\System32\Config\SYSTEM.LOG2, which is according to https://learn.microsoft.com/en-us/windows/win32/sysinfo/registry-hives a transaction log of changes in registry hives.

    THREAD ffff8c0552609080  Cid 00a0.02d0  Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
        ffff8c0566c20160  SynchronizationEvent
    IRP List:
        ffff8c0567284010: (0006,05e0) Flags: 00060a00  Mdl: ffff8c056610d1c0
    Not impersonating
    Owning Process            ffff8c05469e9040       Image:         Registry
    Attached Process          N/A            Image:         N/A
    Wait Start TickCount      90003          Ticks: 3014 (0:00:00:47.093)
    Context Switch Count      1123           IdealProcessor: 2             
    UserTime                  00:00:00.000
    KernelTime                00:00:00.125
    Win32 Start Address nt!CmpLazyWriteWorker (0xfffff805f3a68210)
    Stack Init ffff8600b7f5fb70 Current ffff8600b7f5f2c0
    Base ffff8600b7f60000 Limit ffff8600b7f59000 Call 0000000000000000
    Priority 8 BasePriority 8 PriorityDecrement 0 IoPriority 2 PagePriority 5
    Child-SP          RetAddr               : Args to Child                                                           : Call Site
    ffff8600`b7f5f300 fffff805`f36a7064     : 00000000`00000047 00000000`00000000 00000000`00000000 ffff9e80`2456d2a8 : nt!KiSwapContext+0x76
    ffff8600`b7f5f440 fffff805`f3740fbd     : 00000000`00000000 00000000`00000000 ffff8600`b7f5f5d0 00000000`00008000 : nt!KiSwapThread+0x6d4
    ffff8600`b7f5f4d0 fffff805`f373f1b9     : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : nt!KiCommitThreadWait+0x39d
    ffff8600`b7f5f560 fffff805`f373d7c1     : ffff8c05`66c20160 ffff8c05`00000000 ffff8c05`68887000 ffff8600`b7f5f700 : nt!KeWaitForSingleObject+0x859
    ffff8600`b7f5f640 fffff805`f3e4fbed     : 00000000`00000057 00000000`00000000 ffffbe82`7a2dc000 00000000`00000000 : nt!KeWaitForMultipleObjects+0xa1
    ffff8600`b7f5f740 fffff805`f3e4f8e4     : ffffffff`80000428 7fffffff`00000000 ffffbe82`6c2dc000 00000000`00000001 : nt!CmpDoFileWrite+0x2fd
    ffff8600`b7f5f7f0 fffff805`f3d80546     : 00000000`00000000 fffff805`f3459ee5 00000002`00000037 fffff805`f34546eb : nt!CmpFileWrite+0x34
    ffff8600`b7f5f830 fffff805`f3d7d772     : 00000000`00000001 ffff8600`b7f5f980 00000000`00000000 ffffbe82`6c2dc000 : nt!HvWriteLogFile+0x12a
    ffff8600`b7f5f880 fffff805`f3d0afb5     : ffffbe82`6c2d8000 ffffbe82`6c2dc000 fffff805`f4203700 ffffffff`ffffffff : nt!CmpFlushHive+0x4be
    ffff8600`b7f5fa40 fffff805`f3a68298     : fffff805`f42036f0 00000000`00000080 fffff805`f4203788 fffff805`f4203600 : nt!CmpDoFlushNextHive+0x155
    ffff8600`b7f5fa80 fffff805`f38870ba     : ffff8c05`52609080 fffff805`f3a68200 00000000`18750790 024fa46f`b19bbfff : nt!CmpLazyWriteWorker+0x88
    ffff8600`b7f5faf0 fffff805`f3aabdf4     : ffff9e80`24563180 ffff8c05`52609080 fffff805`f3887060 7410c2f6`6c518b21 : nt!PspSystemThreadStartup+0x5a
    ffff8600`b7f5fb40 00000000`00000000     : ffff8600`b7f60000 ffff8600`b7f59000 00000000`00000000 00000000`00000000 : nt!KiStartSystemThread+0x34
0: kd> !irp ffff8c0567284010
Irp is active with 16 stacks 15 is current (= 0xffff8c05672844d0)
Mdl=ffff8c056610d1c0: No System Buffer: Thread ffff8c0552609080:  Irp stack trace.
cmd  flg cl Device   File     Completion-Context

[... cut part ...]

[IRP_MJ_WRITE(4), N/A(0)]
0 e1 ffff8c0548da2030 ffff8c0552ff8cf0 fffff80585133070-ffff8c055eb3a8a0 Success Error Cancel pending
\FileSystem\Ntfs	FLTMGR!FltpPassThroughCompletion
Args: 00008000 00000000 00009000 00000000
[IRP_MJ_WRITE(4), N/A(0)]
0  1 ffff8c0548c56cb0 ffff8c0552ff8cf0 00000000-00000000    pending
\FileSystem\FltMgr
Args: 00008000 00000000 00009000 00000000

0: kd> dt nt!_io_stack_location 0xffff8c05672844d0
+0x000 MajorFunction    : 0x4 ''
+0x001 MinorFunction    : 0 ''
+0x002 Flags            : 0 ''
+0x003 Control          : 0xe1 ''
+0x008 Parameters       :
+0x028 DeviceObject     : 0xffff8c0548da2030 _DEVICE_OBJECT    +0x030 FileObject       : 0xffff8c0552ff8cf0 _FILE_OBJECT
+0x038 CompletionRoutine : 0xfffff80585133070     long  FLTMGR!FltpPassThroughCompletion+0    +0x040 Context          : 0xffff8c055eb3a8a0 Void
0: kd> dx -id 0,0,ffff8c056474b080 -r1 ((ntkrnlmp!_FILE_OBJECT *)0xffff8c0552ff8cf0)
((ntkrnlmp!_FILE_OBJECT *)0xffff8c0552ff8cf0)                 : 0xffff8c0552ff8cf0 : "\Windows\System32\config\SYSTEM.LOG2" - Device for "\FileSystem\Ntfs" [Type: _FILE_OBJECT *]
[]     [Type: _FILE_OBJECT]
RelatedFile      : 0x0 [Type: _FILE_OBJECT *]
Device           : 0xffff8c0548d81a40 : Device for "\Driver\volmgr" FileSystem:"\FileSystem\Ntfs" [Type: _DEVICE_OBJECT *]

I think that the cache is overloaded in data to flush and its too slow. This is why all my writes are stalled. But how can I know why this write is taking so long ?

As I said in my previous post:

Sorry for the late response Scott.

Lazy Writer isn’t in the system process, I guess for now I will abandon this dump.

Thank you for your answers.

There’s NO threads in the Cc in the system process? Seems unlikely, but then again it’s a big strange world out there…