Very high LSWN count in vm listing and SRV threads waiting for a long time

ntfsd · January 6, 2007, 6:20pm

Hi All,

I am noticing a problem with the SRV in Windows 2003. The stack of some of the SRV threads are inlined below. I have few questions:

What is the # of the SRV threads in a system? Is the # configurable?
In this scenario the LSWN tag’s (worker item for SRV) is really high in the vm listing. However, there are atleast 4-5 SRV threads which are just waiting on a QueueObject instead of servicing the Worker Items. The wait count is very high. Also, other threads are waiting on IopParseDevice call. Any idea why?
How many Worker Item Queues are there in a 2003 system?
Is there a way to debug the QueueObject and the Queue?

THREAD 8958b0f8 Cid 0004.0328 Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable
88314624 NotificationEvent
IRP List:
88384008: (0006,01fc) Flags: 00000884 Mdl: 00000000
Impersonation token: e4b30120 (Level Impersonation)
Owning Process 89f9e648 Image: System
Wait Start TickCount 63409804 Ticks: 10781797 (1:22:47:45.578)
Context Switch Count 928546
UserTime 00:00:00.0000
KernelTime 00:06:09.0812
Start Address srv!WorkerThread (0xb8acc394)
Stack Init b8c95000 Current b8c94840 Base b8c95000 Limit b8c92000 Call 0
Priority 9 BasePriority 9 PriorityDecrement 0
ChildEBP RetAddr
b8c94858 8083e6a2 nt!KiSwapContext+0x26 (FPO: [Uses EBP] [0,0,4])
b8c94884 8083f164 nt!KiSwapThread+0x284 (FPO: [Non-Fpo])
b8c948cc 808f1a11 nt!KeWaitForSingleObject+0x346 (FPO: [Non-Fpo])
b8c949c8 8093a934 nt!IopParseDevice+0xa4e (FPO: [Non-Fpo])
b8c94a00 80936848 nt!IopParseFile+0x46 (FPO: [Non-Fpo])
b8c94a80 80936aa5 nt!ObpLookupObjectName+0x11f (FPO: [Non-Fpo])
b8c94ad4 80936f27 nt!ObOpenObjectByName+0xea (FPO: [Non-Fpo])
b8c94b50 80936ff8 nt!IopCreateFile+0x447 (FPO: [Non-Fpo])
b8c94bac b8ad1efd nt!IoCreateFile+0xa3 (FPO: [Non-Fpo])
b8c94c1c b8ad2286 srv!SrvIoCreateFile+0x36d (FPO: [Non-Fpo])
b8c94cec b8b0ece3 srv!SrvSmbQueryPathInformation+0x4b6 (FPO: [Non-Fpo])
b8c94d78 b8ab6bc6 srv!_NULL_IMPORT_DESCRIPTOR (srv+0x59ce3)
b8c94d84 b8acc451 srv!SrvProcessSmb+0xb7 (FPO: [0,0,0])
b8c94dac 8092ccff srv!WorkerThread+0x138 (FPO: [Non-Fpo])
b8c94ddc 80841a96 nt!PspSystemThreadStartup+0x2e (FPO: [Non-Fpo])
00000000 00000000 nt!KiThreadStartup+0x16

THREAD 89bc40f8 Cid 0004.0324 Teb: 00000000 Win32Thread: 00000000 WAIT: (Unknown) KernelMode Non-Alertable
899da6e4 QueueObject
Not impersonating
DeviceMap e1000758
Owning Process 89f9e648 Image: System
Wait Start TickCount 63397474 Ticks: 10794127 (1:22:50:58.234)
Context Switch Count 51115
UserTime 00:00:00.0000
KernelTime 00:00:05.0265
Start Address srv!WorkerThread (0xb8acc394)
Stack Init b8c8d000 Current b8c8ccec Base b8c8d000 Limit b8c8a000 Call 0
Priority 13 BasePriority 9 PriorityDecrement 0
ChildEBP RetAddr
b8c8cd04 8083e6a2 nt!KiSwapContext+0x26 (FPO: [Uses EBP] [0,0,4])
b8c8cd30 8083f2ac nt!KiSwapThread+0x284 (FPO: [Non-Fpo])
b8c8cd78 b8acc405 nt!KeRemoveQueue+0x417 (FPO: [Non-Fpo])
b8c8cdac 8092ccff srv!WorkerThread+0x7c (FPO: [Non-Fpo])
b8c8cddc 80841a96 nt!PspSystemThreadStartup+0x2e (FPO: [Non-Fpo])
00000000 00000000 nt!KiThreadStartup+0x16

OSR_Community_User · January 8, 2007, 5:48am

Sorry, but where is the problem? Any crashes or hungs? any malfunctions?

–
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

wrote in message news:xxxxx@ntfsd…
> Hi All,
>
> I am noticing a problem with the SRV in Windows 2003. The stack of some of
the SRV threads are inlined below. I have few questions:
>
> 1. What is the # of the SRV threads in a system? Is the # configurable?
>
> 2. In this scenario the LSWN tag’s (worker item for SRV) is really high in
the vm listing. However, there are atleast 4-5 SRV threads which are just
waiting on a QueueObject instead of servicing the Worker Items. The wait count
is very high. Also, other threads are waiting on IopParseDevice call. Any
idea why?
>
> 3. How many Worker Item Queues are there in a 2003 system?
>
> 4. Is there a way to debug the QueueObject and the Queue?
>
> THREAD 8958b0f8 Cid 0004.0328 Teb: 00000000 Win32Thread: 00000000 WAIT:
(Unknown) KernelMode Non-Alertable
> 88314624 NotificationEvent
> IRP List:
> 88384008: (0006,01fc) Flags: 00000884 Mdl: 00000000
> Impersonation token: e4b30120 (Level Impersonation)
> Owning Process 89f9e648 Image: System
> Wait Start TickCount 63409804 Ticks: 10781797 (1:22:47:45.578)
> Context Switch Count 928546
> UserTime 00:00:00.0000
> KernelTime 00:06:09.0812
> Start Address srv!WorkerThread (0xb8acc394)
> Stack Init b8c95000 Current b8c94840 Base b8c95000 Limit b8c92000 Call 0
> Priority 9 BasePriority 9 PriorityDecrement 0
> ChildEBP RetAddr
> b8c94858 8083e6a2 nt!KiSwapContext+0x26 (FPO: [Uses EBP] [0,0,4])
> b8c94884 8083f164 nt!KiSwapThread+0x284 (FPO: [Non-Fpo])
> b8c948cc 808f1a11 nt!KeWaitForSingleObject+0x346 (FPO: [Non-Fpo])
> b8c949c8 8093a934 nt!IopParseDevice+0xa4e (FPO: [Non-Fpo])
> b8c94a00 80936848 nt!IopParseFile+0x46 (FPO: [Non-Fpo])
> b8c94a80 80936aa5 nt!ObpLookupObjectName+0x11f (FPO: [Non-Fpo])
> b8c94ad4 80936f27 nt!ObOpenObjectByName+0xea (FPO: [Non-Fpo])
> b8c94b50 80936ff8 nt!IopCreateFile+0x447 (FPO: [Non-Fpo])
> b8c94bac b8ad1efd nt!IoCreateFile+0xa3 (FPO: [Non-Fpo])
> b8c94c1c b8ad2286 srv!SrvIoCreateFile+0x36d (FPO: [Non-Fpo])
> b8c94cec b8b0ece3 srv!SrvSmbQueryPathInformation+0x4b6 (FPO: [Non-Fpo])
> b8c94d78 b8ab6bc6 srv!_NULL_IMPORT_DESCRIPTOR (srv+0x59ce3)
> b8c94d84 b8acc451 srv!SrvProcessSmb+0xb7 (FPO: [0,0,0])
> b8c94dac 8092ccff srv!WorkerThread+0x138 (FPO: [Non-Fpo])
> b8c94ddc 80841a96 nt!PspSystemThreadStartup+0x2e (FPO: [Non-Fpo])
> 00000000 00000000 nt!KiThreadStartup+0x16
>
>
> THREAD 89bc40f8 Cid 0004.0324 Teb: 00000000 Win32Thread: 00000000 WAIT:
(Unknown) KernelMode Non-Alertable
> 899da6e4 QueueObject
> Not impersonating
> DeviceMap e1000758
> Owning Process 89f9e648 Image: System
> Wait Start TickCount 63397474 Ticks: 10794127 (1:22:50:58.234)
> Context Switch Count 51115
> UserTime 00:00:00.0000
> KernelTime 00:00:05.0265
> Start Address srv!WorkerThread (0xb8acc394)
> Stack Init b8c8d000 Current b8c8ccec Base b8c8d000 Limit b8c8a000 Call 0
> Priority 13 BasePriority 9 PriorityDecrement 0
> ChildEBP RetAddr
> b8c8cd04 8083e6a2 nt!KiSwapContext+0x26 (FPO: [Uses EBP] [0,0,4])
> b8c8cd30 8083f2ac nt!KiSwapThread+0x284 (FPO: [Non-Fpo])
> b8c8cd78 b8acc405 nt!KeRemoveQueue+0x417 (FPO: [Non-Fpo])
> b8c8cdac 8092ccff srv!WorkerThread+0x7c (FPO: [Non-Fpo])
> b8c8cddc 80841a96 nt!PspSystemThreadStartup+0x2e (FPO: [Non-Fpo])
> 00000000 00000000 nt!KiThreadStartup+0x16
>
>
>
>

ntfsd · January 8, 2007, 6:22pm

Hi Maxim,

You are right. The symptoms are not clear from the email. The LSWN (worker items) tag count in the non paged memory is very high and SRV is not able to process any more requests. Net view etc are all hanging. I am noticing less # of SRV threads in the system process listing than usual and some of them are waiting on a queue object instead of processing the worker items.

THREAD 89bc40f8 Cid 0004.0324 Teb: 00000000 Win32Thread: 00000000 WAIT:
(Unknown) KernelMode Non-Alertable
899da6e4 QueueObject
Not impersonating
DeviceMap e1000758
Owning Process 89f9e648 Image: System
Wait Start TickCount 63397474 Ticks: 10794127 (1:22:50:58.234)
Context Switch Count 51115
UserTime 00:00:00.0000
KernelTime 00:00:05.0265
Start Address srv!WorkerThread (0xb8acc394)
Stack Init b8c8d000 Current b8c8ccec Base b8c8d000 Limit b8c8a000 Call 0
Priority 13 BasePriority 9 PriorityDecrement 0
ChildEBP RetAddr
b8c8cd04 8083e6a2 nt!KiSwapContext+0x26 (FPO: [Uses EBP] [0,0,4])
b8c8cd30 8083f2ac nt!KiSwapThread+0x284 (FPO: [Non-Fpo])
b8c8cd78 b8acc405 nt!KeRemoveQueue+0x417 (FPO: [Non-Fpo])
b8c8cdac 8092ccff srv!WorkerThread+0x7c (FPO: [Non-Fpo])
b8c8cddc 80841a96 nt!PspSystemThreadStartup+0x2e (FPO: [Non-Fpo])
00000000 00000000 nt!KiThreadStartup+0x16

Our driver is also on some of the SRV threads’ stack. Are there any issues to keep in mind for the same? Has anyone had the need to do some processing in the SRV thread context? How was the processing achieved so as to minimize holding the SRV thread?

David_Beaver · January 8, 2007, 10:17pm

We have seen this sort of thing in the past, typically when something is keeping the server IO (to the disk) from completing. That causes the server to congest upstream, giving you symptoms similar to what you’re seeing. Typically only happens under a fairly high load.

Does your filter do anything that might delay completion of IO by the server?

…dave

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@yahoo.com
Sent: Monday, January 08, 2007 3:22 PM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] Very high LSWN count in vm listing and SRV threads waiting for a long time

Hi Maxim,

You are right. The symptoms are not clear from the email. The LSWN (worker items) tag count in the non paged memory is very high and SRV is not able to process any more requests. Net view etc are all hanging. I am noticing less # of SRV threads in the system process listing than usual and some of them are waiting on a queue object instead of processing the worker items.

THREAD 89bc40f8 Cid 0004.0324 Teb: 00000000 Win32Thread: 00000000 WAIT:
(Unknown) KernelMode Non-Alertable
899da6e4 QueueObject
Not impersonating
DeviceMap e1000758
Owning Process 89f9e648 Image: System
Wait Start TickCount 63397474 Ticks: 10794127 (1:22:50:58.234)
Context Switch Count 51115
UserTime 00:00:00.0000
KernelTime 00:00:05.0265
Start Address srv!WorkerThread (0xb8acc394)
Stack Init b8c8d000 Current b8c8ccec Base b8c8d000 Limit b8c8a000 Call 0
Priority 13 BasePriority 9 PriorityDecrement 0
ChildEBP RetAddr
b8c8cd04 8083e6a2 nt!KiSwapContext+0x26 (FPO: [Uses EBP] [0,0,4])
b8c8cd30 8083f2ac nt!KiSwapThread+0x284 (FPO: [Non-Fpo])
b8c8cd78 b8acc405 nt!KeRemoveQueue+0x417 (FPO: [Non-Fpo])
b8c8cdac 8092ccff srv!WorkerThread+0x7c (FPO: [Non-Fpo])
b8c8cddc 80841a96 nt!PspSystemThreadStartup+0x2e (FPO: [Non-Fpo])
00000000 00000000 nt!KiThreadStartup+0x16

Our driver is also on some of the SRV threads’ stack. Are there any issues to keep in mind for the same? Has anyone had the need to do some processing in the SRV thread context? How was the processing achieved so as to minimize holding the SRV thread?

Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@exmsft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com