Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Before Posting...
Please check out the Community Guidelines in the Announcements and Administration Category.

More Info on Driver Writing and Debugging


The free OSR Learning Library has more than 50 articles on a wide variety of topics about writing and debugging device drivers and Minifilters. From introductory level to advanced. All the articles have been recently reviewed and updated, and are written using the clear and definitive style you've come to expect from OSR over the years.


Check out The OSR Learning Library at: https://www.osr.com/osr-learning-library/


Identify driver holding on to zombie process handle

Caleb_ChamplinCaleb_Champlin Member - All Emails Posts: 3
Hi all, I'm fairly new to windbg, but I've got an issue where after a few days of work my windows 10 machine will usually have a few hundred thousand zombie processes on it and has to be rebooted to free the memory up.

I suspect that there is a kernel driver that's holding to handles, but I'm not sure how I would go about identifying which driver has the handles. Could someone help me with some steps I could take to try and track down the culprit?

Here is a quick example of a zombie process which is owned by "System"

lkd> !process 0 7 cgo.exe
PROCESS ffffb90d8255d580
SessionId: 1 Cid: 3f5c Peb: 00381000 ParentCid: 43e8
DirBase: 381102000 ObjectTable: 00000000 HandleCount: 0.
Image: cgo.exe
VadRoot 0000000000000000 Vads 0 Clone 0 Private 11. Modified 1. Locked 0.
DeviceMap ffff900d6d739d50
Token ffff900d77df5060
ElapsedTime 00:35:27.199
UserTime 00:00:00.015
KernelTime 00:00:00.015
QuotaPoolUsage[PagedPool] 0
QuotaPoolUsage[NonPagedPool] 0
Working Set Sizes (now,min,max) (12, 50, 345) (48KB, 200KB, 1380KB)
PeakWorkingSetSize 1290
VirtualSize 0 Mb
PeakVirtualSize 4177 Mb
PageFaultCount 1386
MemoryPriority BACKGROUND
BasePriority 8
CommitCharge 15

No active threads

!object ffffb90d8255d580
Object: ffffb90d8255d580 Type: (ffffb90d77a7a740) Process
ObjectHeader: ffffb90d8255d550 (new version)
HandleCount: 2 PointerCount: 65536

Comments

  • Alan_AdamsAlan_Adams Member - All Emails Posts: 6
    Look into the !obtrace functionality of WinDBG, in which you configure
    GFLAGS to track objects from a particular pool tag, to have the call
    stack captured whenever the objects are referenced or dereferenced.
    And then use !obtrace to view the call stacks captured for a specific
    object, and investigate which one seems to be a reference which was
    never released.

    Alan Adams
    Client for Open Enterprise Server
    Micro Focus
    [email protected]
  • Caleb_ChamplinCaleb_Champlin Member - All Emails Posts: 3
    Thanks for the information. Can you tell me how I determine which pool tag(s) to track?
  • Scott_Noone_(OSR)Scott_Noone_(OSR) Administrator Posts: 3,299
    Proc is the tag used for processes. However, I'd expect this to be VERY
    noisy. Often it's difficult to figure out what's a "good" reference and
    what's a bad reference...

    If you have a dump, you can do the following to try and find which process
    has the handle(s) to an offending process:

    !handle 0 3 0 Process

    If the handles are in the System process then handle tracing is enabled if
    you enable Driver Verifier. You can then dump the handle trace buffer with
    !htrace.

    -scott
    OSR
    @OSRDrivers

    -scott
    OSR

  • Caleb_ChamplinCaleb_Champlin Member - All Emails Posts: 3
    Thanks. That's helpful. I don't have a dump, will this still work for local debugging? The handles are definitely getting lost in the System process.

    I guess my follow up is...is this going to be so noisy/convoluted that it's not going to be worth my time to try and track down and I should just live with the problem? I don't necessarily want to go down a huge rabbit hole with this.
  • Scott_Noone_(OSR)Scott_Noone_(OSR) Administrator Posts: 3,299
    Yes, should work with local debugging.

    <QUOTE>
    The handles are definitely getting lost in the System process.
    </QUOTE>

    That means a kernel component (probably a driver) is leaking the handles.
    This should be tractable, the basic idea is:

    1. Enable Driver Verifier's "Miscellaneous Checks" option on any driver you
    like (really doesn't matter which, this just had the side effect of enabling
    handle tracing on the System process)

    2. Reproduce your situation where you have leaky handles

    3. Run !handle 0 3 0 Process and note a handle to the offending process

    4. Run !htrace <handle> <system process address>

    This should give you a call stack for the handle and (potentially) point to
    the offending driver.

    <QUOTE>
    I don't necessarily want to go down a huge rabbit hole with this.
    </QUOTE>

    Oh, this could absolutely be a huge, massive, never ending time suck :) If
    the above goes according to plan it shouldn't be too bad, but no plan
    survives contact with the enemy...

    But, in all seriousness, TRYING the above shouldn't take very long. If it
    doesn't immediately point to the bad driver then you can at least say you
    tried.

    -scott
    OSR
    @OSRDrivers

    -scott
    OSR

Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Upcoming OSR Seminars
OSR has suspended in-person seminars due to the Covid-19 outbreak. But, don't miss your training! Attend via the internet instead!
Kernel Debugging 30 Mar 2020 OSR Seminar Space
Developing Minifilters 15 Jun 2020 LIVE ONLINE
Writing WDF Drivers 22 June 2020 LIVE ONLINE
Internals & Software Drivers 28 Sept 2020 Dulles, VA