Device cleanup not called after surprise removal

Stephen_Glow · May 18, 2007, 10:27pm

I’m working on a KMDF function driver for a PCI card. As part of my testing I am using a cardbus -> PCI bridge to test the driver’s surprise removal handling.

What I’m seeing is that everything works nicely when the device is properly removed, but when a surprise removal is sent the device’s registered cleanup function is never called.

I’ve reduced my KMDF drive to an absolute shell that does nothing, and this still seems to be the case.

In my current minimal driver I do only the following:

In DriverEntry I create the driver and register an unload callback
In AddDevice I create a device object and register a cleanup callback

Nothing else, no queues, no pnp handlers, nothing.

The cleanup and unload callbacks don’t do anything but print debug messages.

On a normal remove, both cleanup then unload are called. On a surprise remove, neither one is called.

Any idea what I could be doing wrong here?

Thanks,
Steve

Doron_Holan · May 18, 2007, 11:59pm

Code please. Also, if the device has an open handle against it, you
will not get a cleanup on the device until the handle is closed (e.g.
until the device receives a pnp remove)

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@embeddedintelligence.com
Sent: Friday, May 18, 2007 7:29 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Device cleanup not called after surprise removal

I’m working on a KMDF function driver for a PCI card. As part of my
testing I am using a cardbus -> PCI bridge to test the driver’s surprise
removal handling.

What I’m seeing is that everything works nicely when the device is
properly removed, but when a surprise removal is sent the device’s
registered cleanup function is never called.

I’ve reduced my KMDF drive to an absolute shell that does nothing, and
this still seems to be the case.

In my current minimal driver I do only the following:

In DriverEntry I create the driver and register an unload callback
In AddDevice I create a device object and register a cleanup callback

Nothing else, no queues, no pnp handlers, nothing.

The cleanup and unload callbacks don’t do anything but print debug
messages.

On a normal remove, both cleanup then unload are called. On a surprise
remove, neither one is called.

Any idea what I could be doing wrong here?

Thanks,
Steve

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Stephen_Glow · May 19, 2007, 8:45am

Hi Doron;

Thanks for the quick reply.

I've reduced my driver to a single source file, included below. Still seeing exactly the same behavior.

My test system is a laptop running XP home edition. I have a cardbus->PCI bus extender which I'm using to test the surprise removal. See www.magma.com

I restart the system and insert the bus extender. The DriverEntry and AddDevice calls are successful. If I then stop the device using the 'safely remove hardware' tool, then the cleanup and driver unload callbacks are made.

If instead of stopping the hardware safely I just pull the PCI bus extender out of the cardbus slot then neither cleanup nor unload are called.

Before I cut the driver down to the bones for testing I had callback functions for some PNP events. When I had the callbacks I would see the following with a surprise removal:

Surprise removal
Disable Ints
D0 Exit
Release hardware

I would never see the device cleanup or driver unload called with a surprise removal. I would get these with a normal shutdown.

At no time have I accessed the driver / hardware from any application, so I don't know what would have an open handle to it.

Thanks,
Steve

Complete Driver code:

#include <wdm.h>
#include <wdf.h>

static void CleanupDevice( WDFOBJECT dev )
{
KdPrint(( "Cleanup called for device 0x%08x\n", (int)dev ));
}

static void DriverUnload( WDFDRIVER driver )
{
KdPrint(( "Driver unload called for: 0x%08x\n", (int)driver ));
}

static NTSTATUS AddDevice( WDFDRIVER driver, PWDFDEVICE_INIT DeviceInit )
{
WDFDEVICE Device;
WDF_OBJECT_ATTRIBUTES attrib;
NTSTATUS status = 0;

WDF_OBJECT_ATTRIBUTES_INIT( &attrib );
attrib.EvtCleanupCallback = CleanupDevice;

status = WdfDeviceCreate( &DeviceInit, &attrib, &Device );
KdPrint(( "create device returned: 0x%08x\n", status ));
return status;
}

NTSTATUS DriverEntry( PDRIVER_OBJECT DriverObject, PUNICODE_STRING regPath )
{
NTSTATUS status;
WDF_DRIVER_CONFIG config;
WDF_OBJECT_ATTRIBUTES attrib;
WDFDRIVER driver;

WDF_OBJECT_ATTRIBUTES_INIT( &attrib );
WDF_DRIVER_CONFIG_INIT( &config, AddDevice );
config.EvtDriverUnload = DriverUnload;

status = WdfDriverCreate( DriverObject, regPath, &attrib, &config, &driver );
KdPrint(( "Finished with DriverEntry. Status:0x%08x\n", (int)status ));
return status;
}
----------------------------------------------------------
end of code</wdf.h></wdm.h>

Doron_Holan · May 19, 2007, 3:32pm

I think there is a hw problem that is stopping the pnp remove. are there
any pci devices plugged into the extender when you surprise remove the
cardbus card? Or more succinctly, is there a device hierarchy behind
the card that also gets enumerated when the card is plugged in? If so,
any of these child devices can be stuck in the surprise remove state
which would mean that the parent is also in this state. A parent only
goes to the removesd state when all of its children are in the removed
state as well. Make sure you have correct symbols and send the output
of

!wdfkd.wdfdevice
!devstack from !wdfdevice)
!devnode
D

(PS you should use %p to print pointers, no need for the (int) casting
either)

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@embeddedintelligence.com
Sent: Saturday, May 19, 2007 5:47 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Device cleanup not called after surprise removal

Hi Doron;

Thanks for the quick reply.

I’ve reduced my driver to a single source file, included below. Still
seeing exactly the same behavior.

My test system is a laptop running XP home edition. I have a
cardbus->PCI bus extender which I’m using to test the surprise removal.
See www.magma.com

I restart the system and insert the bus extender. The DriverEntry and
AddDevice calls are successful. If I then stop the device using the
‘safely remove hardware’ tool, then the cleanup and driver unload
callbacks are made.

If instead of stopping the hardware safely I just pull the PCI bus
extender out of the cardbus slot then neither cleanup nor unload are
called.

Before I cut the driver down to the bones for testing I had callback
functions for some PNP events. When I had the callbacks I would see the
following with a surprise removal:
- Surprise removal
- Disable Ints
- D0 Exit
- Release hardware

I would never see the device cleanup or driver unload called with a
surprise removal. I would get these with a normal shutdown.

At no time have I accessed the driver / hardware from any application,
so I don’t know what would have an open handle to it.

Thanks,
Steve

Complete Driver code:
-------------------------------------------------

#include <wdm.h>
#include <wdf.h>

static void CleanupDevice( WDFOBJECT dev )
{
KdPrint(( “Cleanup called for device 0x%08x\n”, (int)dev ));
}

static void DriverUnload( WDFDRIVER driver )
{
KdPrint(( “Driver unload called for: 0x%08x\n”, (int)driver ));
}

static NTSTATUS AddDevice( WDFDRIVER driver, PWDFDEVICE_INIT DeviceInit
)
{
WDFDEVICE Device;
WDF_OBJECT_ATTRIBUTES attrib;
NTSTATUS status = 0;

WDF_OBJECT_ATTRIBUTES_INIT( &attrib );
attrib.EvtCleanupCallback = CleanupDevice;

status = WdfDeviceCreate( &DeviceInit, &attrib, &Device );
KdPrint(( “create device returned: 0x%08x\n”, status ));
return status;
}

NTSTATUS DriverEntry( PDRIVER_OBJECT DriverObject, PUNICODE_STRING
regPath )
{
NTSTATUS status;
WDF_DRIVER_CONFIG config;
WDF_OBJECT_ATTRIBUTES attrib;
WDFDRIVER driver;

WDF_OBJECT_ATTRIBUTES_INIT( &attrib );
WDF_DRIVER_CONFIG_INIT( &config, AddDevice );
config.EvtDriverUnload = DriverUnload;

status = WdfDriverCreate( DriverObject, regPath, &attrib, &config,
&driver );
KdPrint(( “Finished with DriverEntry. Status:0x%08x\n”, (int)status
));
return status;
}
----------------------------------------------------------
end of code

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer</wdf.h></wdm.h>

Stephen_Glow · May 23, 2007, 10:47am

Hi Doron;

The device that I'm developing the driver for is a PCI card, not the bus extender itself. I have my card plugged into the bus extender when I surprise remove it (the extender) from the system. This sends the surprise remove to my card.

After I've inserted the bus extender and my card has been recognized, the following info is available from the debugger:

lkd> !wdfkd.wdfdevice 0x7b5daa60

Dumping WDFDEVICE 0x7b5daa60

WDM PDEVICE_OBJECTs:self 84a44b98, attached 84a87a60, pdo 85552030

Pnp state: 119 ( WdfDevStatePnpStarted )
Power state: 307 ( WdfDevStatePowerD0 )
Power Pol state: 565 ( WdfDevStatePwrPolStarted )

Default WDFIOTARGET: 7aa81ed8

No pended pnp or power irps
Device is the power policy owner for the stack

lkd> !devstack 84a44b98
!DevObj !DrvObj !DevExt ObjectName

84a44b98 \Driver\copleycan 84a25760
84a87a60 \Driver\mrtcb 84a87b18
85552030 \Driver\PCI 855520e8 NTPNP_PCI0027
!DevNode 84a84800 :
DeviceInst is "PCI\VEN_10EE&DEV_D154&SUBSYS_D15410EE&REV_00\6&2b8c85f2&0&20200050"
ServiceName is "copleycan"

lkd> !devnode 84a84800
DevNode 0x84a84800 for PDO 0x85552030
Parent 0x84e803f0 Sibling 0000000000 Child 0000000000
InstancePath is "PCI\VEN_10EE&DEV_D154&SUBSYS_D15410EE&REV_00\6&2b8c85f2&0&20200050"
ServiceName is "copleycan"
State = DeviceNodeStarted (0x308)
Previous State = DeviceNodeEnumerateCompletion (0x30d)
StateHistory[07] = DeviceNodeEnumerateCompletion (0x30d)
StateHistory[06] = DeviceNodeStarted (0x308)
StateHistory[05] = DeviceNodeStartPostWork (0x307)
StateHistory[04] = DeviceNodeStartCompletion (0x306)
StateHistory[03] = DeviceNodeResourcesAssigned (0x304)
StateHistory[02] = DeviceNodeDriversAdded (0x303)
StateHistory[01] = DeviceNodeInitialized (0x302)
StateHistory[00] = DeviceNodeUninitialized (0x301)
StateHistory[19] = Unknown State (0x0)
StateHistory[18] = Unknown State (0x0)
StateHistory[17] = Unknown State (0x0)
StateHistory[16] = Unknown State (0x0)
StateHistory[15] = Unknown State (0x0)
StateHistory[14] = Unknown State (0x0)
StateHistory[13] = Unknown State (0x0)
StateHistory[12] = Unknown State (0x0)
StateHistory[11] = Unknown State (0x0)
StateHistory[10] = Unknown State (0x0)
StateHistory[09] = Unknown State (0x0)
StateHistory[08] = Unknown State (0x0)
Flags (0x00000030) DNF_ENUMERATED, DNF_IDS_QUERIED

After surprise removal of the bus extender:

lkd> !wdfkd.wdfdevice 0x7b5daa60

Dumping WDFDEVICE 0x7b5daa60

WDM PDEVICE_OBJECTs:self 84a44b98, attached 84a87a60, pdo 85552030

Pnp state: 12e ( WdfDevStatePnpFailedWaitForRemove )
Power state: 315 ( WdfDevStatePowerStopped )
Power Pol state: 563 ( WdfDevStatePwrPolStopped )

Default WDFIOTARGET: 7aa81ed8

No pended pnp or power irps
Device is the power policy owner for the stack

lkd> !devstack 84a44b98
!DevObj !DrvObj !DevExt ObjectName

84a44b98 \Driver\copleycan 84a25760
84a87a60 \Driver\mrtcb 84a87b18

lkd> !devnode 84a84800
DevNode 0x84a84800 for PDO 0x85552030
Parent 0000000000 Sibling 0000000000 Child 0000000000
InstancePath is "PCI\VEN_10EE&DEV_D154&SUBSYS_D15410EE&REV_00\6&2b8c85f2&0&20200050"
ServiceName is "copleycan"
State = DeviceNodeDeleted (0x314)
Previous State = DeviceNodeDeletePendingCloses (0x313)
StateHistory[12] = DeviceNodeDeletePendingCloses (0x313)
StateHistory[11] = DeviceNodeRemovePendingCloses (0x311)
StateHistory[10] = DeviceNodeStarted (0x308)
StateHistory[09] = DeviceNodeAwaitingQueuedDeletion (0x30e)
StateHistory[08] = DeviceNodeStarted (0x308)
StateHistory[07] = DeviceNodeEnumerateCompletion (0x30d)
StateHistory[06] = DeviceNodeStarted (0x308)
StateHistory[05] = DeviceNodeStartPostWork (0x307)
StateHistory[04] = DeviceNodeStartCompletion (0x306)
StateHistory[03] = DeviceNodeResourcesAssigned (0x304)
StateHistory[02] = DeviceNodeDriversAdded (0x303)
StateHistory[01] = DeviceNodeInitialized (0x302)
StateHistory[00] = DeviceNodeUninitialized (0x301)
StateHistory[19] = Unknown State (0x0)
StateHistory[18] = Unknown State (0x0)
StateHistory[17] = Unknown State (0x0)
StateHistory[16] = Unknown State (0x0)
StateHistory[15] = Unknown State (0x0)
StateHistory[14] = Unknown State (0x0)
StateHistory[13] = Unknown State (0x0)
Flags (0x00002020) DNF_IDS_QUERIED, DNF_HAS_PROBLEM
Problem = CM_PROB_DEVICE_NOT_THERE

I'm trying to make sense of this info, but am light on experience. Any thoughts?

Thanks,
Steve

Doron_Holan · May 23, 2007, 12:40pm

Whose driver is mrtcb?

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@embeddedintelligence.com
Sent: Wednesday, May 23, 2007 7:48 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Device cleanup not called after surprise removal

Hi Doron;

The device that I'm developing the driver for is a PCI card, not the bus
extender itself. I have my card plugged into the bus extender when I
surprise remove it (the extender) from the system. This sends the
surprise remove to my card.

After I've inserted the bus extender and my card has been recognized,
the following info is available from the debugger:

lkd> !wdfkd.wdfdevice 0x7b5daa60

Dumping WDFDEVICE 0x7b5daa60

WDM PDEVICE_OBJECTs:self 84a44b98, attached 84a87a60, pdo 85552030

Pnp state: 119 ( WdfDevStatePnpStarted )
Power state: 307 ( WdfDevStatePowerD0 )
Power Pol state: 565 ( WdfDevStatePwrPolStarted )

Default WDFIOTARGET: 7aa81ed8

No pended pnp or power irps
Device is the power policy owner for the stack

lkd> !devstack 84a44b98
!DevObj !DrvObj !DevExt ObjectName

84a44b98 \Driver\copleycan 84a25760
84a87a60 \Driver\mrtcb 84a87b18
85552030 \Driver\PCI 855520e8 NTPNP_PCI0027
!DevNode 84a84800 :
DeviceInst is
"PCI\VEN_10EE&DEV_D154&SUBSYS_D15410EE&REV_00\6&2b8c85f2&0&20200050"
ServiceName is "copleycan"

lkd> !devnode 84a84800
DevNode 0x84a84800 for PDO 0x85552030
Parent 0x84e803f0 Sibling 0000000000 Child 0000000000
InstancePath is
"PCI\VEN_10EE&DEV_D154&SUBSYS_D15410EE&REV_00\6&2b8c85f2&0&20200050"
ServiceName is "copleycan"
State = DeviceNodeStarted (0x308)
Previous State = DeviceNodeEnumerateCompletion (0x30d)
StateHistory[07] = DeviceNodeEnumerateCompletion (0x30d)
StateHistory[06] = DeviceNodeStarted (0x308)
StateHistory[05] = DeviceNodeStartPostWork (0x307)
StateHistory[04] = DeviceNodeStartCompletion (0x306)
StateHistory[03] = DeviceNodeResourcesAssigned (0x304)
StateHistory[02] = DeviceNodeDriversAdded (0x303)
StateHistory[01] = DeviceNodeInitialized (0x302)
StateHistory[00] = DeviceNodeUninitialized (0x301)
StateHistory[19] = Unknown State (0x0)
StateHistory[18] = Unknown State (0x0)
StateHistory[17] = Unknown State (0x0)
StateHistory[16] = Unknown State (0x0)
StateHistory[15] = Unknown State (0x0)
StateHistory[14] = Unknown State (0x0)
StateHistory[13] = Unknown State (0x0)
StateHistory[12] = Unknown State (0x0)
StateHistory[11] = Unknown State (0x0)
StateHistory[10] = Unknown State (0x0)
StateHistory[09] = Unknown State (0x0)
StateHistory[08] = Unknown State (0x0)
Flags (0x00000030) DNF_ENUMERATED, DNF_IDS_QUERIED

After surprise removal of the bus extender:

lkd> !wdfkd.wdfdevice 0x7b5daa60

Dumping WDFDEVICE 0x7b5daa60

WDM PDEVICE_OBJECTs:self 84a44b98, attached 84a87a60, pdo 85552030

Pnp state: 12e ( WdfDevStatePnpFailedWaitForRemove )
Power state: 315 ( WdfDevStatePowerStopped )
Power Pol state: 563 ( WdfDevStatePwrPolStopped )

Default WDFIOTARGET: 7aa81ed8

No pended pnp or power irps
Device is the power policy owner for the stack

lkd> !devstack 84a44b98
!DevObj !DrvObj !DevExt ObjectName

84a44b98 \Driver\copleycan 84a25760
84a87a60 \Driver\mrtcb 84a87b18

lkd> !devnode 84a84800
DevNode 0x84a84800 for PDO 0x85552030
Parent 0000000000 Sibling 0000000000 Child 0000000000
InstancePath is
"PCI\VEN_10EE&DEV_D154&SUBSYS_D15410EE&REV_00\6&2b8c85f2&0&20200050"
ServiceName is "copleycan"
State = DeviceNodeDeleted (0x314)
Previous State = DeviceNodeDeletePendingCloses (0x313)
StateHistory[12] = DeviceNodeDeletePendingCloses (0x313)
StateHistory[11] = DeviceNodeRemovePendingCloses (0x311)
StateHistory[10] = DeviceNodeStarted (0x308)
StateHistory[09] = DeviceNodeAwaitingQueuedDeletion (0x30e)
StateHistory[08] = DeviceNodeStarted (0x308)
StateHistory[07] = DeviceNodeEnumerateCompletion (0x30d)
StateHistory[06] = DeviceNodeStarted (0x308)
StateHistory[05] = DeviceNodeStartPostWork (0x307)
StateHistory[04] = DeviceNodeStartCompletion (0x306)
StateHistory[03] = DeviceNodeResourcesAssigned (0x304)
StateHistory[02] = DeviceNodeDriversAdded (0x303)
StateHistory[01] = DeviceNodeInitialized (0x302)
StateHistory[00] = DeviceNodeUninitialized (0x301)
StateHistory[19] = Unknown State (0x0)
StateHistory[18] = Unknown State (0x0)
StateHistory[17] = Unknown State (0x0)
StateHistory[16] = Unknown State (0x0)
StateHistory[15] = Unknown State (0x0)
StateHistory[14] = Unknown State (0x0)
StateHistory[13] = Unknown State (0x0)
Flags (0x00002020) DNF_IDS_QUERIED, DNF_HAS_PROBLEM
Problem = CM_PROB_DEVICE_NOT_THERE

I'm trying to make sense of this info, but am light on experience. Any
thoughts?

Thanks,
Steve

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at

Stephen_Glow · May 23, 2007, 1:46pm

That the driver for the cardbus PCI extender.

The device is a single slot extender built by Magma (www.magma.com).

I’ve just confirmed that I’m running their latest drivers for this product.

You think it’s a problem with this driver?

Doron_Holan · May 23, 2007, 2:10pm

It could very well be. Does this driver have a management app as well?
One thing you can do is register for
EvtDeviceFileCreate/EvtFileCleanup/EvtFileClose by calling
WdfDeviceInitSetFileObjectConfig, setting a bp on each of these
functions. If they are being hit, a handle is being created. Upon
breaking in on EvtDeviceFileCreate, get a stack trace (k) and try to get
the app’s name (.reload /user might be helpful and then rerun the ‘k’
command)

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@embeddedintelligence.com
Sent: Wednesday, May 23, 2007 10:46 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Device cleanup not called after surprise removal

That the driver for the cardbus PCI extender.

The device is a single slot extender built by Magma (www.magma.com).

I’ve just confirmed that I’m running their latest drivers for this
product.

You think it’s a problem with this driver?

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Stephen_Glow · May 23, 2007, 4:58pm

Hi Doron;

The bridge doesn’t have any type of management app that I’m aware of.

I added the FileCreate / Close / Cleanup callbacks and none of them are getting called. I also used the debugger command !wdfkd.wdfopenhandles to check for open handles on my device and didn’t find any, either before or after the surprise remove.

One more test I did was to load the driver on a desktop machine and call WdfDeviceSetFailed to force a surprise remove. This seemed to be handled correctly in that the cleanup and driver unload functions were called. I tried the same test on the laptop with the bus extender and the cleanup / unload were still not called.

At this point I’m ready to call this a bug in the lower level bus extender driver and be done with it. I’ll let the company that makes this device know and will work with them to resolve the issue if they are interested.

Thanks for your help with this Doron. I really appreciate how accessible you are for such problems.

Best regards,
Steve