0xD1 crash

Prasad_Talekar-2 · August 17, 2009, 9:55am

Hi,

I am seeing 0xD1 crash with my WDF based driver. From the wdflogdump it is observed that my driver gets an IRP from Child device even after the Child PDO gets deleted and is removed from the bus.

wdflogdump is given below

32070: FxChildList::ProcessBusRelations - PDO WDFDEVICE 7A0D61C8 !devobj 85F21A10 reported as missing to pnp
32071: FxPkgPnp::HandleQueryBusRelations - WDFDEVICE 791E1FB0 returning 0 devices in relations E33A81D0
32072: FxPkgPnp::Dispatch - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10, IRP_MJ_PNP, 0x00000007(IRP_MN_QUERY_DEVICE_RELATIONS) IRP 0x862F37A8
32073: FxPkgPnp::Dispatch - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10, IRP_MJ_PNP, 0x00000002(IRP_MN_REMOVE_DEVICE) IRP 0x862F37A8
32074: FxPkgPnp::PnpEnterNewState - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10 entering PnP State WdfDevStatePnpRemoved from WdfDevStatePnpFailedWaitForRemove
32075: FxPkgPnp::PnpEnterNewState - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10 entering PnP State WdfDevStatePnpRemovedChildrenRemoved from WdfDevStatePnpRemoved
32076: FxPkgPnp::PnpEnterNewState - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10 entering PnP State WdfDevStatePnpCheckForDevicePresence from WdfDevStatePnpRemovedChildrenRemoved
32077: FxPkgPnp::PnpEnterNewState - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10 entering PnP State WdfDevStatePnpPdoRemoved from WdfDevStatePnpCheckForDevicePresence
32078: FxIoQueue::QueuePurge - All WDFQUEUE 0x7A0AE818 requests cancelled
32079: FxIoQueue::QueuePurge - All driver cancellable requests cancelled in WDFQUEUE 0x7A0AE818
32080: FxIoQueue::QueuePurge - All WDFQUEUE 0x79BDF398 requests cancelled
32081: FxIoQueue::QueuePurge - All driver cancellable requests cancelled in WDFQUEUE 0x79BDF398
32082: FxSelfManagedIoMachine::ProcessEvent - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10 entering self managed io state FxSelfManagedIoCleanup from FxSelfManagedIoFlushed
32083: FxSelfManagedIoMachine::ProcessEvent - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10 entering self managed io state FxSelfManagedIoFinal from FxSelfManagedIoCleanup
32084: FxObject::~FxObject - Object 85F29E30, WDFOBJECT 7A0D61C8 transitioning from FxObjectStateCreated to FxObjectStateDisposingEarly
32085: FxObject::~FxObject - Object 85F29E30, WDFOBJECT 7A0D61C8 transitioning from FxObjectStateDisposingEarly to FxObjectStateDisposingDisposeChildren
32086: FxObject::~FxObject - Object 85F517E0, WDFOBJECT 7A0AE818 transitioning from FxObjectStateCreated to FxObjectStateDisposingEarly
…
…
…
…
32116: FxObject::~FxObject - Object 85F70048, WDFOBJECT 7A08FFB0 transitioning from FxObjectStateDeletedAndDisposed to FxObjectStateDestroyed
32117: FxObject::~FxObject - Object 85F827A0, WDFOBJECT 7A07D858 transitioning from FxObjectStateDisposed to FxObjectStateDeletedAndDisposed
32118: FxObject::~FxObject - Object 85F6E740, WDFOBJECT 7A0918B8 transitioning from FxObjectStateDisposed to FxObjectStateDeletedAndDisposed
32119: FxPkgPnp::PnpEnterNewState - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10 entering PnP State WdfDevStatePnpFinal from WdfDevStatePnpPdoRemoved
32120: FxPkgPnp::_PnpRemoveDevice - WDFDEVICE 7A0D61C8, !devobj 85F21A10 waiting for remove event to finish processing
32121: FxDevice::Destroy - Deleting !devobj 85F21A10, WDFDEVICE 7A0D61C8, attached to !devobj 00000000
32122: FxObject::~FxObject - Object 85F29E30, WDFOBJECT 7A0D61C8 transitioning from FxObjectStateDisposed to FxObjectStateDeletedAndDisposed
32123: FxObject::~FxObject - Object 85F29E30, WDFOBJECT 7A0D61C8 transitioning from FxObjectStateDeletedAndDisposed to FxObjectStateDestroyed

How does it possible? What is going wrong here? Any help?

\Prasad

Doron_Holan · August 17, 2009, 10:05am

What is the callstack? Who is sending the io? What irp is it? What is the output of !analyze -v? What stack does the pdo load? Do you enable any device interfaces on the pdo itself?

Typically receiving io on a deleted pdo is a bug in the caller, not your driver but there is not enough context yet to know that is the root cause or not

d

Sent from my phone with no t9, all spilling mistakes are not intentional.

-----Original Message-----
From: xxxxx@yahoo.com
Sent: Monday, August 17, 2009 6:56 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] 0xD1 crash

Hi,

I am seeing 0xD1 crash with my WDF based driver. From the wdflogdump it is observed that my driver gets an IRP from Child device even after the Child PDO gets deleted and is removed from the bus.

wdflogdump is given below

32070: FxChildList::ProcessBusRelations - PDO WDFDEVICE 7A0D61C8 !devobj 85F21A10 reported as missing to pnp
32071: FxPkgPnp::HandleQueryBusRelations - WDFDEVICE 791E1FB0 returning 0 devices in relations E33A81D0
32072: FxPkgPnp::Dispatch - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10, IRP_MJ_PNP, 0x00000007(IRP_MN_QUERY_DEVICE_RELATIONS) IRP 0x862F37A8
32073: FxPkgPnp::Dispatch - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10, IRP_MJ_PNP, 0x00000002(IRP_MN_REMOVE_DEVICE) IRP 0x862F37A8
32074: FxPkgPnp::PnpEnterNewState - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10 entering PnP State WdfDevStatePnpRemoved from WdfDevStatePnpFailedWaitForRemove
32075: FxPkgPnp::PnpEnterNewState - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10 entering PnP State WdfDevStatePnpRemovedChildrenRemoved from WdfDevStatePnpRemoved
32076: FxPkgPnp::PnpEnterNewState - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10 entering PnP State WdfDevStatePnpCheckForDevicePresence from WdfDevStatePnpRemovedChildrenRemoved
32077: FxPkgPnp::PnpEnterNewState - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10 entering PnP State WdfDevStatePnpPdoRemoved from WdfDevStatePnpCheckForDevicePresence
32078: FxIoQueue::QueuePurge - All WDFQUEUE 0x7A0AE818 requests cancelled
32079: FxIoQueue::QueuePurge - All driver cancellable requests cancelled in WDFQUEUE 0x7A0AE818
32080: FxIoQueue::QueuePurge - All WDFQUEUE 0x79BDF398 requests cancelled
32081: FxIoQueue::QueuePurge - All driver cancellable requests cancelled in WDFQUEUE 0x79BDF398
32082: FxSelfManagedIoMachine::ProcessEvent - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10 entering self managed io state FxSelfManagedIoCleanup from FxSelfManagedIoFlushed
32083: FxSelfManagedIoMachine::ProcessEvent - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10 entering self managed io state FxSelfManagedIoFinal from FxSelfManagedIoCleanup
32084: FxObject::~FxObject - Object 85F29E30, WDFOBJECT 7A0D61C8 transitioning from FxObjectStateCreated to FxObjectStateDisposingEarly
32085: FxObject::~FxObject - Object 85F29E30, WDFOBJECT 7A0D61C8 transitioning from FxObjectStateDisposingEarly to FxObjectStateDisposingDisposeChildren
32086: FxObject::~FxObject - Object 85F517E0, WDFOBJECT 7A0AE818 transitioning from FxObjectStateCreated to FxObjectStateDisposingEarly
…
…
…
…
32116: FxObject::~FxObject - Object 85F70048, WDFOBJECT 7A08FFB0 transitioning from FxObjectStateDeletedAndDisposed to FxObjectStateDestroyed
32117: FxObject::~FxObject - Object 85F827A0, WDFOBJECT 7A07D858 transitioning from FxObjectStateDisposed to FxObjectStateDeletedAndDisposed
32118: FxObject::~FxObject - Object 85F6E740, WDFOBJECT 7A0918B8 transitioning from FxObjectStateDisposed to FxObjectStateDeletedAndDisposed
32119: FxPkgPnp::PnpEnterNewState - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10 entering PnP State WdfDevStatePnpFinal from WdfDevStatePnpPdoRemoved
32120: FxPkgPnp::_PnpRemoveDevice - WDFDEVICE 7A0D61C8, !devobj 85F21A10 waiting for remove event to finish processing
32121: FxDevice::Destroy - Deleting !devobj 85F21A10, WDFDEVICE 7A0D61C8, attached to !devobj 00000000
32122: FxObject::~FxObject - Object 85F29E30, WDFOBJECT 7A0D61C8 transitioning from FxObjectStateDisposed to FxObjectStateDeletedAndDisposed
32123: FxObject::~FxObject - Object 85F29E30, WDFOBJECT 7A0D61C8 transitioning from FxObjectStateDeletedAndDisposed to FxObjectStateDestroyed

How does it possible? What is going wrong here? Any help?

\Prasad

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Prasad_Talekar-2 · August 17, 2009, 11:05am

It is Wireless USB (WUSB) stack. I have WUSB_HOST driver. Child driver is WUSB_HUB. From logdump it observed that REMOVE_DEVICE IRP is not completed. Other details including call stack and irp are given below

0: kd> !analyze -v

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an interrupt request level (IRQL) that is too high. This is usually caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: ffffffe8, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, value 0 = read operation, 1 = write operation
Arg4: f73d85f1, address which referenced memory

Debugging Details:

*** No owner thread found for resource 8055a4e0
*** No owner thread found for resource 8055a4e0
*** No owner thread found for resource 8055a4e0

READ_ADDRESS: ffffffe8

CURRENT_IRQL: 2

FAULTING_IP:
wdf01000!FxDevice::Dispatch+b
f73d85f1 8b48e8 mov ecx,dword ptr [eax-18h]

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xD1

PROCESS_NAME: System

TRAP_FRAME: f78d6870 – (.trap 0xfffffffff78d6870)
ErrCode = 00000000
eax=00000000 ebx=880cef02 ecx=85bf0b10 edx=87a9af20 esi=86636040 edi=85bf0b10
eip=f73d85f1 esp=f78d68e4 ebp=f78d68e4 iopl=0 nv up ei ng nz na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010286
wdf01000!FxDevice::Dispatch+0xb:
f73d85f1 8b48e8 mov ecx,dword ptr [eax-18h] ds:0023:ffffffe8=???
Resetting default scope

LOCK_ADDRESS: 8055a560 – (!locks 8055a560)

Resource @ nt!IopDeviceTreeLock (0x8055a560) Shared 1 owning threads
Contention Count = 2
Threads: 86c20b30-01<*>
1 total locks, 1 locks currently held

PNP_TRIAGE:
Lock address : 0x8055a560
Thread Count : 1
Thread address: 0x86c20b30
Thread wait : 0x256a

LAST_CONTROL_TRANSFER: from f73d85f1 to 805436e0

STACK_TEXT:
f78d6870 f73d85f1 badb0d00 87a9af20 806624a2 nt!KiTrap0E+0x238
f78d68e4 804eeec5 85bf0b10 87a9af20 806e4428 wdf01000!FxDevice::Dispatch+0xb
f78d68f4 80656128 86ec0f88 85c67afc 880cef20 nt!IopfCallDriver+0x31
f78d6918 f4cf53c9 f78d693c f4cf5504 85bf0b10 nt!IovCallDriver+0xa0
WARNING: Stack unwind information not available. Following frames may be wrong.
f78d6920 f4cf5504 85bf0b10 87a9af20 86ec0f94 WUSB_HUB+0x23c9
f78d693c f4cf6a99 85bf0b10 87a9af20 86ec0f94 WUSB_HUB+0x2504
f78d6974 f4cfd7d8 85bf0b10 85c67a58 87a9af20 WUSB_HUB+0x3a99
f78d69a0 f4cfdae5 880cef20 860166b0 8668ef38 WUSB_HUB+0xa7d8
f78d69b8 f4cf72d3 860166b0 880cef20 f78d69fc WUSB_HUB+0xaae5
f78d69c8 804eeec5 860166b0 880cef20 806e4428 WUSB_HUB+0x42d3
f78d69d8 80656128 85c679a8 85c67afc 880cef20 nt!IopfCallDriver+0x31
f78d69fc f7648c15 85c50001 85c679a8 00000001 nt!IovCallDriver+0xa0
f78d6a10 f7648d5f 85c679a8 880cef20 85c67afc usbhub!USBH_ChangeIndicationQueryChange+0xd5
f78d6a38 80656330 00000000 86790068 86790000 usbhub!USBH_ChangeIndication+0x13d
f78d6a5c 804f13f6 00000000 88264f20 f78d6ac0 nt!IovpLocalCompletionRoutine+0xb4
f78d6a8c 806567b8 c0000120 878b0ff0 88264f20 nt!IopfCompleteRequest+0xa2
f78d6af8 f4cf34a3 85d09bd8 f78d6b2c f4cf6be0 nt!IovCompleteRequest+0x9a
f78d6b04 f4cf6be0 88264f20 c0000120 00000000 WUSB_HUB+0x4a3
f78d6b2c f4cf4787 85d090d8 880c6f20 880c6fd8 WUSB_HUB+0x3be0
f78d6b44 f4cf4c32 85d09020 880c6f20 85d09020 WUSB_HUB+0x1787
f78d6b70 f4cf72d3 85d09020 880c6f20 f78d6bb4 WUSB_HUB+0x1c32
f78d6b80 804eeec5 85d09020 880c6f20 806e4428 WUSB_HUB+0x42d3
f78d6b90 80656128 880c6ffc f78d6c30 880c6f20 nt!IopfCallDriver+0x31
f78d6bb4 8059181f 85bf0b10 85bf0b10 00000002 nt!IovCallDriver+0xa0
f78d6be0 80591a81 85d09020 f78d6c0c 00000000 nt!IopSynchronousCall+0xb7
f78d6c34 804f6ca2 85bf0b10 00000002 00000000 nt!IopRemoveDevice+0x93
f78d6c5c 8059344a e10ab780 00000018 e114aa30 nt!IopRemoveLockedDeviceNode+0x160
f78d6c74 805934b1 85d072c0 00000002 e114aa30 nt!IopDeleteLockedDeviceNode+0x34
f78d6ca8 8059911f 85bf0b10 0214aa30 00000002 nt!IopDeleteLockedDeviceNodes+0x3f
f78d6d3c 805993e2 f78d6d78 806e4974 e318f1f8 nt!PiProcessQueryRemoveAndEject+0x76b
f78d6d58 80599528 f78d6d78 85c71a50 8056375c nt!PiProcessTargetDeviceEvent+0x2a
f78d6d7c 8053776b 85c71a50 00000000 86c20b30 nt!PiWalkDeviceList+0xea
f78d6dac 805ce7b2 85c71a50 00000000 00000000 nt!ExpWorkerThread+0xef
f78d6ddc 805450de 8053767c 00000001 00000000 nt!PspSystemThreadStartup+0x34
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

======================================================

IRP details are

0: kd> !irp 0x880C6F20
Irp is active with 3 stacks 3 is current (= 0x880c6fd8)
No Mdl: No System Buffer: Thread 86c20b30: Irp stack trace.
cmd flg cl Device File Completion-Context
[0, 0] 0 10 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[1b, 0] 0 10 85bf0b10 00000000 f4cf73d4-f78d6b00
\Driver\WUSB_HOST

WUSB_HUB
Args: 00000000 00000000 00000000 00000000

[1b, 2] 0 0 85d09020 00000000 00000000-00000000
\Driver\ WUSB_HUB
Args: 00000000 00000000 00000000 00000000

0: kd> !irp 87a9af20
Irp is active with 3 stacks 3 is current (= 0x87a9afd8)
No Mdl: No System Buffer: Thread 00000000: Irp stack trace.
cmd flg cl Device File Completion-Context
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 10 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000

[f, 0] 0 e0 85bf0b10 00000000 f4cf6852-86ec0f88 Success Error Cancel
\Driver\ WUSB_HOSTWUSB_HUB
Args: 86ec0f94 00000000 00220003 00000000

======================================================

Doron, one more doubt from your previous statement, ?Typically receiving io on a deleted pdo is a bug in the caller?. How come the stack still active even after deleting the PDO?

\Prasad

Doron_Holan · August 17, 2009, 11:15am

Fix your symbols. What irp is being dispatched? A pnp remove?

d

Sent from my phone with no t9, all spilling mistakes are not intentional.

-----Original Message-----
From: xxxxx@yahoo.com
Sent: Monday, August 17, 2009 8:05 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] 0xD1 crash

It is Wireless USB (WUSB) stack. I have WUSB_HOST driver. Child driver is WUSB_HUB. From logdump it observed that REMOVE_DEVICE IRP is not completed. Other details including call stack and irp are given below

0: kd> !analyze -v

DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
An attempt was made to access a pageable (or completely invalid) address at an interrupt request level (IRQL) that is too high. This is usually caused by drivers using improper addresses.
If kernel debugger is available get stack backtrace.
Arguments:
Arg1: ffffffe8, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000000, value 0 = read operation, 1 = write operation
Arg4: f73d85f1, address which referenced memory

Debugging Details:
------------------

No owner thread found for resource 8055a4e0
No owner thread found for resource 8055a4e0
*** No owner thread found for resource 8055a4e0

READ_ADDRESS: ffffffe8

CURRENT_IRQL: 2

FAULTING_IP:
wdf01000!FxDevice::Dispatch+b
f73d85f1 8b48e8 mov ecx,dword ptr [eax-18h]

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xD1

PROCESS_NAME: System

TRAP_FRAME: f78d6870 – (.trap 0xfffffffff78d6870)
ErrCode = 00000000
eax=00000000 ebx=880cef02 ecx=85bf0b10 edx=87a9af20 esi=86636040 edi=85bf0b10
eip=f73d85f1 esp=f78d68e4 ebp=f78d68e4 iopl=0 nv up ei ng nz na pe nc
cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000 efl=00010286
wdf01000!FxDevice::Dispatch+0xb:
f73d85f1 8b48e8 mov ecx,dword ptr [eax-18h] ds:0023:ffffffe8=???
Resetting default scope

LOCK_ADDRESS: 8055a560 – (!locks 8055a560)

Resource @ nt!IopDeviceTreeLock (0x8055a560) Shared 1 owning threads
Contention Count = 2
Threads: 86c20b30-01<*>
1 total locks, 1 locks currently held

PNP_TRIAGE:
Lock address : 0x8055a560
Thread Count : 1
Thread address: 0x86c20b30
Thread wait : 0x256a

LAST_CONTROL_TRANSFER: from f73d85f1 to 805436e0

STACK_TEXT:
f78d6870 f73d85f1 badb0d00 87a9af20 806624a2 nt!KiTrap0E+0x238
f78d68e4 804eeec5 85bf0b10 87a9af20 806e4428 wdf01000!FxDevice::Dispatch+0xb
f78d68f4 80656128 86ec0f88 85c67afc 880cef20 nt!IopfCallDriver+0x31
f78d6918 f4cf53c9 f78d693c f4cf5504 85bf0b10 nt!IovCallDriver+0xa0
WARNING: Stack unwind information not available. Following frames may be wrong.
f78d6920 f4cf5504 85bf0b10 87a9af20 86ec0f94 WUSB_HUB+0x23c9
f78d693c f4cf6a99 85bf0b10 87a9af20 86ec0f94 WUSB_HUB+0x2504
f78d6974 f4cfd7d8 85bf0b10 85c67a58 87a9af20 WUSB_HUB+0x3a99
f78d69a0 f4cfdae5 880cef20 860166b0 8668ef38 WUSB_HUB+0xa7d8
f78d69b8 f4cf72d3 860166b0 880cef20 f78d69fc WUSB_HUB+0xaae5
f78d69c8 804eeec5 860166b0 880cef20 806e4428 WUSB_HUB+0x42d3
f78d69d8 80656128 85c679a8 85c67afc 880cef20 nt!IopfCallDriver+0x31
f78d69fc f7648c15 85c50001 85c679a8 00000001 nt!IovCallDriver+0xa0
f78d6a10 f7648d5f 85c679a8 880cef20 85c67afc usbhub!USBH_ChangeIndicationQueryChange+0xd5
f78d6a38 80656330 00000000 86790068 86790000 usbhub!USBH_ChangeIndication+0x13d
f78d6a5c 804f13f6 00000000 88264f20 f78d6ac0 nt!IovpLocalCompletionRoutine+0xb4
f78d6a8c 806567b8 c0000120 878b0ff0 88264f20 nt!IopfCompleteRequest+0xa2
f78d6af8 f4cf34a3 85d09bd8 f78d6b2c f4cf6be0 nt!IovCompleteRequest+0x9a
f78d6b04 f4cf6be0 88264f20 c0000120 00000000 WUSB_HUB+0x4a3
f78d6b2c f4cf4787 85d090d8 880c6f20 880c6fd8 WUSB_HUB+0x3be0
f78d6b44 f4cf4c32 85d09020 880c6f20 85d09020 WUSB_HUB+0x1787
f78d6b70 f4cf72d3 85d09020 880c6f20 f78d6bb4 WUSB_HUB+0x1c32
f78d6b80 804eeec5 85d09020 880c6f20 806e4428 WUSB_HUB+0x42d3
f78d6b90 80656128 880c6ffc f78d6c30 880c6f20 nt!IopfCallDriver+0x31
f78d6bb4 8059181f 85bf0b10 85bf0b10 00000002 nt!IovCallDriver+0xa0
f78d6be0 80591a81 85d09020 f78d6c0c 00000000 nt!IopSynchronousCall+0xb7
f78d6c34 804f6ca2 85bf0b10 00000002 00000000 nt!IopRemoveDevice+0x93
f78d6c5c 8059344a e10ab780 00000018 e114aa30 nt!IopRemoveLockedDeviceNode+0x160
f78d6c74 805934b1 85d072c0 00000002 e114aa30 nt!IopDeleteLockedDeviceNode+0x34
f78d6ca8 8059911f 85bf0b10 0214aa30 00000002 nt!IopDeleteLockedDeviceNodes+0x3f
f78d6d3c 805993e2 f78d6d78 806e4974 e318f1f8 nt!PiProcessQueryRemoveAndEject+0x76b
f78d6d58 80599528 f78d6d78 85c71a50 8056375c nt!PiProcessTargetDeviceEvent+0x2a
f78d6d7c 8053776b 85c71a50 00000000 86c20b30 nt!PiWalkDeviceList+0xea
f78d6dac 805ce7b2 85c71a50 00000000 00000000 nt!ExpWorkerThread+0xef
f78d6ddc 805450de 8053767c 00000001 00000000 nt!PspSystemThreadStartup+0x34
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16

======================================================

IRP details are

0: kd> !irp 0x880C6F20
Irp is active with 3 stacks 3 is current (= 0x880c6fd8)
No Mdl: No System Buffer: Thread 86c20b30: Irp stack trace.
cmd flg cl Device File Completion-Context
[0, 0] 0 10 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[1b, 0] 0 10 85bf0b10 00000000 f4cf73d4-f78d6b00
\Driver\WUSB_HOST

WUSB_HUB
Args: 00000000 00000000 00000000 00000000

>[1b, 2] 0 0 85d09020 00000000 00000000-00000000
\Driver\ WUSB_HUB
Args: 00000000 00000000 00000000 00000000

0: kd> !irp 87a9af20
Irp is active with 3 stacks 3 is current (= 0x87a9afd8)
No Mdl: No System Buffer: Thread 00000000: Irp stack trace.
cmd flg cl Device File Completion-Context
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 10 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
>[f, 0] 0 e0 85bf0b10 00000000 f4cf6852-86ec0f88 Success Error Cancel
\Driver\ WUSB_HOSTWUSB_HUB
Args: 86ec0f94 00000000 00220003 00000000

======================================================

Doron, one more doubt from your previous statement, ?Typically receiving io on a deleted pdo is a bug in the caller?. How come the stack still active even after deleting the PDO?

\Prasad

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Prasad_Talekar · August 18, 2009, 12:43am

I don’t have symbols for WUSB_HUB driver as it is third party driver.

Yeh, IRP being dispatched is IRP_MN_REMOVE_DEVICE.

\Prasad

On Mon, Aug 17, 2009 at 8:44 PM, Doron Holan wrote:

> Fix your symbols. What irp is being dispatched? A pnp remove?
>
> d
>
> Sent from my phone with no t9, all spilling mistakes are not intentional.
>
> -----Original Message-----
> From: xxxxx@yahoo.com
> Sent: Monday, August 17, 2009 8:05 AM
> To: Windows System Software Devs Interest List
> Subject: RE:[ntdev] 0xD1 crash
>
>
> It is Wireless USB (WUSB) stack. I have WUSB_HOST driver. Child driver is
> WUSB_HUB. From logdump it observed that REMOVE_DEVICE IRP is not completed.
> Other details including call stack and irp are given below
>
> 0: kd> !analyze -v
>
> DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
> An attempt was made to access a pageable (or completely invalid) address at
> an interrupt request level (IRQL) that is too high. This is usually caused
> by drivers using improper addresses.
> If kernel debugger is available get stack backtrace.
> Arguments:
> Arg1: ffffffe8, memory referenced
> Arg2: 00000002, IRQL
> Arg3: 00000000, value 0 = read operation, 1 = write operation
> Arg4: f73d85f1, address which referenced memory
>
> Debugging Details:
> ------------------
>
> No owner thread found for resource 8055a4e0
> No owner thread found for resource 8055a4e0
> *** No owner thread found for resource 8055a4e0
>
> READ_ADDRESS: ffffffe8
>
> CURRENT_IRQL: 2
>
> FAULTING_IP:
> wdf01000!FxDevice::Dispatch+b
> f73d85f1 8b48e8 mov ecx,dword ptr [eax-18h]
>
> DEFAULT_BUCKET_ID: DRIVER_FAULT
>
> BUGCHECK_STR: 0xD1
>
> PROCESS_NAME: System
>
> TRAP_FRAME: f78d6870 – (.trap 0xfffffffff78d6870)
> ErrCode = 00000000
> eax=00000000 ebx=880cef02 ecx=85bf0b10 edx=87a9af20 esi=86636040
> edi=85bf0b10
> eip=f73d85f1 esp=f78d68e4 ebp=f78d68e4 iopl=0 nv up ei ng nz na pe
> nc
> cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000
> efl=00010286
> wdf01000!FxDevice::Dispatch+0xb:
> f73d85f1 8b48e8 mov ecx,dword ptr [eax-18h]
> ds:0023:ffffffe8=???
> Resetting default scope
>
> LOCK_ADDRESS: 8055a560 – (!locks 8055a560)
>
> Resource @ nt!IopDeviceTreeLock (0x8055a560) Shared 1 owning threads
> Contention Count = 2
> Threads: 86c20b30-01<*>
> 1 total locks, 1 locks currently held
>
> PNP_TRIAGE:
> Lock address : 0x8055a560
> Thread Count : 1
> Thread address: 0x86c20b30
> Thread wait : 0x256a
>
> LAST_CONTROL_TRANSFER: from f73d85f1 to 805436e0
>
> STACK_TEXT:
> f78d6870 f73d85f1 badb0d00 87a9af20 806624a2 nt!KiTrap0E+0x238
> f78d68e4 804eeec5 85bf0b10 87a9af20 806e4428
> wdf01000!FxDevice::Dispatch+0xb
> f78d68f4 80656128 86ec0f88 85c67afc 880cef20 nt!IopfCallDriver+0x31
> f78d6918 f4cf53c9 f78d693c f4cf5504 85bf0b10 nt!IovCallDriver+0xa0
> WARNING: Stack unwind information not available. Following frames may be
> wrong.
> f78d6920 f4cf5504 85bf0b10 87a9af20 86ec0f94 WUSB_HUB+0x23c9
> f78d693c f4cf6a99 85bf0b10 87a9af20 86ec0f94 WUSB_HUB+0x2504
> f78d6974 f4cfd7d8 85bf0b10 85c67a58 87a9af20 WUSB_HUB+0x3a99
> f78d69a0 f4cfdae5 880cef20 860166b0 8668ef38 WUSB_HUB+0xa7d8
> f78d69b8 f4cf72d3 860166b0 880cef20 f78d69fc WUSB_HUB+0xaae5
> f78d69c8 804eeec5 860166b0 880cef20 806e4428 WUSB_HUB+0x42d3
> f78d69d8 80656128 85c679a8 85c67afc 880cef20 nt!IopfCallDriver+0x31
> f78d69fc f7648c15 85c50001 85c679a8 00000001 nt!IovCallDriver+0xa0
> f78d6a10 f7648d5f 85c679a8 880cef20 85c67afc
> usbhub!USBH_ChangeIndicationQueryChange+0xd5
> f78d6a38 80656330 00000000 86790068 86790000
> usbhub!USBH_ChangeIndication+0x13d
> f78d6a5c 804f13f6 00000000 88264f20 f78d6ac0
> nt!IovpLocalCompletionRoutine+0xb4
> f78d6a8c 806567b8 c0000120 878b0ff0 88264f20 nt!IopfCompleteRequest+0xa2
> f78d6af8 f4cf34a3 85d09bd8 f78d6b2c f4cf6be0 nt!IovCompleteRequest+0x9a
> f78d6b04 f4cf6be0 88264f20 c0000120 00000000 WUSB_HUB+0x4a3
> f78d6b2c f4cf4787 85d090d8 880c6f20 880c6fd8 WUSB_HUB+0x3be0
> f78d6b44 f4cf4c32 85d09020 880c6f20 85d09020 WUSB_HUB+0x1787
> f78d6b70 f4cf72d3 85d09020 880c6f20 f78d6bb4 WUSB_HUB+0x1c32
> f78d6b80 804eeec5 85d09020 880c6f20 806e4428 WUSB_HUB+0x42d3
> f78d6b90 80656128 880c6ffc f78d6c30 880c6f20 nt!IopfCallDriver+0x31
> f78d6bb4 8059181f 85bf0b10 85bf0b10 00000002 nt!IovCallDriver+0xa0
> f78d6be0 80591a81 85d09020 f78d6c0c 00000000 nt!IopSynchronousCall+0xb7
> f78d6c34 804f6ca2 85bf0b10 00000002 00000000 nt!IopRemoveDevice+0x93
> f78d6c5c 8059344a e10ab780 00000018 e114aa30
> nt!IopRemoveLockedDeviceNode+0x160
> f78d6c74 805934b1 85d072c0 00000002 e114aa30
> nt!IopDeleteLockedDeviceNode+0x34
> f78d6ca8 8059911f 85bf0b10 0214aa30 00000002
> nt!IopDeleteLockedDeviceNodes+0x3f
> f78d6d3c 805993e2 f78d6d78 806e4974 e318f1f8
> nt!PiProcessQueryRemoveAndEject+0x76b
> f78d6d58 80599528 f78d6d78 85c71a50 8056375c
> nt!PiProcessTargetDeviceEvent+0x2a
> f78d6d7c 8053776b 85c71a50 00000000 86c20b30 nt!PiWalkDeviceList+0xea
> f78d6dac 805ce7b2 85c71a50 00000000 00000000 nt!ExpWorkerThread+0xef
> f78d6ddc 805450de 8053767c 00000001 00000000 nt!PspSystemThreadStartup+0x34
> 00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16
>
> ======================================================
>
> IRP details are
>
> 0: kd> !irp 0x880C6F20
> Irp is active with 3 stacks 3 is current (= 0x880c6fd8)
> No Mdl: No System Buffer: Thread 86c20b30: Irp stack trace.
> cmd flg cl Device File Completion-Context
> [0, 0] 0 10 00000000 00000000 00000000-00000000
>
> Args: 00000000 00000000 00000000 00000000
> [1b, 0] 0 10 85bf0b10 00000000 f4cf73d4-f78d6b00
> \Driver\WUSB_HOST
>
> WUSB_HUB
> Args: 00000000 00000000 00000000 00000000
>
> >[1b, 2] 0 0 85d09020 00000000 00000000-00000000
> \Driver\ WUSB_HUB
> Args: 00000000 00000000 00000000 00000000
>
> 0: kd> !irp 87a9af20
> Irp is active with 3 stacks 3 is current (= 0x87a9afd8)
> No Mdl: No System Buffer: Thread 00000000: Irp stack trace.
> cmd flg cl Device File Completion-Context
> [0, 0] 0 0 00000000 00000000 00000000-00000000
>
> Args: 00000000 00000000 00000000 00000000
> [0, 0] 0 10 00000000 00000000 00000000-00000000
>
> Args: 00000000 00000000 00000000 00000000
> >[f, 0] 0 e0 85bf0b10 00000000 f4cf6852-86ec0f88 Success Error Cancel
> \Driver\ WUSB_HOSTWUSB_HUB
> Args: 86ec0f94 00000000 00220003 00000000
>
> ======================================================
>
> Doron, one more doubt from your previous statement, ?Typically receiving io
> on a deleted pdo is a bug in the caller?. How come the stack still active
> even after deleting the PDO?
>
> \Prasad
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

Prasad_Talekar-2 · August 18, 2009, 12:46am

I don’t have symbols for WUSB_HUB driver as it is third party driver.

Yeh, IRP being dispatched is IRP_MN_REMOVE_DEVICE.

\Prasad

Prasad_Talekar-2 · August 18, 2009, 7:14am

IRP details are

IRP_MN_REMOVE_DEVICE.

0: kd> !irp 0x880C6F20
Irp is active with 3 stacks 3 is current (= 0x880c6fd8)
No Mdl: No System Buffer: Thread 86c20b30: Irp stack trace.
cmd flg cl Device File Completion-Context
[0, 0] 0 10 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[1b, 0] 0 10 85bf0b10 00000000 f4cf73d4-f78d6b00
\Driver\WUSB_HOST

WUSB_HUB
Args: 00000000 00000000 00000000 00000000

[1b, 2] 0 0 85d09020 00000000 00000000-00000000
\Driver\ WUSB_HUB
Args: 00000000 00000000 00000000 00000000

IRP when system crashed

0: kd> !irp 87a9af20
Irp is active with 3 stacks 3 is current (= 0x87a9afd8)
No Mdl: No System Buffer: Thread 00000000: Irp stack trace.
cmd flg cl Device File Completion-Context
[0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[0, 0] 0 10 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000

[f, 0] 0 e0 85bf0b10 00000000 f4cf6852-86ec0f88 Success Error Cancel
\Driver\ WUSB_HOSTWUSB_HUB
Args: 86ec0f94 00000000 00220003 00000000

Prasad_Talekar-2 · August 20, 2009, 1:13am

Hi Doron,

I have mentioned IRP and other details in my previous posts. Any help regarding 0xD1 crash?

Doron_Holan · August 20, 2009, 2:00pm

Well the driver above your PDO is sending an internal IOCTL after it has send down the remove irp. On receiving the remove irp, kmdf has deleted the device object. It is a bug in the wusb_hub driver that they need to fix.

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@yahoo.com
Sent: Wednesday, August 19, 2009 10:14 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] 0xD1 crash

Hi Doron,

I have mentioned IRP and other details in my previous posts. Any help regarding 0xD1 crash?

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Abhishek_R · August 20, 2009, 2:09pm

Are the log dump and call stack taken from different test runs? The remove
IRP and device object pointers are not the same, so I am unable to get the
full picture.

For example, the !irp output shows that the remove IRP pointer is 0x880C6F20
and the WUSB_HOST device object is 0x85bf0b10. However, the log dump in your
previous post shows different values.
32073: FxPkgPnp::Dispatch - WDFDEVICE 0x7A0D61C8 !devobj 0x85F21A10,
IRP_MJ_PNP, 0x00000002(IRP_MN_REMOVE_DEVICE) IRP 0x862F37A8

Anyway, assuming they’re from different test runs, then it seems like you’re
getting an IO request after you’ve finished processing the remove IRP. As
Doron said, this is typically a bug in the caller.

wrote in message news:xxxxx@ntdev…
> It is Wireless USB (WUSB) stack. I have WUSB_HOST driver. Child driver is
> WUSB_HUB. From logdump it observed that REMOVE_DEVICE IRP is not
> completed. Other details including call stack and irp are given below
>
> 0: kd> !analyze -v
>
> DRIVER_IRQL_NOT_LESS_OR_EQUAL (d1)
> An attempt was made to access a pageable (or completely invalid) address
> at an interrupt request level (IRQL) that is too high. This is usually
> caused by drivers using improper addresses.
> If kernel debugger is available get stack backtrace.
> Arguments:
> Arg1: ffffffe8, memory referenced
> Arg2: 00000002, IRQL
> Arg3: 00000000, value 0 = read operation, 1 = write operation
> Arg4: f73d85f1, address which referenced memory
>
> Debugging Details:
> ------------------
>
> No owner thread found for resource 8055a4e0
> No owner thread found for resource 8055a4e0
> *** No owner thread found for resource 8055a4e0
>
> READ_ADDRESS: ffffffe8
>
> CURRENT_IRQL: 2
>
> FAULTING_IP:
> wdf01000!FxDevice::Dispatch+b
> f73d85f1 8b48e8 mov ecx,dword ptr [eax-18h]
>
> DEFAULT_BUCKET_ID: DRIVER_FAULT
>
> BUGCHECK_STR: 0xD1
>
> PROCESS_NAME: System
>
> TRAP_FRAME: f78d6870 – (.trap 0xfffffffff78d6870)
> ErrCode = 00000000
> eax=00000000 ebx=880cef02 ecx=85bf0b10 edx=87a9af20 esi=86636040
> edi=85bf0b10
> eip=f73d85f1 esp=f78d68e4 ebp=f78d68e4 iopl=0 nv up ei ng nz na pe
> nc
> cs=0008 ss=0010 ds=0023 es=0023 fs=0030 gs=0000
> efl=00010286
> wdf01000!FxDevice::Dispatch+0xb:
> f73d85f1 8b48e8 mov ecx,dword ptr [eax-18h]
> ds:0023:ffffffe8=???
> Resetting default scope
>
> LOCK_ADDRESS: 8055a560 – (!locks 8055a560)
>
> Resource @ nt!IopDeviceTreeLock (0x8055a560) Shared 1 owning threads
> Contention Count = 2
> Threads: 86c20b30-01<*>
> 1 total locks, 1 locks currently held
>
> PNP_TRIAGE:
> Lock address : 0x8055a560
> Thread Count : 1
> Thread address: 0x86c20b30
> Thread wait : 0x256a
>
> LAST_CONTROL_TRANSFER: from f73d85f1 to 805436e0
>
> STACK_TEXT:
> f78d6870 f73d85f1 badb0d00 87a9af20 806624a2 nt!KiTrap0E+0x238
> f78d68e4 804eeec5 85bf0b10 87a9af20 806e4428
> wdf01000!FxDevice::Dispatch+0xb
> f78d68f4 80656128 86ec0f88 85c67afc 880cef20 nt!IopfCallDriver+0x31
> f78d6918 f4cf53c9 f78d693c f4cf5504 85bf0b10 nt!IovCallDriver+0xa0
> WARNING: Stack unwind information not available. Following frames may be
> wrong.
> f78d6920 f4cf5504 85bf0b10 87a9af20 86ec0f94 WUSB_HUB+0x23c9
> f78d693c f4cf6a99 85bf0b10 87a9af20 86ec0f94 WUSB_HUB+0x2504
> f78d6974 f4cfd7d8 85bf0b10 85c67a58 87a9af20 WUSB_HUB+0x3a99
> f78d69a0 f4cfdae5 880cef20 860166b0 8668ef38 WUSB_HUB+0xa7d8
> f78d69b8 f4cf72d3 860166b0 880cef20 f78d69fc WUSB_HUB+0xaae5
> f78d69c8 804eeec5 860166b0 880cef20 806e4428 WUSB_HUB+0x42d3
> f78d69d8 80656128 85c679a8 85c67afc 880cef20 nt!IopfCallDriver+0x31
> f78d69fc f7648c15 85c50001 85c679a8 00000001 nt!IovCallDriver+0xa0
> f78d6a10 f7648d5f 85c679a8 880cef20 85c67afc
> usbhub!USBH_ChangeIndicationQueryChange+0xd5
> f78d6a38 80656330 00000000 86790068 86790000
> usbhub!USBH_ChangeIndication+0x13d
> f78d6a5c 804f13f6 00000000 88264f20 f78d6ac0
> nt!IovpLocalCompletionRoutine+0xb4
> f78d6a8c 806567b8 c0000120 878b0ff0 88264f20 nt!IopfCompleteRequest+0xa2
> f78d6af8 f4cf34a3 85d09bd8 f78d6b2c f4cf6be0 nt!IovCompleteRequest+0x9a
> f78d6b04 f4cf6be0 88264f20 c0000120 00000000 WUSB_HUB+0x4a3
> f78d6b2c f4cf4787 85d090d8 880c6f20 880c6fd8 WUSB_HUB+0x3be0
> f78d6b44 f4cf4c32 85d09020 880c6f20 85d09020 WUSB_HUB+0x1787
> f78d6b70 f4cf72d3 85d09020 880c6f20 f78d6bb4 WUSB_HUB+0x1c32
> f78d6b80 804eeec5 85d09020 880c6f20 806e4428 WUSB_HUB+0x42d3
> f78d6b90 80656128 880c6ffc f78d6c30 880c6f20 nt!IopfCallDriver+0x31
> f78d6bb4 8059181f 85bf0b10 85bf0b10 00000002 nt!IovCallDriver+0xa0
> f78d6be0 80591a81 85d09020 f78d6c0c 00000000 nt!IopSynchronousCall+0xb7
> f78d6c34 804f6ca2 85bf0b10 00000002 00000000 nt!IopRemoveDevice+0x93
> f78d6c5c 8059344a e10ab780 00000018 e114aa30
> nt!IopRemoveLockedDeviceNode+0x160
> f78d6c74 805934b1 85d072c0 00000002 e114aa30
> nt!IopDeleteLockedDeviceNode+0x34
> f78d6ca8 8059911f 85bf0b10 0214aa30 00000002
> nt!IopDeleteLockedDeviceNodes+0x3f
> f78d6d3c 805993e2 f78d6d78 806e4974 e318f1f8
> nt!PiProcessQueryRemoveAndEject+0x76b
> f78d6d58 80599528 f78d6d78 85c71a50 8056375c
> nt!PiProcessTargetDeviceEvent+0x2a
> f78d6d7c 8053776b 85c71a50 00000000 86c20b30 nt!PiWalkDeviceList+0xea
> f78d6dac 805ce7b2 85c71a50 00000000 00000000 nt!ExpWorkerThread+0xef
> f78d6ddc 805450de 8053767c 00000001 00000000
> nt!PspSystemThreadStartup+0x34
> 00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x16
>
> ======================================================
>
> IRP details are
>
> 0: kd> !irp 0x880C6F20
> Irp is active with 3 stacks 3 is current (= 0x880c6fd8)
> No Mdl: No System Buffer: Thread 86c20b30: Irp stack trace.
> cmd flg cl Device File Completion-Context
> [0, 0] 0 10 00000000 00000000 00000000-00000000
>
> Args: 00000000 00000000 00000000 00000000
> [1b, 0] 0 10 85bf0b10 00000000 f4cf73d4-f78d6b00
> \Driver\WUSB_HOST
>
> WUSB_HUB
> Args: 00000000 00000000 00000000 00000000
>
>>[1b, 2] 0 0 85d09020 00000000 00000000-00000000
> \Driver\ WUSB_HUB
> Args: 00000000 00000000 00000000 00000000
>
> 0: kd> !irp 87a9af20
> Irp is active with 3 stacks 3 is current (= 0x87a9afd8)
> No Mdl: No System Buffer: Thread 00000000: Irp stack trace.
> cmd flg cl Device File Completion-Context
> [0, 0] 0 0 00000000 00000000 00000000-00000000
>
> Args: 00000000 00000000 00000000 00000000
> [0, 0] 0 10 00000000 00000000 00000000-00000000
>
> Args: 00000000 00000000 00000000 00000000
>>[f, 0] 0 e0 85bf0b10 00000000 f4cf6852-86ec0f88 Success Error Cancel
> \Driver\ WUSB_HOSTWUSB_HUB
> Args: 86ec0f94 00000000 00220003 00000000
>
> ======================================================
>
> Doron, one more doubt from your previous statement, ?Typically receiving
> io on a deleted pdo is a bug in the caller?. How come the stack still
> active even after deleting the PDO?
>
> \Prasad
>
>