System crash after minifilter unload

I am stress testing my minifilter while Driver Verify is active,
including “Low Resource Simulation”. After several hours of testing, I
can successfully unload my minifilter, but within a few seconds
(sometimes up to about a minute) of unloading, I *sometimes* get a
system crash (debugging info below).

My minifilter is no longer loaded (it appears in the unloaded modules
list) and I can’t seem to find any droppings of it anywhere. It has to
be something related to my minifilter, but I can’t seem to find anything
that will lead me to it…any suggestions would be appreciated…thanks


IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address
at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000004, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000001, value 0 = read operation, 1 = write operation
Arg4: 804dbe9b, address which referenced memory

Debugging Details:

WRITE_ADDRESS: 00000004

CURRENT_IRQL: 2

FAULTING_IP:
nt!KiInsertTimerTable+4e
804dbe9b 894204 mov [edx+0x4],eax

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xA

LAST_CONTROL_TRANSFER: from 8053225b to 804e3592

STACK_TEXT:
8054fb18 8053225b 00000003 8054fe74 00000000
nt!RtlpBreakWithStatusInstruction
8054fb64 80532d2e 00000003 00000004 804dbe9b
nt!KiBugCheckDebugBreak+0x19
8054ff44 804e187f 0000000a 00000004 00000002 nt!KeBugCheck2+0x574
8054ff44 804dbe9b 0000000a 00000004 00000002 nt!KiTrap0E+0x233
8054ffe0 804dbeff fff79b90 ffffffff 98a651c0 nt!KiInsertTimerTable+0x4e
8054fffc 804dc3cb fff79b90 ffffffff f40b91d0 nt!KiInsertTreeTimer+0x7d
8055001c 804dc402 000b91a0 fff79b90 ffffffff nt!KeSetTimerEx+0x4b
80550038 f40af405 f40b91a0 fff79b90 ffffffff nt!KeSetTimer+0x18
80550064 804dc4fd f40b91e0 00000000 ad860f80 rdbss!RxTimerDispatch+0xbe
80550180 804dc378 80558e80 80558c20 ffdff000 nt!KiTimerListExpire+0x122
805501ac 804dbbd4 80559280 00000000 000c2b73 nt!KiTimerExpiration+0xaf
805501d0 804dbb4d 00000000 0000000e 00000000 nt!KiRetireDpcList+0x46
805501d4 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0x26

STACK_COMMAND: kb

FOLLOWUP_IP:
rdbss!RxTimerDispatch+be
f40af405 8b45f4 mov eax,[ebp-0xc]

FAULTING_SOURCE_CODE:

SYMBOL_STACK_INDEX: 8

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: rdbss!RxTimerDispatch+be

MODULE_NAME: rdbss

IMAGE_NAME: rdbss.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 418047d5

FAILURE_BUCKET_ID: 0xA_W_VRF_rdbss!RxTimerDispatch+be

BUCKET_ID: 0xA_W_VRF_rdbss!RxTimerDispatch+be

A very quick glance at that stack suggests (to me at least) that the
timer list has been corrupted. Do you use timer objects in your
mini-filter? In the other crashes you have seen, do you observe a
pattern of timer presence or of rdbss presence? I rate timer list
corruption highest here, but it is too little information to rule out
something going on within the bowels of rdbss.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

Looking forward to seeing you at the next OSR File Systems class in
Boston, MA April 18-21, 2006 (note new date - MS scheduled plugfest the
same week again.)

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Vossen, Joseph (ISS
Atlanta)
Sent: Wednesday, March 15, 2006 7:50 AM
To: ntfsd redirect
Subject: [ntfsd] System crash after minifilter unload

I am stress testing my minifilter while Driver Verify is active,
including “Low Resource Simulation”. After several hours of testing, I
can successfully unload my minifilter, but within a few seconds
(sometimes up to about a minute) of unloading, I *sometimes* get a
system crash (debugging info below).

My minifilter is no longer loaded (it appears in the unloaded modules
list) and I can’t seem to find any droppings of it anywhere. It has to
be something related to my minifilter, but I can’t seem to find anything
that will lead me to it…any suggestions would be appreciated…thanks


IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address
at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000004, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000001, value 0 = read operation, 1 = write operation
Arg4: 804dbe9b, address which referenced memory

Debugging Details:

WRITE_ADDRESS: 00000004

CURRENT_IRQL: 2

FAULTING_IP:
nt!KiInsertTimerTable+4e
804dbe9b 894204 mov [edx+0x4],eax

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xA

LAST_CONTROL_TRANSFER: from 8053225b to 804e3592

STACK_TEXT:
8054fb18 8053225b 00000003 8054fe74 00000000
nt!RtlpBreakWithStatusInstruction
8054fb64 80532d2e 00000003 00000004 804dbe9b
nt!KiBugCheckDebugBreak+0x19
8054ff44 804e187f 0000000a 00000004 00000002 nt!KeBugCheck2+0x574
8054ff44 804dbe9b 0000000a 00000004 00000002 nt!KiTrap0E+0x233
8054ffe0 804dbeff fff79b90 ffffffff 98a651c0 nt!KiInsertTimerTable+0x4e
8054fffc 804dc3cb fff79b90 ffffffff f40b91d0 nt!KiInsertTreeTimer+0x7d
8055001c 804dc402 000b91a0 fff79b90 ffffffff nt!KeSetTimerEx+0x4b
80550038 f40af405 f40b91a0 fff79b90 ffffffff nt!KeSetTimer+0x18
80550064 804dc4fd f40b91e0 00000000 ad860f80 rdbss!RxTimerDispatch+0xbe
80550180 804dc378 80558e80 80558c20 ffdff000 nt!KiTimerListExpire+0x122
805501ac 804dbbd4 80559280 00000000 000c2b73 nt!KiTimerExpiration+0xaf
805501d0 804dbb4d 00000000 0000000e 00000000 nt!KiRetireDpcList+0x46
805501d4 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0x26

STACK_COMMAND: kb

FOLLOWUP_IP:
rdbss!RxTimerDispatch+be
f40af405 8b45f4 mov eax,[ebp-0xc]

FAULTING_SOURCE_CODE:

SYMBOL_STACK_INDEX: 8

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: rdbss!RxTimerDispatch+be

MODULE_NAME: rdbss

IMAGE_NAME: rdbss.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 418047d5

FAILURE_BUCKET_ID: 0xA_W_VRF_rdbss!RxTimerDispatch+be

BUCKET_ID: 0xA_W_VRF_rdbss!RxTimerDispatch+be


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

I do indeed have a timer object (actually several, one per attached
instance). These timer objects are cancelled prior to my minifilter
unloading. I have always assumed that Driver Verifier ensures that
timer objects are cancelled prior to unload, is that true? If so, I
would have thought I would have seen a Driver Verifier error during the
unload.

I’ll go dig through the code paths related to the handling of the
timers.

All of the crashes have the identical stack trace.

Thanks!!

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Wednesday, March 15, 2006 9:23 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] System crash after minifilter unload

A very quick glance at that stack suggests (to me at least)
that the timer list has been corrupted. Do you use timer
objects in your mini-filter? In the other crashes you have
seen, do you observe a pattern of timer presence or of rdbss
presence? I rate timer list corruption highest here, but it
is too little information to rule out something going on
within the bowels of rdbss.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

Looking forward to seeing you at the next OSR File Systems
class in Boston, MA April 18-21, 2006 (note new date - MS
scheduled plugfest the same week again.)

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Vossen,
Joseph (ISS
Atlanta)
Sent: Wednesday, March 15, 2006 7:50 AM
To: ntfsd redirect
Subject: [ntfsd] System crash after minifilter unload

I am stress testing my minifilter while Driver Verify is
active, including “Low Resource Simulation”. After several
hours of testing, I can successfully unload my minifilter,
but within a few seconds (sometimes up to about a minute) of
unloading, I *sometimes* get a system crash (debugging info below).

My minifilter is no longer loaded (it appears in the unloaded modules
list) and I can’t seem to find any droppings of it anywhere.
It has to be something related to my minifilter, but I can’t
seem to find anything that will lead me to it…any
suggestions would be appreciated…thanks


IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely
invalid) address at an interrupt request level (IRQL) that is
too high. This is usually caused by drivers using improper
addresses. If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000004, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000001, value 0 = read operation, 1 = write operation
Arg4: 804dbe9b, address which referenced memory

Debugging Details:

WRITE_ADDRESS: 00000004

CURRENT_IRQL: 2

FAULTING_IP:
nt!KiInsertTimerTable+4e
804dbe9b 894204 mov [edx+0x4],eax

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xA

LAST_CONTROL_TRANSFER: from 8053225b to 804e3592

STACK_TEXT:
8054fb18 8053225b 00000003 8054fe74 00000000
nt!RtlpBreakWithStatusInstruction 8054fb64 80532d2e 00000003
00000004 804dbe9b nt!KiBugCheckDebugBreak+0x19 8054ff44
804e187f 0000000a 00000004 00000002 nt!KeBugCheck2+0x574
8054ff44 804dbe9b 0000000a 00000004 00000002
nt!KiTrap0E+0x233 8054ffe0 804dbeff fff79b90 ffffffff
98a651c0 nt!KiInsertTimerTable+0x4e 8054fffc 804dc3cb
fff79b90 ffffffff f40b91d0 nt!KiInsertTreeTimer+0x7d 8055001c
804dc402 000b91a0 fff79b90 ffffffff nt!KeSetTimerEx+0x4b
80550038 f40af405 f40b91a0 fff79b90 ffffffff
nt!KeSetTimer+0x18 80550064 804dc4fd f40b91e0 00000000
ad860f80 rdbss!RxTimerDispatch+0xbe 80550180 804dc378
80558e80 80558c20 ffdff000 nt!KiTimerListExpire+0x122
805501ac 804dbbd4 80559280 00000000 000c2b73
nt!KiTimerExpiration+0xaf 805501d0 804dbb4d 00000000 0000000e
00000000 nt!KiRetireDpcList+0x46 805501d4 00000000 0000000e
00000000 00000000 nt!KiIdleLoop+0x26

STACK_COMMAND: kb

FOLLOWUP_IP:
rdbss!RxTimerDispatch+be
f40af405 8b45f4 mov eax,[ebp-0xc]

FAULTING_SOURCE_CODE:

SYMBOL_STACK_INDEX: 8

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: rdbss!RxTimerDispatch+be

MODULE_NAME: rdbss

IMAGE_NAME: rdbss.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 418047d5

FAILURE_BUCKET_ID: 0xA_W_VRF_rdbss!RxTimerDispatch+be

BUCKET_ID: 0xA_W_VRF_rdbss!RxTimerDispatch+be


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag
argument: ‘’ To unsubscribe send a blank email to
xxxxx@lists.osr.com


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag
argument: ‘’ To unsubscribe send a blank email to
xxxxx@lists.osr.com

Timer objects are a normal example of “use this, make your driver
unloadable” - this is because when the timer expires, it is placed on a
private list inside a DPC routine that then processes the expired
timers. Cancelling the timer doesn’t remove the timer from this list
(because it is private to the DPC function executing the timer). As odd
as it may seem, if the timer cancellation function indicates it DID
cancel the timer, you are safe. If it indicated it did NOT cancel the
timer, you can be in this state.

This is hinted at in the KeCancelTimer documentation (the cryptic
comments about “… returns FALSE if the timer DPC has begun executing”
and then later when it says “The driver must ensure that the DPC has
completed before freeing any resources used by the DPC.”) I can’t see
how you make this work right as part of unload (I can see how to make
the window vanishingly small, but MP systems sort of screw things up
here…)

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

Looking forward to seeing you at the next OSR File Systems class in
Boston, MA April 18-21, 2006 (note new date - MS scheduled plugfest the
same week again.)

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Vossen, Joseph (ISS
Atlanta)
Sent: Wednesday, March 15, 2006 9:44 AM
To: ntfsd redirect
Subject: RE: [ntfsd] System crash after minifilter unload

I do indeed have a timer object (actually several, one per attached
instance). These timer objects are cancelled prior to my minifilter
unloading. I have always assumed that Driver Verifier ensures that
timer objects are cancelled prior to unload, is that true? If so, I
would have thought I would have seen a Driver Verifier error during the
unload.

I’ll go dig through the code paths related to the handling of the
timers.

All of the crashes have the identical stack trace.

Thanks!!

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Wednesday, March 15, 2006 9:23 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] System crash after minifilter unload

A very quick glance at that stack suggests (to me at least)
that the timer list has been corrupted. Do you use timer
objects in your mini-filter? In the other crashes you have
seen, do you observe a pattern of timer presence or of rdbss
presence? I rate timer list corruption highest here, but it
is too little information to rule out something going on
within the bowels of rdbss.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

Looking forward to seeing you at the next OSR File Systems
class in Boston, MA April 18-21, 2006 (note new date - MS
scheduled plugfest the same week again.)

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Vossen,
Joseph (ISS
Atlanta)
Sent: Wednesday, March 15, 2006 7:50 AM
To: ntfsd redirect
Subject: [ntfsd] System crash after minifilter unload

I am stress testing my minifilter while Driver Verify is
active, including “Low Resource Simulation”. After several
hours of testing, I can successfully unload my minifilter,
but within a few seconds (sometimes up to about a minute) of
unloading, I *sometimes* get a system crash (debugging info below).

My minifilter is no longer loaded (it appears in the unloaded modules
list) and I can’t seem to find any droppings of it anywhere.
It has to be something related to my minifilter, but I can’t
seem to find anything that will lead me to it…any
suggestions would be appreciated…thanks


IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely
invalid) address at an interrupt request level (IRQL) that is
too high. This is usually caused by drivers using improper
addresses. If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000004, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000001, value 0 = read operation, 1 = write operation
Arg4: 804dbe9b, address which referenced memory

Debugging Details:

WRITE_ADDRESS: 00000004

CURRENT_IRQL: 2

FAULTING_IP:
nt!KiInsertTimerTable+4e
804dbe9b 894204 mov [edx+0x4],eax

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xA

LAST_CONTROL_TRANSFER: from 8053225b to 804e3592

STACK_TEXT:
8054fb18 8053225b 00000003 8054fe74 00000000
nt!RtlpBreakWithStatusInstruction 8054fb64 80532d2e 00000003
00000004 804dbe9b nt!KiBugCheckDebugBreak+0x19 8054ff44
804e187f 0000000a 00000004 00000002 nt!KeBugCheck2+0x574
8054ff44 804dbe9b 0000000a 00000004 00000002
nt!KiTrap0E+0x233 8054ffe0 804dbeff fff79b90 ffffffff
98a651c0 nt!KiInsertTimerTable+0x4e 8054fffc 804dc3cb
fff79b90 ffffffff f40b91d0 nt!KiInsertTreeTimer+0x7d 8055001c
804dc402 000b91a0 fff79b90 ffffffff nt!KeSetTimerEx+0x4b
80550038 f40af405 f40b91a0 fff79b90 ffffffff
nt!KeSetTimer+0x18 80550064 804dc4fd f40b91e0 00000000
ad860f80 rdbss!RxTimerDispatch+0xbe 80550180 804dc378
80558e80 80558c20 ffdff000 nt!KiTimerListExpire+0x122
805501ac 804dbbd4 80559280 00000000 000c2b73
nt!KiTimerExpiration+0xaf 805501d0 804dbb4d 00000000 0000000e
00000000 nt!KiRetireDpcList+0x46 805501d4 00000000 0000000e
00000000 00000000 nt!KiIdleLoop+0x26

STACK_COMMAND: kb

FOLLOWUP_IP:
rdbss!RxTimerDispatch+be
f40af405 8b45f4 mov eax,[ebp-0xc]

FAULTING_SOURCE_CODE:

SYMBOL_STACK_INDEX: 8

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: rdbss!RxTimerDispatch+be

MODULE_NAME: rdbss

IMAGE_NAME: rdbss.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 418047d5

FAILURE_BUCKET_ID: 0xA_W_VRF_rdbss!RxTimerDispatch+be

BUCKET_ID: 0xA_W_VRF_rdbss!RxTimerDispatch+be


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag
argument: ‘’ To unsubscribe send a blank email to
xxxxx@lists.osr.com


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag
argument: ‘’ To unsubscribe send a blank email to
xxxxx@lists.osr.com


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Well…yuck…

Is that behavior (as you described) still true if the ‘dpc’ parameter
when calling KeSetTimer() is set to NULL?

thanks

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Wednesday, March 15, 2006 10:23 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] System crash after minifilter unload

Timer objects are a normal example of “use this, make your
driver unloadable” - this is because when the timer expires,
it is placed on a private list inside a DPC routine that then
processes the expired timers. Cancelling the timer doesn’t
remove the timer from this list (because it is private to the
DPC function executing the timer). As odd as it may seem, if
the timer cancellation function indicates it DID cancel the
timer, you are safe. If it indicated it did NOT cancel the
timer, you can be in this state.

This is hinted at in the KeCancelTimer documentation (the
cryptic comments about “… returns FALSE if the timer DPC
has begun executing” and then later when it says “The driver
must ensure that the DPC has completed before freeing any
resources used by the DPC.”) I can’t see how you make this
work right as part of unload (I can see how to make the
window vanishingly small, but MP systems sort of screw things up
here…)

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

Looking forward to seeing you at the next OSR File Systems
class in Boston, MA April 18-21, 2006 (note new date - MS
scheduled plugfest the same week again.)

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Vossen,
Joseph (ISS
Atlanta)
Sent: Wednesday, March 15, 2006 9:44 AM
To: ntfsd redirect
Subject: RE: [ntfsd] System crash after minifilter unload

I do indeed have a timer object (actually several, one per
attached instance). These timer objects are cancelled prior
to my minifilter unloading. I have always assumed that
Driver Verifier ensures that timer objects are cancelled
prior to unload, is that true? If so, I would have thought I
would have seen a Driver Verifier error during the unload.

I’ll go dig through the code paths related to the handling of
the timers.

All of the crashes have the identical stack trace.

Thanks!!

> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
> Sent: Wednesday, March 15, 2006 9:23 AM
> To: Windows File Systems Devs Interest List
> Subject: RE: [ntfsd] System crash after minifilter unload
>
>
> A very quick glance at that stack suggests (to me at least)
> that the timer list has been corrupted. Do you use timer
> objects in your mini-filter? In the other crashes you have
> seen, do you observe a pattern of timer presence or of rdbss
> presence? I rate timer list corruption highest here, but it
> is too little information to rule out something going on
> within the bowels of rdbss.
>
> Regards,
>
> Tony

[snip]

There’s two DPCs involved here:

  • The generic OS DPC that manages timer expiration; the private list is
    in this routine as I understand it;
  • The optional DPC function that you can specify.

I believe this is related to the implementation model: timers are
maintained on a delta queue. When they expire the clock interrupt logic
notices and moves this to the timer expiration queue (that annoying
private list.) The timer expiration DPC then runs later (after the ISR)
and will call the DPC function (if there is one) or change the timer
state (which shouldn’t be done at elevated IRQL.)

Thus, cancelling the timer removes it from the delta queue, but
processing it after expiration is the real problem.

In all fairness, I’m doing this from my recollection of the details of
this issue. I generally just “know” that timer objects are not safely
used in unloadable driver, but I’m dredging my memory for details here
(and implementations DO change).

Bottom line: I believe the issue you are observing is likely related to
those timer objects. If you have other crashes and the driver involved
changes over time but the actual OS code remains the same (ergo, timer
list manipulation) I’d consider that to be rather strong evidence.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

Looking forward to seeing you at the next OSR File Systems class in
Boston, MA April 18-21, 2006 (note new date - MS scheduled plugfest the
same week again.)

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Vossen, Joseph (ISS
Atlanta)
Sent: Wednesday, March 15, 2006 10:45 AM
To: ntfsd redirect
Subject: RE: [ntfsd] System crash after minifilter unload

Well…yuck…

Is that behavior (as you described) still true if the ‘dpc’ parameter
when calling KeSetTimer() is set to NULL?

thanks

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Wednesday, March 15, 2006 10:23 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] System crash after minifilter unload

Timer objects are a normal example of “use this, make your
driver unloadable” - this is because when the timer expires,
it is placed on a private list inside a DPC routine that then
processes the expired timers. Cancelling the timer doesn’t
remove the timer from this list (because it is private to the
DPC function executing the timer). As odd as it may seem, if
the timer cancellation function indicates it DID cancel the
timer, you are safe. If it indicated it did NOT cancel the
timer, you can be in this state.

This is hinted at in the KeCancelTimer documentation (the
cryptic comments about “… returns FALSE if the timer DPC
has begun executing” and then later when it says “The driver
must ensure that the DPC has completed before freeing any
resources used by the DPC.”) I can’t see how you make this
work right as part of unload (I can see how to make the
window vanishingly small, but MP systems sort of screw things up
here…)

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

Looking forward to seeing you at the next OSR File Systems
class in Boston, MA April 18-21, 2006 (note new date - MS
scheduled plugfest the same week again.)

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Vossen,
Joseph (ISS
Atlanta)
Sent: Wednesday, March 15, 2006 9:44 AM
To: ntfsd redirect
Subject: RE: [ntfsd] System crash after minifilter unload

I do indeed have a timer object (actually several, one per
attached instance). These timer objects are cancelled prior
to my minifilter unloading. I have always assumed that
Driver Verifier ensures that timer objects are cancelled
prior to unload, is that true? If so, I would have thought I
would have seen a Driver Verifier error during the unload.

I’ll go dig through the code paths related to the handling of
the timers.

All of the crashes have the identical stack trace.

Thanks!!

> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
> Sent: Wednesday, March 15, 2006 9:23 AM
> To: Windows File Systems Devs Interest List
> Subject: RE: [ntfsd] System crash after minifilter unload
>
>
> A very quick glance at that stack suggests (to me at least)
> that the timer list has been corrupted. Do you use timer
> objects in your mini-filter? In the other crashes you have
> seen, do you observe a pattern of timer presence or of rdbss
> presence? I rate timer list corruption highest here, but it
> is too little information to rule out something going on
> within the bowels of rdbss.
>
> Regards,
>
> Tony

[snip]


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Thanks for that insight…I’ll revisit that logic and see what I can do
with it.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Wednesday, March 15, 2006 12:45 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] System crash after minifilter unload

There’s two DPCs involved here:

  • The generic OS DPC that manages timer expiration; the
    private list is in this routine as I understand it;
  • The optional DPC function that you can specify.

I believe this is related to the implementation model:
timers are maintained on a delta queue. When they expire the
clock interrupt logic notices and moves this to the timer
expiration queue (that annoying private list.) The timer
expiration DPC then runs later (after the ISR) and will call
the DPC function (if there is one) or change the timer state
(which shouldn’t be done at elevated IRQL.)

Thus, cancelling the timer removes it from the delta queue,
but processing it after expiration is the real problem.

In all fairness, I’m doing this from my recollection of the
details of this issue. I generally just “know” that timer
objects are not safely used in unloadable driver, but I’m
dredging my memory for details here (and implementations DO change).

Bottom line: I believe the issue you are observing is likely
related to those timer objects. If you have other crashes
and the driver involved changes over time but the actual OS
code remains the same (ergo, timer list manipulation) I’d
consider that to be rather strong evidence.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

Looking forward to seeing you at the next OSR File Systems
class in Boston, MA April 18-21, 2006 (note new date - MS
scheduled plugfest the same week again.)

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Vossen,
Joseph (ISS
Atlanta)
Sent: Wednesday, March 15, 2006 10:45 AM
To: ntfsd redirect
Subject: RE: [ntfsd] System crash after minifilter unload

Well…yuck…

Is that behavior (as you described) still true if the ‘dpc’
parameter when calling KeSetTimer() is set to NULL?

thanks

> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
> Sent: Wednesday, March 15, 2006 10:23 AM
> To: Windows File Systems Devs Interest List
> Subject: RE: [ntfsd] System crash after minifilter unload
>
>
> Timer objects are a normal example of “use this, make your
> driver unloadable” - this is because when the timer expires,
> it is placed on a private list inside a DPC routine that then
> processes the expired timers. Cancelling the timer doesn’t
> remove the timer from this list (because it is private to the
> DPC function executing the timer). As odd as it may seem, if
> the timer cancellation function indicates it DID cancel the
> timer, you are safe. If it indicated it did NOT cancel the
> timer, you can be in this state.
>
> This is hinted at in the KeCancelTimer documentation (the
> cryptic comments about “… returns FALSE if the timer DPC
> has begun executing” and then later when it says “The driver
> must ensure that the DPC has completed before freeing any
> resources used by the DPC.”) I can’t see how you make this
> work right as part of unload (I can see how to make the
> window vanishingly small, but MP systems sort of screw things up
> here…)
>
> Regards,
>
> Tony
>
> Tony Mason
> Consulting Partner
> OSR Open Systems Resources, Inc.
> http://www.osr.com
>
>
> Looking forward to seeing you at the next OSR File Systems
> class in Boston, MA April 18-21, 2006 (note new date - MS
> scheduled plugfest the same week again.)
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Vossen,
> Joseph (ISS
> Atlanta)
> Sent: Wednesday, March 15, 2006 9:44 AM
> To: ntfsd redirect
> Subject: RE: [ntfsd] System crash after minifilter unload
>
> I do indeed have a timer object (actually several, one per
> attached instance). These timer objects are cancelled prior
> to my minifilter unloading. I have always assumed that
> Driver Verifier ensures that timer objects are cancelled
> prior to unload, is that true? If so, I would have thought I
> would have seen a Driver Verifier error during the unload.
>
> I’ll go dig through the code paths related to the handling of
> the timers.
>
> All of the crashes have the identical stack trace.
>
> Thanks!!
>
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
> > Sent: Wednesday, March 15, 2006 9:23 AM
> > To: Windows File Systems Devs Interest List
> > Subject: RE: [ntfsd] System crash after minifilter unload
> >
> >
> > A very quick glance at that stack suggests (to me at
least) that the
> > timer list has been corrupted. Do you use timer objects in your
> > mini-filter? In the other crashes you have seen, do you
observe a
> > pattern of timer presence or of rdbss presence? I rate
timer list
> > corruption highest here, but it is too little information to rule
> > out something going on within the bowels of rdbss.
> >
> > Regards,
> >
> > Tony

[snip]


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag
argument: ‘’ To unsubscribe send a blank email to
xxxxx@lists.osr.com


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag
argument: ‘’ To unsubscribe send a blank email to
xxxxx@lists.osr.com

Are you cancelling all your timers properly on unload?

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Vossen, Joseph (ISS Atlanta)”
To: “Windows File Systems Devs Interest List”
Sent: Wednesday, March 15, 2006 3:49 PM
Subject: [ntfsd] System crash after minifilter unload

I am stress testing my minifilter while Driver Verify is active,
including “Low Resource Simulation”. After several hours of testing, I
can successfully unload my minifilter, but within a few seconds
(sometimes up to about a minute) of unloading, I sometimes get a
system crash (debugging info below).

My minifilter is no longer loaded (it appears in the unloaded modules
list) and I can’t seem to find any droppings of it anywhere. It has to
be something related to my minifilter, but I can’t seem to find anything
that will lead me to it…any suggestions would be appreciated…thanks

- - - - - - - - - - - - - - - - - - -

IRQL_NOT_LESS_OR_EQUAL (a)
An attempt was made to access a pageable (or completely invalid) address
at an
interrupt request level (IRQL) that is too high. This is usually
caused by drivers using improper addresses.
If a kernel debugger is available get the stack backtrace.
Arguments:
Arg1: 00000004, memory referenced
Arg2: 00000002, IRQL
Arg3: 00000001, value 0 = read operation, 1 = write operation
Arg4: 804dbe9b, address which referenced memory

Debugging Details:
------------------

WRITE_ADDRESS: 00000004

CURRENT_IRQL: 2

FAULTING_IP:
nt!KiInsertTimerTable+4e
804dbe9b 894204 mov [edx+0x4],eax

DEFAULT_BUCKET_ID: DRIVER_FAULT

BUGCHECK_STR: 0xA

LAST_CONTROL_TRANSFER: from 8053225b to 804e3592

STACK_TEXT:
8054fb18 8053225b 00000003 8054fe74 00000000
nt!RtlpBreakWithStatusInstruction
8054fb64 80532d2e 00000003 00000004 804dbe9b
nt!KiBugCheckDebugBreak+0x19
8054ff44 804e187f 0000000a 00000004 00000002 nt!KeBugCheck2+0x574
8054ff44 804dbe9b 0000000a 00000004 00000002 nt!KiTrap0E+0x233
8054ffe0 804dbeff fff79b90 ffffffff 98a651c0 nt!KiInsertTimerTable+0x4e
8054fffc 804dc3cb fff79b90 ffffffff f40b91d0 nt!KiInsertTreeTimer+0x7d
8055001c 804dc402 000b91a0 fff79b90 ffffffff nt!KeSetTimerEx+0x4b
80550038 f40af405 f40b91a0 fff79b90 ffffffff nt!KeSetTimer+0x18
80550064 804dc4fd f40b91e0 00000000 ad860f80 rdbss!RxTimerDispatch+0xbe
80550180 804dc378 80558e80 80558c20 ffdff000 nt!KiTimerListExpire+0x122
805501ac 804dbbd4 80559280 00000000 000c2b73 nt!KiTimerExpiration+0xaf
805501d0 804dbb4d 00000000 0000000e 00000000 nt!KiRetireDpcList+0x46
805501d4 00000000 0000000e 00000000 00000000 nt!KiIdleLoop+0x26

STACK_COMMAND: kb

FOLLOWUP_IP:
rdbss!RxTimerDispatch+be
f40af405 8b45f4 mov eax,[ebp-0xc]

FAULTING_SOURCE_CODE:

SYMBOL_STACK_INDEX: 8

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: rdbss!RxTimerDispatch+be

MODULE_NAME: rdbss

IMAGE_NAME: rdbss.sys

DEBUG_FLR_IMAGE_TIMESTAMP: 418047d5

FAILURE_BUCKET_ID: 0xA_W_VRF_rdbss!RxTimerDispatch+be

BUCKET_ID: 0xA_W_VRF_rdbss!RxTimerDispatch+be


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Vossen,

This is a known issue with minifilters. Making timers unload safe is one
of the top items on our filter manager to do list.

Neal Christiansen
Microsoft File System Filter Group Lead
This posting is provided “AS IS” with no warranties, and confers no
Rights

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Vossen, Joseph
(ISS Atlanta)
Sent: Wednesday, March 15, 2006 10:00 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] System crash after minifilter unload

Thanks for that insight…I’ll revisit that logic and see what I can do
with it.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Wednesday, March 15, 2006 12:45 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] System crash after minifilter unload

There’s two DPCs involved here:

  • The generic OS DPC that manages timer expiration; the
    private list is in this routine as I understand it;
  • The optional DPC function that you can specify.

I believe this is related to the implementation model:
timers are maintained on a delta queue. When they expire the
clock interrupt logic notices and moves this to the timer
expiration queue (that annoying private list.) The timer
expiration DPC then runs later (after the ISR) and will call
the DPC function (if there is one) or change the timer state
(which shouldn’t be done at elevated IRQL.)

Thus, cancelling the timer removes it from the delta queue,
but processing it after expiration is the real problem.

In all fairness, I’m doing this from my recollection of the
details of this issue. I generally just “know” that timer
objects are not safely used in unloadable driver, but I’m
dredging my memory for details here (and implementations DO change).

Bottom line: I believe the issue you are observing is likely
related to those timer objects. If you have other crashes
and the driver involved changes over time but the actual OS
code remains the same (ergo, timer list manipulation) I’d
consider that to be rather strong evidence.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

Looking forward to seeing you at the next OSR File Systems
class in Boston, MA April 18-21, 2006 (note new date - MS
scheduled plugfest the same week again.)

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Vossen,
Joseph (ISS
Atlanta)
Sent: Wednesday, March 15, 2006 10:45 AM
To: ntfsd redirect
Subject: RE: [ntfsd] System crash after minifilter unload

Well…yuck…

Is that behavior (as you described) still true if the ‘dpc’
parameter when calling KeSetTimer() is set to NULL?

thanks

> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
> Sent: Wednesday, March 15, 2006 10:23 AM
> To: Windows File Systems Devs Interest List
> Subject: RE: [ntfsd] System crash after minifilter unload
>
>
> Timer objects are a normal example of “use this, make your
> driver unloadable” - this is because when the timer expires,
> it is placed on a private list inside a DPC routine that then
> processes the expired timers. Cancelling the timer doesn’t
> remove the timer from this list (because it is private to the
> DPC function executing the timer). As odd as it may seem, if
> the timer cancellation function indicates it DID cancel the
> timer, you are safe. If it indicated it did NOT cancel the
> timer, you can be in this state.
>
> This is hinted at in the KeCancelTimer documentation (the
> cryptic comments about “… returns FALSE if the timer DPC
> has begun executing” and then later when it says “The driver
> must ensure that the DPC has completed before freeing any
> resources used by the DPC.”) I can’t see how you make this
> work right as part of unload (I can see how to make the
> window vanishingly small, but MP systems sort of screw things up
> here…)
>
> Regards,
>
> Tony
>
> Tony Mason
> Consulting Partner
> OSR Open Systems Resources, Inc.
> http://www.osr.com
>
>
> Looking forward to seeing you at the next OSR File Systems
> class in Boston, MA April 18-21, 2006 (note new date - MS
> scheduled plugfest the same week again.)
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Vossen,
> Joseph (ISS
> Atlanta)
> Sent: Wednesday, March 15, 2006 9:44 AM
> To: ntfsd redirect
> Subject: RE: [ntfsd] System crash after minifilter unload
>
> I do indeed have a timer object (actually several, one per
> attached instance). These timer objects are cancelled prior
> to my minifilter unloading. I have always assumed that
> Driver Verifier ensures that timer objects are cancelled
> prior to unload, is that true? If so, I would have thought I
> would have seen a Driver Verifier error during the unload.
>
> I’ll go dig through the code paths related to the handling of
> the timers.
>
> All of the crashes have the identical stack trace.
>
> Thanks!!
>
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
> > Sent: Wednesday, March 15, 2006 9:23 AM
> > To: Windows File Systems Devs Interest List
> > Subject: RE: [ntfsd] System crash after minifilter unload
> >
> >
> > A very quick glance at that stack suggests (to me at
least) that the
> > timer list has been corrupted. Do you use timer objects in your
> > mini-filter? In the other crashes you have seen, do you
observe a
> > pattern of timer presence or of rdbss presence? I rate
timer list
> > corruption highest here, but it is too little information to rule
> > out something going on within the bowels of rdbss.
> >
> > Regards,
> >
> > Tony

[snip]


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag
argument: ‘’ To unsubscribe send a blank email to
xxxxx@lists.osr.com


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag
argument: ‘’ To unsubscribe send a blank email to
xxxxx@lists.osr.com


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com