Server 2008 + verifier + FSD issue

NTFSD
1

I’m attempting to determine the cause of a verifier induced crash in the
development version of the OpenAFS Windows redirector. I have an
hypothesis about the crash and would like to see of other people think
it is a reasonable explanition.

Configuration:

Standard Server 2008
Verifier monitoring the driver in question
Tried VMware & real hardware: no difference
Tried 32 & 64-bit: no difference

Test:

Start OpenAFS
Connect to a cell
Write a 25 Mbyte file to the cell
Boom

Without verifier, all indications are that the operation completed
without fault.

Crash excerpt:

DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught.
This is
because the driver was specified in the registry as being suspect
(by the

Arguments:
Arg1: 000000c5, Thread APC disable count changed by driver dispatch
routine.
Arg2: 95bc7df0, Driver dispatch routine address.
Arg3: 00000000, Current thread APC disable count.
Arg4: 0000ffff, Thread APC disable count before calling driver
dispatch routine.
The APC disable count is decremented each time a driver calls
KeEnterCriticalRegion, KeInitializeMutex, or
FsRtlEnterFileSystem. The APC
disable count is incremented each time a driver calls
KeLeaveCriticalRegion,
KeReleaseMutex, or FsRtlExitFileSystem. Since these calls
should always be in
pairs, this value should be zero when a thread exits. A
negative value
indicates that a driver has disabled APC calls without
re-enabling them. A
positive value indicates that the reverse is true.

FAULTING_SOURCE_CODE:
70: //
71: NTSTATUS
72: AFSWrite( IN PDEVICE_OBJECT DeviceObject,
73: IN PIRP Irp)
> 74: {
75: return AFSCommonWrite(DeviceObject, Irp, NULL);
76: }
77:
78: NTSTATUS
79: AFSCommonWrite( IN PDEVICE_OBJECT DeviceObject,

The driver supports callbacks for CM AcquireForLazyWrite and Fast I/O
AcquireForModWrite.

It does not support Fast I/O write. The FastIoWrite callback is
implemented but returns FALSE. From a performance point of view, this is
not an issue. I/O Manager frequently attempts Fast I/O but is forced to
generate an IRP.

When either AcquireForLazyWrite or AcquireForModWrite is called, FCB
resource(s) are acquired as intended. Since the
ExAcquireResource…Lite functions have KeEnterCriticalRegion as a
prerequsite, that is called first. KeEnterCriticalRegion turns off APCs.

As far as I can determine, this action is immediately followed by an
IRP_MJ_WRITE IRP and verifier crashes the system because ACPs are off.

Put differently, every time an IRP_MJ_WRITE IRP occurs with APCs
disabled, the preceeding activity was a call to either
AcquireForLazyWrite or AcquireForModWrite.

Questions:

Am I missing something?

If Fast I/O write was implemented, the I/O Manager would probably use
that in place of the IRP. Would verifier still trigger because APCs are off?

Would the newer version of Server 2008 (with presumably newer verifier)
behave differently? (I suspect no but it doesn’t hurt to ask.)

Thanks,
Mickey Lane

2

What is the call stack? probably the dispatch routine have called KeEnterCriticalRegion and forgot to undo it.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

“Mickey Lane” wrote in message news:xxxxx@ntfsd…
> I’m attempting to determine the cause of a verifier induced crash in the
> development version of the OpenAFS Windows redirector. I have an
> hypothesis about the crash and would like to see of other people think
> it is a reasonable explanition.
>
>
> Configuration:
>
> Standard Server 2008
> Verifier monitoring the driver in question
> Tried VMware & real hardware: no difference
> Tried 32 & 64-bit: no difference
>
>
> Test:
>
> Start OpenAFS
> Connect to a cell
> Write a 25 Mbyte file to the cell
> Boom
>
> Without verifier, all indications are that the operation completed
> without fault.
>
>
> Crash excerpt:
>
> DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
> A device driver attempting to corrupt the system has been caught.
> This is
> because the driver was specified in the registry as being suspect
> (by the
>
> Arguments:
> Arg1: 000000c5, Thread APC disable count changed by driver dispatch
> routine.
> Arg2: 95bc7df0, Driver dispatch routine address.
> Arg3: 00000000, Current thread APC disable count.
> Arg4: 0000ffff, Thread APC disable count before calling driver
> dispatch routine.
> The APC disable count is decremented each time a driver calls
> KeEnterCriticalRegion, KeInitializeMutex, or
> FsRtlEnterFileSystem. The APC
> disable count is incremented each time a driver calls
> KeLeaveCriticalRegion,
> KeReleaseMutex, or FsRtlExitFileSystem. Since these calls
> should always be in
> pairs, this value should be zero when a thread exits. A
> negative value
> indicates that a driver has disabled APC calls without
> re-enabling them. A
> positive value indicates that the reverse is true.
>
> FAULTING_SOURCE_CODE:
> 70: //
> 71: NTSTATUS
> 72: AFSWrite( IN PDEVICE_OBJECT DeviceObject,
> 73: IN PIRP Irp)
> > 74: {
> 75: return AFSCommonWrite(DeviceObject, Irp, NULL);
> 76: }
> 77:
> 78: NTSTATUS
> 79: AFSCommonWrite( IN PDEVICE_OBJECT DeviceObject,
>
>
> The driver supports callbacks for CM AcquireForLazyWrite and Fast I/O
> AcquireForModWrite.
>
> It does not support Fast I/O write. The FastIoWrite callback is
> implemented but returns FALSE. From a performance point of view, this is
> not an issue. I/O Manager frequently attempts Fast I/O but is forced to
> generate an IRP.
>
> When either AcquireForLazyWrite or AcquireForModWrite is called, FCB
> resource(s) are acquired as intended. Since the
> ExAcquireResource…Lite functions have KeEnterCriticalRegion as a
> prerequsite, that is called first. KeEnterCriticalRegion turns off APCs.
>
> As far as I can determine, this action is immediately followed by an
> IRP_MJ_WRITE IRP and verifier crashes the system because ACPs are off.
>
> Put differently, every time an IRP_MJ_WRITE IRP occurs with APCs
> disabled, the preceeding activity was a call to either
> AcquireForLazyWrite or AcquireForModWrite.
>
> Questions:
>
> Am I missing something?
>
> If Fast I/O write was implemented, the I/O Manager would probably use
> that in place of the IRP. Would verifier still trigger because APCs are off?
>
> Would the newer version of Server 2008 (with presumably newer verifier)
> behave differently? (I suspect no but it doesn’t hurt to ask.)
>
> Thanks,
> Mickey Lane
>
>
>

3

Maxim S. Shatskih wrote:

What is the call stack? probably the dispatch routine have called KeEnterCriticalRegion and forgot to undo it.

I’m pretty sure there are cases where an enter was not properly followed
by an exit but for the current problem, I don’t think those cases are
playing a part. The top level activity (command line “copy
”) breaks down into a lot (dozens) of IRP_MJ_WRITEs and if my
measuring technique is accurate, all of them function as expected. In
every case where the dispatch routine is entered with APCs off, it’s due
to the activity of AcquireForLazyWrite or AcquireForModWrite - and these
routines are doing exactly what they’re supposed to be doing.

4

Verifier & the Modwriter have an interesting interaction in Vista and beyond
which bites drivers which habitually decrement the APC count when they take
a resource and increment it when they release it. If
AcquireForModWrite/ReleaseForModWrite does this then when the Modwriter
comes along you can get a bogus bugcheck from verifier.

What happens is that Modwriter fires off a write, if the FSD can complete
the IRP immediately then it does. The IRP completion then calls
ReleaseForModWrite. ReleaseForModWrite drops a lock and increments the APC
count. The FSD write dispatch function then exits and verifier spots that
the APC count has gone wrong.

Nobody has done anything wrong, but verifier spots something which is weird.
Because of this I tend to view this as a bug in verifier, although the
verifier people tend to disagree. I usually put in special case code to
change the behavior of AcquireForModWrite/ReleaseForModWrite in the presence
of the verifier.

You mention the lazy writer below - it may well have the same behavior, but
I’ve never seen it.

Rod Widdowson

“Mickey Lane” wrote in message
news:xxxxx@ntfsd…
> Maxim S. Shatskih wrote:
>> What is the call stack? probably the dispatch routine have called
>> KeEnterCriticalRegion and forgot to undo it.
>>
>>
> I’m pretty sure there are cases where an enter was not properly followed
> by an exit but for the current problem, I don’t think those cases are
> playing a part. The top level activity (command line “copy ”)
> breaks down into a lot (dozens) of IRP_MJ_WRITEs and if my measuring
> technique is accurate, all of them function as expected. In every case
> where the dispatch routine is entered with APCs off, it’s due to the
> activity of AcquireForLazyWrite or AcquireForModWrite - and these routines
> are doing exactly what they’re supposed to be doing.
>
>

5

Rod Widdowson wrote:

Verifier & the Modwriter have an interesting interaction in Vista and
beyond which bites drivers which habitually decrement the APC count
> when they take a resource and increment it when they release it.

[…]

Nobody has done anything wrong, but verifier spots something which is
weird. Because of this I tend to view this as a bug in verifier,
> although the verifier people tend to disagree.

Does verifier have different rules for IRP vs. Fast I/O?

I would hazard a guess that most (all?) FSDs implement Fast I/O read
and write for performance reasons. Fast I/O probably isn’t needed
on OpenAFS but if it’ll make this issue go away, it might be worth
looking into.

Thanks for addressing this,
Mickey.

6

Mm will not use fast I/O functions. Thus, if this is coming from the
lazy writer or mapped page writer, you will not see them as fast I/O
operations, since these I/O operations must be done with IRPs (they
aren’t against the cache - which is what fast I/O does.)

We use a nasty little debugging trick of looking at the APC index count
before and after calls to track this sort of thing down (it’s nasty
because there’s no exposed function for obtaining this value. We
“discover” it dynamically when we start running and then just use that
offset to grab the value.) Debugging APC index mismatch issues is
painful, which is why we have that debug code lying around.

Tony
OSR

7

Guys, I work on Verifier and this is the first time I heard about this kind of problem. I will look into it and get back to you.

When you need help or you have any other feedback related to Verifier, feel free to send email to us using verifier @ microsoft.com. That way we will have a better chance to see your message, rather than overlooking it on a Forum.

Thanks,
Dan

8

I have been discussing these issues with folks who work on the Cache Manager and Memory Manager. We concluded that AcquireForLazyWrite and AcquireForModWrite routines don’t need to disable normal kernel APCs. Disabling APCs before acquiring an ERESOURCE is needed to prevent an application sending a Suspend APC and suspending the thread that owns the lock. That scenario cannot happen for those ERESOURCEs acquired by AcquireForLazyWrite or AcquireForModWrite, because these routines can get called just in one of these two circumstances:

  1. With APCs already disabled by the Cache Manager.
  2. In the context of a System thread, that apps cannot suspend.

Bottom line is that the kernel is responsible for ensuring that APCs are properly disabled before it calls any of AcquireForLazyWrite or AcquireForModWrite, and responsible to re-enable APCs only after the corresponding Release routines have been called. The Cache Manager folks will clarify this behavior in the documentation that describes AcquireForLazyWrite. AcquireForModWrite seems to be deprecated, so probably there won’t be a doc update for it.

So we recommend removing the KeEnterCriticalRegion calls from AcquireForLazyWrite or AcquireForModWrite routines, and their corresponding KeLeaveCriticalRegion from the Release routines. This should simplify these driver callback routines, and as an added bonus Verifier will work too.

I have looked at several Microsoft drivers that are using these callbacks and none of these drivers are calling KeEnterCriticalRegion/KeLeaveCriticalRegion here. If I remember correctly, fastfat.sys source code is available outside MS, so you can take a look at its AcquireForLazyWrite routine as an example. In the version I am looking at, fastfat’s routine is even nicely annotated as __drv_mustHoldCriticalRegion.

In the event that I missed some scenarios and there is a corner case where the kernel didn’t disable APCs before calling these routines:
a. Verifier will complain when the driver calls ExAcquireResourceSharedLite or similar.
b. Checked build versions of the kernel will assert from ExAcquireResourceSharedLite.

For future versions of Windows we will look into changing Verifier to be more resilient to coding patterns similar to this one. We were not aware of this coding pattern when we added this Verifier break in Vista. Thanks for pointing this out! Please continue to let us know whenever you conclude that Verifier must have been wrong, so we can try to sort out these problems. We definitely don’t want drivers to have to workaround Verifier behavior.

Thanks,
Dan

9

> For future versions of Windows we will look into changing Verifier to be

more resilient to coding patterns similar to this one

Thanks Dan, I think you will find it is quite a common pattern
(paradoxically because the verifier does such a good job of APC count
checking!).

Sadly, it’s not always as easy as removing the KeEnterCriticalRegion - the
whole business of locking is often abstracted away. This is why, as I
mentioned at the top, I often go for special casing to test whether verifier
is there. That way I can run verifier, but all my extra checking can be
enabled in that path only, by doing a few runs without verifier - this means
I win but it is cumbersome and so it is nice to know that in future versions
I will be able to get rid of the special casing.

And in case it has not been obvious - I am firmly of the belief that
Verifier is one of the most (if not *the* most) useful tools available to
file system writers. Thanks for that too.

Rod

wrote in message news:xxxxx@ntfsd…
>I have been discussing these issues with folks who work on the Cache
>Manager and Memory Manager. We concluded that AcquireForLazyWrite and
>AcquireForModWrite routines don’t need to disable normal kernel APCs.
>Disabling APCs before acquiring an ERESOURCE is needed to prevent an
>application sending a Suspend APC and suspending the thread that owns the
>lock. That scenario cannot happen for those ERESOURCEs acquired by
>AcquireForLazyWrite or AcquireForModWrite, because these routines can get
>called just in one of these two circumstances:
>
> 1. With APCs already disabled by the Cache Manager.
> 2. In the context of a System thread, that apps cannot suspend.
>
> Bottom line is that the kernel is responsible for ensuring that APCs are
> properly disabled before it calls any of AcquireForLazyWrite or
> AcquireForModWrite, and responsible to re-enable APCs only after the
> corresponding Release routines have been called. The Cache Manager folks
> will clarify this behavior in the documentation that describes
> AcquireForLazyWrite. AcquireForModWrite seems to be deprecated, so
> probably there won’t be a doc update for it.
>
> So we recommend removing the KeEnterCriticalRegion calls from
> AcquireForLazyWrite or AcquireForModWrite routines, and their
> corresponding KeLeaveCriticalRegion from the Release routines. This should
> simplify these driver callback routines, and as an added bonus Verifier
> will work too.
>
> I have looked at several Microsoft drivers that are using these callbacks
> and none of these drivers are calling
> KeEnterCriticalRegion/KeLeaveCriticalRegion here. If I remember correctly,
> fastfat.sys source code is available outside MS, so you can take a look at
> its AcquireForLazyWrite routine as an example. In the version I am looking
> at, fastfat’s routine is even nicely annotated as
> __drv_mustHoldCriticalRegion.
>
> In the event that I missed some scenarios and there is a corner case where
> the kernel didn’t disable APCs before calling these routines:
> a. Verifier will complain when the driver calls
> ExAcquireResourceSharedLite or similar.
> b. Checked build versions of the kernel will assert from
> ExAcquireResourceSharedLite.
>
> For future versions of Windows we will look into changing Verifier to be
> more resilient to coding patterns similar to this one. We were not aware
> of this coding pattern when we added this Verifier break in Vista. Thanks
> for pointing this out! Please continue to let us know whenever you
> conclude that Verifier must have been wrong, so we can try to sort out
> these problems. We definitely don’t want drivers to have to workaround
> Verifier behavior.
>
> Thanks,
> Dan
>
>
>
>
>
>
>
>
>

10

> Sadly, it’s not always as easy as removing the KeEnterCriticalRegion - the

whole business of locking is often abstracted away.

Then rework the abstractions to make such removal possible.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

11

Dan,

When we explain it to the FS Filter team at MS and are told that this is
a feature, not a bug, we naturally assume that sending it to a generic
alias at Microsoft is unlikely to obtain any additional satisfaction and
piss off the folks that said “feature, not a bug.” Many of us must work
actively with the filter manager team, so annoying them is seldom good
practice.

Tony
OSR

12

Tony, I was not aware of these discussions, and I cannot comment on them. I promise to try to help out with any issues related to Verifier, as long as I am responsible for Verifier. Also, the other folks in my team are available on the Verifier email alias that I mentioned.

I still hope that designs similar to AcquireForLazyWrite & AcquireForModWrite are uncommon, because they seem to me as being too fragile. Steps:

a. Driver1 calls into Driver2.
b. Driver2 acquires a lock and returns.
c. Driver1 does a bunch of work.
d. Driver1 calls again into Driver2.
e. Driver2 releases the lock and returns.

I would rather discourage this kind of designs, instead of spending a bunch of energy trying to support them with Verifier, and missing more important driver bugs in that process. In this example, hopefully Driver2 will able to acquire whatever internal locks it needs during step (c), so we can eliminate all the other four steps, and the need for these two drivers to be aware of each other’s locks.

So please continue to show me examples of scenarios that will lead to similar Verifier false positives. At this point I still hope these scenarios are uncommon and we can replace them with better designs in the future.

Thanks,
Dan

13

Tony,

Can you please elaborate on specific conversations/communication you
have had with the FS Filter team, that you are referring to below?

As Dan has correctly pointed out, this specific issue is NOT an
overactive verifier check. Windows file systems (FAT, NTFS, …) have to
live with the same IO pattern but can still get this right. I do not see
why a filter or a third-party file system cannot do the same.

In general, I have not had anyone bring a spurious verifier check to my
notice - either in person (at plugfest, etc) or in correspondence
(e-mail, ntfsd, etc). If you find one, I will be happy to have it taken
care of.

Regards,
Sarosh.
File System Filter Lead
Microsoft Corp

This posting is provided “AS IS” with no warranties, and confers no Rights

Tony Mason wrote:

Dan,

When we explain it to the FS Filter team at MS and are told that this is
a feature, not a bug, we naturally assume that sending it to a generic
alias at Microsoft is unlikely to obtain any additional satisfaction and
piss off the folks that said “feature, not a bug.” Many of us must work
actively with the filter manager team, so annoying them is seldom good
practice.

Tony
OSR

14

I agree with Maxim here. If the abstractions are getting in the way of
correctness (or testability with respect to verifier) then they need to go.

As Dan correctly pointed out, the FastFat sample has to live with the
same IO pattern but can still get this right.

Regards,
Sarosh.
File System Filter Lead
Microsoft Corp

This posting is provided “AS IS” with no warranties, and confers no Rights

Maxim S. Shatskih wrote:

> Sadly, it’s not always as easy as removing the KeEnterCriticalRegion - the
> whole business of locking is often abstracted away.

Then rework the abstractions to make such removal possible.

15

Sarosh,

The two issues that my team discussed with you at prior plug fests of
which I am aware:

(1) The kernel handle issue, where using a kernel handle is now enforced
by verifier, but this leads to a serious performance issue due to the
added context switch (of course, only in VERY narrow cases.) Even when
we could demonstrate that our usage was not introducing the bug for
which the verifier check was being made, we were told that we MUST use
kernel handles. So we “fixed it” - when verifier is active, we use
kernel handles. That makes verifier happy and nobody is going to do
performance studies with verifier running.

(2) This APC index issue. Let me quote from the documentation for
ExAcquireResourceExclusiveLite:

“Normal kernel APC delivery must be disabled before calling this
routine. Disable normal kernel APC delivery by calling
KeEnterCriticalRegion. Delivery must remain disabled until the resource
is released, at which point it can be reenabled by calling
KeLeaveCriticalRegion.”

When the verifier check for this was FIRST introduced, we changed to a
model in which our code always acquired the lock via an indirect
mechanism - a stub/wrapper around the acquire that called
KeEnterCriticalRegion (a/k/a FsRtlEnterFileSystem.) Since that time
we’ve added additional infrastructure around that, including a
comprehensive way to check that locks are acquired in order and that we
never exit without releasing them - all by watching the apc index
disable field.

What I find fascinating about this now is that we’re being told “your
simple abstraction is broken, you really should NOT do what the
documentation says [call it a “doc bug” if you’d like] and instead
should special case this.” It seems small wonder why file system
filters become hideously complicated rat’s nests of special cases and
exceptions. Follow the rules, make the code work, enhance the system
(the apc index is very useful when combined with a wrapper model this
way because it allows us to find unreleased locks) and then have someone
on your end decide that “the way the OS file systems do this is the
correct way” and discard consideration of any other model is galling.

Of course, ultimately it is your OS. We changed our abstraction - we
vary the behavior based upon whether or not verifier is present. In
that fashion, we can still find resource leaks and we can still enforce
our lock hierarchy using our standard primitives. But the code is more
complicated and not quite as easy to understand.

Dan - I long ago gave up trying to discuss many of these cases very
actively because I found that it was seldom a fruitful exercise - once
you (Microsoft) have implemented and released a behavior, I have to
support it, even if you change that behavior in a future version of the
OS. I have a code base replete with exactly these types of changes.

I agree that the pattern of locking here is complicated, but it is
inherent in the nature of the random callback from other components
model of the OS with respect to file systems. When I receive a
callback, I don’t actually know where it originated - “most of the time”
it will have originated with the Cc/Mm components, but “once in a while”
we’ll find some other layered component that is clever and calls back
into our entry points. The teams at Microsoft are in a superior
position here - if a 3rd party product is installed and causes a
verifier failure or a crash, the 3rd party product will be (by
definition) “at fault”. However, when it comes to two 3rd party
products, it will normally be the fault of the last component installed
on the box. We actually both have the same perspective - we don’t want
a BSOD to point back to our code. You guys have the advantage of being
able to build the tools, so “!analyze -v” can walk the stack and find
the first 3rd party driver and blame it for the crash, thus diverting
attention away. I do not have that luxury, so I try to always do
everything I can to make sure I’m going to work right. If you send my
file object to NTFS, NTFS will crash. If you send NTFS’s file object to
me, I’ll return an error - because I don’t want to crash in my
code/driver.

So I have a different way of looking at Verifier than others do: to me,
you are defining what you decide is correct behavior of the OS. Since
you’re with Microsoft, your definition is all that matters. I’ll
“change my abstractions” to work the way you define they should work.
Sometimes that might not be in a fashion of which you approve, but odds
are you’ll be gone from the scene in another few years and someone else
will be in your shoes telling me how THEY think it should be done (and
it’ll be the opposite of what you told me.)

Tony
OSR

16

Out of curiosity, what was the kernel handle issue that you were alluding to, or can you not share? I’d be curious to understand the circumstances where you were seeing significant differences here performance-wise.

  • S

-----Original Message-----
From: Tony Mason
Sent: Tuesday, March 31, 2009 08:09
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Server 2008 + verifier + FSD issue

Sarosh,

The two issues that my team discussed with you at prior plug fests of
which I am aware:

(1) The kernel handle issue, where using a kernel handle is now enforced
by verifier, but this leads to a serious performance issue due to the
added context switch (of course, only in VERY narrow cases.) Even when
we could demonstrate that our usage was not introducing the bug for
which the verifier check was being made, we were told that we MUST use
kernel handles. So we “fixed it” - when verifier is active, we use
kernel handles. That makes verifier happy and nobody is going to do
performance studies with verifier running.

(2) This APC index issue. Let me quote from the documentation for
ExAcquireResourceExclusiveLite:

“Normal kernel APC delivery must be disabled before calling this
routine. Disable normal kernel APC delivery by calling
KeEnterCriticalRegion. Delivery must remain disabled until the resource
is released, at which point it can be reenabled by calling
KeLeaveCriticalRegion.”

When the verifier check for this was FIRST introduced, we changed to a
model in which our code always acquired the lock via an indirect
mechanism - a stub/wrapper around the acquire that called
KeEnterCriticalRegion (a/k/a FsRtlEnterFileSystem.) Since that time
we’ve added additional infrastructure around that, including a
comprehensive way to check that locks are acquired in order and that we
never exit without releasing them - all by watching the apc index
disable field.

What I find fascinating about this now is that we’re being told “your
simple abstraction is broken, you really should NOT do what the
documentation says [call it a “doc bug” if you’d like] and instead
should special case this.” It seems small wonder why file system
filters become hideously complicated rat’s nests of special cases and
exceptions. Follow the rules, make the code work, enhance the system
(the apc index is very useful when combined with a wrapper model this
way because it allows us to find unreleased locks) and then have someone
on your end decide that “the way the OS file systems do this is the
correct way” and discard consideration of any other model is galling.

Of course, ultimately it is your OS. We changed our abstraction - we
vary the behavior based upon whether or not verifier is present. In
that fashion, we can still find resource leaks and we can still enforce
our lock hierarchy using our standard primitives. But the code is more
complicated and not quite as easy to understand.

Dan - I long ago gave up trying to discuss many of these cases very
actively because I found that it was seldom a fruitful exercise - once
you (Microsoft) have implemented and released a behavior, I have to
support it, even if you change that behavior in a future version of the
OS. I have a code base replete with exactly these types of changes.

I agree that the pattern of locking here is complicated, but it is
inherent in the nature of the random callback from other components
model of the OS with respect to file systems. When I receive a
callback, I don’t actually know where it originated - “most of the time”
it will have originated with the Cc/Mm components, but “once in a while”
we’ll find some other layered component that is clever and calls back
into our entry points. The teams at Microsoft are in a superior
position here - if a 3rd party product is installed and causes a
verifier failure or a crash, the 3rd party product will be (by
definition) “at fault”. However, when it comes to two 3rd party
products, it will normally be the fault of the last component installed
on the box. We actually both have the same perspective - we don’t want
a BSOD to point back to our code. You guys have the advantage of being
able to build the tools, so “!analyze -v” can walk the stack and find
the first 3rd party driver and blame it for the crash, thus diverting
attention away. I do not have that luxury, so I try to always do
everything I can to make sure I’m going to work right. If you send my
file object to NTFS, NTFS will crash. If you send NTFS’s file object to
me, I’ll return an error - because I don’t want to crash in my
code/driver.

So I have a different way of looking at Verifier than others do: to me,
you are defining what you decide is correct behavior of the OS. Since
you’re with Microsoft, your definition is all that matters. I’ll
“change my abstractions” to work the way you define they should work.
Sometimes that might not be in a fashion of which you approve, but odds
are you’ll be gone from the scene in another few years and someone else
will be in your shoes telling me how THEY think it should be done (and
it’ll be the opposite of what you told me.)

Tony
OSR

—
NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

17

>I’d be curious to understand the circumstances where you were seeing

significant differences here performance-wise.

On older revs of the O/S (XP for sure), if you’re not in the system process
context when you create a kernel handle Ob does a context switch to the
system process (KeStackAttachProcess). Another switch happens when you close
the handle.

If you do enough rapid opens/closes the overhead can become large.

-scott

–
Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

“Skywing” wrote in message news:xxxxx@ntfsd…
Out of curiosity, what was the kernel handle issue that you were alluding
to, or can you not share? I’d be curious to understand the circumstances
where you were seeing significant differences here performance-wise.

- S

-----Original Message-----
From: Tony Mason
Sent: Tuesday, March 31, 2009 08:09
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Server 2008 + verifier + FSD issue

Sarosh,

The two issues that my team discussed with you at prior plug fests of
which I am aware:

(1) The kernel handle issue, where using a kernel handle is now enforced
by verifier, but this leads to a serious performance issue due to the
added context switch (of course, only in VERY narrow cases.) Even when
we could demonstrate that our usage was not introducing the bug for
which the verifier check was being made, we were told that we MUST use
kernel handles. So we “fixed it” - when verifier is active, we use
kernel handles. That makes verifier happy and nobody is going to do
performance studies with verifier running.

(2) This APC index issue. Let me quote from the documentation for
ExAcquireResourceExclusiveLite:

“Normal kernel APC delivery must be disabled before calling this
routine. Disable normal kernel APC delivery by calling
KeEnterCriticalRegion. Delivery must remain disabled until the resource
is released, at which point it can be reenabled by calling
KeLeaveCriticalRegion.”

When the verifier check for this was FIRST introduced, we changed to a
model in which our code always acquired the lock via an indirect
mechanism - a stub/wrapper around the acquire that called
KeEnterCriticalRegion (a/k/a FsRtlEnterFileSystem.) Since that time
we’ve added additional infrastructure around that, including a
comprehensive way to check that locks are acquired in order and that we
never exit without releasing them - all by watching the apc index
disable field.

What I find fascinating about this now is that we’re being told “your
simple abstraction is broken, you really should NOT do what the
documentation says [call it a “doc bug” if you’d like] and instead
should special case this.” It seems small wonder why file system
filters become hideously complicated rat’s nests of special cases and
exceptions. Follow the rules, make the code work, enhance the system
(the apc index is very useful when combined with a wrapper model this
way because it allows us to find unreleased locks) and then have someone
on your end decide that “the way the OS file systems do this is the
correct way” and discard consideration of any other model is galling.

Of course, ultimately it is your OS. We changed our abstraction - we
vary the behavior based upon whether or not verifier is present. In
that fashion, we can still find resource leaks and we can still enforce
our lock hierarchy using our standard primitives. But the code is more
complicated and not quite as easy to understand.

Dan - I long ago gave up trying to discuss many of these cases very
actively because I found that it was seldom a fruitful exercise - once
you (Microsoft) have implemented and released a behavior, I have to
support it, even if you change that behavior in a future version of the
OS. I have a code base replete with exactly these types of changes.

I agree that the pattern of locking here is complicated, but it is
inherent in the nature of the random callback from other components
model of the OS with respect to file systems. When I receive a
callback, I don’t actually know where it originated - “most of the time”
it will have originated with the Cc/Mm components, but “once in a while”
we’ll find some other layered component that is clever and calls back
into our entry points. The teams at Microsoft are in a superior
position here - if a 3rd party product is installed and causes a
verifier failure or a crash, the 3rd party product will be (by
definition) “at fault”. However, when it comes to two 3rd party
products, it will normally be the fault of the last component installed
on the box. We actually both have the same perspective - we don’t want
a BSOD to point back to our code. You guys have the advantage of being
able to build the tools, so “!analyze -v” can walk the stack and find
the first 3rd party driver and blame it for the crash, thus diverting
attention away. I do not have that luxury, so I try to always do
everything I can to make sure I’m going to work right. If you send my
file object to NTFS, NTFS will crash. If you send NTFS’s file object to
me, I’ll return an error - because I don’t want to crash in my
code/driver.

So I have a different way of looking at Verifier than others do: to me,
you are defining what you decide is correct behavior of the OS. Since
you’re with Microsoft, your definition is all that matters. I’ll
“change my abstractions” to work the way you define they should work.
Sometimes that might not be in a fashion of which you approve, but odds
are you’ll be gone from the scene in another few years and someone else
will be in your shoes telling me how THEY think it should be done (and
it’ll be the opposite of what you told me.)

Tony
OSR

—
NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

18

Okay, that makes sense. What safeguards you from the local app altering the handle, though? Are you only running during early process init?

(Sorry if you’ve already explained this before.)

  • S

-----Original Message-----
From: Scott Noone
Sent: Tuesday, March 31, 2009 10:55
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Server 2008 + verifier + FSD issue

>I’d be curious to understand the circumstances where you were seeing
>significant differences here performance-wise.

On older revs of the O/S (XP for sure), if you’re not in the system process
context when you create a kernel handle Ob does a context switch to the
system process (KeStackAttachProcess). Another switch happens when you close
the handle.

If you do enough rapid opens/closes the overhead can become large.

-scott

–
Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

“Skywing” wrote in message news:xxxxx@ntfsd…
Out of curiosity, what was the kernel handle issue that you were alluding
to, or can you not share? I’d be curious to understand the circumstances
where you were seeing significant differences here performance-wise.

- S

-----Original Message-----
From: Tony Mason
Sent: Tuesday, March 31, 2009 08:09
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Server 2008 + verifier + FSD issue

Sarosh,

The two issues that my team discussed with you at prior plug fests of
which I am aware:

(1) The kernel handle issue, where using a kernel handle is now enforced
by verifier, but this leads to a serious performance issue due to the
added context switch (of course, only in VERY narrow cases.) Even when
we could demonstrate that our usage was not introducing the bug for
which the verifier check was being made, we were told that we MUST use
kernel handles. So we “fixed it” - when verifier is active, we use
kernel handles. That makes verifier happy and nobody is going to do
performance studies with verifier running.

(2) This APC index issue. Let me quote from the documentation for
ExAcquireResourceExclusiveLite:

“Normal kernel APC delivery must be disabled before calling this
routine. Disable normal kernel APC delivery by calling
KeEnterCriticalRegion. Delivery must remain disabled until the resource
is released, at which point it can be reenabled by calling
KeLeaveCriticalRegion.”

When the verifier check for this was FIRST introduced, we changed to a
model in which our code always acquired the lock via an indirect
mechanism - a stub/wrapper around the acquire that called
KeEnterCriticalRegion (a/k/a FsRtlEnterFileSystem.) Since that time
we’ve added additional infrastructure around that, including a
comprehensive way to check that locks are acquired in order and that we
never exit without releasing them - all by watching the apc index
disable field.

What I find fascinating about this now is that we’re being told “your
simple abstraction is broken, you really should NOT do what the
documentation says [call it a “doc bug” if you’d like] and instead
should special case this.” It seems small wonder why file system
filters become hideously complicated rat’s nests of special cases and
exceptions. Follow the rules, make the code work, enhance the system
(the apc index is very useful when combined with a wrapper model this
way because it allows us to find unreleased locks) and then have someone
on your end decide that “the way the OS file systems do this is the
correct way” and discard consideration of any other model is galling.

Of course, ultimately it is your OS. We changed our abstraction - we
vary the behavior based upon whether or not verifier is present. In
that fashion, we can still find resource leaks and we can still enforce
our lock hierarchy using our standard primitives. But the code is more
complicated and not quite as easy to understand.

Dan - I long ago gave up trying to discuss many of these cases very
actively because I found that it was seldom a fruitful exercise - once
you (Microsoft) have implemented and released a behavior, I have to
support it, even if you change that behavior in a future version of the
OS. I have a code base replete with exactly these types of changes.

I agree that the pattern of locking here is complicated, but it is
inherent in the nature of the random callback from other components
model of the OS with respect to file systems. When I receive a
callback, I don’t actually know where it originated - “most of the time”
it will have originated with the Cc/Mm components, but “once in a while”
we’ll find some other layered component that is clever and calls back
into our entry points. The teams at Microsoft are in a superior
position here - if a 3rd party product is installed and causes a
verifier failure or a crash, the 3rd party product will be (by
definition) “at fault”. However, when it comes to two 3rd party
products, it will normally be the fault of the last component installed
on the box. We actually both have the same perspective - we don’t want
a BSOD to point back to our code. You guys have the advantage of being
able to build the tools, so “!analyze -v” can walk the stack and find
the first 3rd party driver and blame it for the crash, thus diverting
attention away. I do not have that luxury, so I try to always do
everything I can to make sure I’m going to work right. If you send my
file object to NTFS, NTFS will crash. If you send NTFS’s file object to
me, I’ll return an error - because I don’t want to crash in my
code/driver.

So I have a different way of looking at Verifier than others do: to me,
you are defining what you decide is correct behavior of the OS. Since
you’re with Microsoft, your definition is all that matters. I’ll
“change my abstractions” to work the way you define they should work.
Sometimes that might not be in a fashion of which you approve, but odds
are you’ll be gone from the scene in another few years and someone else
will be in your shoes telling me how THEY think it should be done (and
it’ll be the opposite of what you told me.)

Tony
OSR

—
NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

19

>What safeguards you from the local app altering the handle, though? Are

you only running during early process init?

Well, the good news is that in this particular usage case we only need to be
crash proof (and not give the user access to anything they shouldn’t have
access to, of course).

For example, one is part of our, “is this particular file encrypted” logic
(open it, get a file object, look for our signature) and if the app does
something nasty the worst that happens is that we tell him the wrong answer.
Not saying it’s pretty and it adds hidden complexity to the code (don’t let
the intern update it), but it makes a measurable difference in CPU
utilization.

-scott

–
Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

“Skywing” wrote in message news:xxxxx@ntfsd…
Okay, that makes sense. What safeguards you from the local app altering the
handle, though? Are you only running during early process init?

(Sorry if you’ve already explained this before.)

- S

-----Original Message-----
From: Scott Noone
Sent: Tuesday, March 31, 2009 10:55
To: Windows File Systems Devs Interest List
Subject: Re:[ntfsd] Server 2008 + verifier + FSD issue

>I’d be curious to understand the circumstances where you were seeing
>significant differences here performance-wise.

On older revs of the O/S (XP for sure), if you’re not in the system process
context when you create a kernel handle Ob does a context switch to the
system process (KeStackAttachProcess). Another switch happens when you close
the handle.

If you do enough rapid opens/closes the overhead can become large.

-scott

–
Scott Noone
Consulting Associate
OSR Open Systems Resources, Inc.
http://www.osronline.com

“Skywing” wrote in message news:xxxxx@ntfsd…
Out of curiosity, what was the kernel handle issue that you were alluding
to, or can you not share? I’d be curious to understand the circumstances
where you were seeing significant differences here performance-wise.

- S

-----Original Message-----
From: Tony Mason
Sent: Tuesday, March 31, 2009 08:09
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Server 2008 + verifier + FSD issue

Sarosh,

The two issues that my team discussed with you at prior plug fests of
which I am aware:

(1) The kernel handle issue, where using a kernel handle is now enforced
by verifier, but this leads to a serious performance issue due to the
added context switch (of course, only in VERY narrow cases.) Even when
we could demonstrate that our usage was not introducing the bug for
which the verifier check was being made, we were told that we MUST use
kernel handles. So we “fixed it” - when verifier is active, we use
kernel handles. That makes verifier happy and nobody is going to do
performance studies with verifier running.

(2) This APC index issue. Let me quote from the documentation for
ExAcquireResourceExclusiveLite:

“Normal kernel APC delivery must be disabled before calling this
routine. Disable normal kernel APC delivery by calling
KeEnterCriticalRegion. Delivery must remain disabled until the resource
is released, at which point it can be reenabled by calling
KeLeaveCriticalRegion.”

When the verifier check for this was FIRST introduced, we changed to a
model in which our code always acquired the lock via an indirect
mechanism - a stub/wrapper around the acquire that called
KeEnterCriticalRegion (a/k/a FsRtlEnterFileSystem.) Since that time
we’ve added additional infrastructure around that, including a
comprehensive way to check that locks are acquired in order and that we
never exit without releasing them - all by watching the apc index
disable field.

What I find fascinating about this now is that we’re being told “your
simple abstraction is broken, you really should NOT do what the
documentation says [call it a “doc bug” if you’d like] and instead
should special case this.” It seems small wonder why file system
filters become hideously complicated rat’s nests of special cases and
exceptions. Follow the rules, make the code work, enhance the system
(the apc index is very useful when combined with a wrapper model this
way because it allows us to find unreleased locks) and then have someone
on your end decide that “the way the OS file systems do this is the
correct way” and discard consideration of any other model is galling.

Of course, ultimately it is your OS. We changed our abstraction - we
vary the behavior based upon whether or not verifier is present. In
that fashion, we can still find resource leaks and we can still enforce
our lock hierarchy using our standard primitives. But the code is more
complicated and not quite as easy to understand.

Dan - I long ago gave up trying to discuss many of these cases very
actively because I found that it was seldom a fruitful exercise - once
you (Microsoft) have implemented and released a behavior, I have to
support it, even if you change that behavior in a future version of the
OS. I have a code base replete with exactly these types of changes.

I agree that the pattern of locking here is complicated, but it is
inherent in the nature of the random callback from other components
model of the OS with respect to file systems. When I receive a
callback, I don’t actually know where it originated - “most of the time”
it will have originated with the Cc/Mm components, but “once in a while”
we’ll find some other layered component that is clever and calls back
into our entry points. The teams at Microsoft are in a superior
position here - if a 3rd party product is installed and causes a
verifier failure or a crash, the 3rd party product will be (by
definition) “at fault”. However, when it comes to two 3rd party
products, it will normally be the fault of the last component installed
on the box. We actually both have the same perspective - we don’t want
a BSOD to point back to our code. You guys have the advantage of being
able to build the tools, so “!analyze -v” can walk the stack and find
the first 3rd party driver and blame it for the crash, thus diverting
attention away. I do not have that luxury, so I try to always do
everything I can to make sure I’m going to work right. If you send my
file object to NTFS, NTFS will crash. If you send NTFS’s file object to
me, I’ll return an error - because I don’t want to crash in my
code/driver.

So I have a different way of looking at Verifier than others do: to me,
you are defining what you decide is correct behavior of the OS. Since
you’re with Microsoft, your definition is all that matters. I’ll
“change my abstractions” to work the way you define they should work.
Sometimes that might not be in a fashion of which you approve, but odds
are you’ll be gone from the scene in another few years and someone else
will be in your shoes telling me how THEY think it should be done (and
it’ll be the opposite of what you told me.)

Tony
OSR

—
NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
NTFSD is sponsored by OSR

For our schedule of debugging and file system seminars
(including our new fs mini-filter seminar) visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

20

This was more than a couple of plugfests ago, but I think I remember
this now.

The DMK was opening an handle without the OBJ_KERNEL_HANDLE flag and
subsequently issuing IO on the handle.

Here is one of the problems with this:

  1. Opening an handle without the OBJ_KERNEL_HANDLE flag would cause the
    handle to be put in user’s handle table.

  2. A malicious user could close this handle in user-mode.

  3. The filter would then issue IO on that handle in kernel-mode and crash.

Are you saying this is not a problem?

Regards,
Sarosh.
File System Filter Lead
Microsoft Corp

This posting is provided “AS IS” with no warranties, and confers no Rights

Scott Noone wrote:

> What safeguards you from the local app altering the handle, though? Are
> you only running during early process init?

Well, the good news is that in this particular usage case we only need to be
crash proof (and not give the user access to anything they shouldn’t have
access to, of course).

For example, one is part of our, “is this particular file encrypted” logic
(open it, get a file object, look for our signature) and if the app does
something nasty the worst that happens is that we tell him the wrong answer.
Not saying it’s pretty and it adds hidden complexity to the code (don’t let
the intern update it), but it makes a measurable difference in CPU
utilization.

-scott