Avoid Re-entrancy

Hello All,

I am very interested in reading the dialog between Marc and Joe. It
is certainly a very good ideal by “borrowing” or “hijacking” the IRP
to do your own create request. If Microsoft can provide the downwards
Create API, it will make our life much easier.

I’d like to ask a similar question, can I “borrow” or “hijack” a Clean or
Close IRP to do a Create task in Filter? Is there any one done this type of
work before? I am trying to avoid the re-entrance from Clean/Close to
Create dispatcher. Any suggestions?

Thanks in advance.
Lixin

/*

I actually do something similar with my last file system filter. In
that filter, i would always open the target of the create and do exactly
what you described (I called it ‘hijacking’ the irp). I agree, it works
well (though they’re are a few subtle things I had to work out, especially
with regards to relative creates and filename handling in NTFS).
With this new filter I am writing, though, I do not open the target of
the create, but I open each subdirectory leading up to the target file. For
example, for ‘C:\USERS\FRED\TEMP.TXT’ I open '', then ‘\USERS’, then
‘\USERS\FRED’. I considered using the ‘hijack’ technique here, too, and it
would probably work fine, but it gets a little hairy when dealing with
relative opens. For example, if the create is relative to C:\USERS\FRED, I
would have to NULL the RelatedFileObject for the first few iterations, then
restore it for the ‘real’ create. I do not know if this would work. Like
everything else, it is probably more complicated than it looks. I may try
it, because it would solve the problem with a proven technique. If we ever
get a FAQ for this mailing list, maybe a code snippet showing how to do this
would be nice. I’d be happy to provide my implementation.
My rail was basically a plea to those who can do something about it
(i.e. Microsoft) to either release a sample in the IFS kit or export a
function that allows file system filter implementers to perform ‘downward
only’ creates. Judging by some of the ‘home grown’ implementations of this,
it shouldn’t be very difficult. If CREATE handling is the #1 source of
interoperability problems between file system filter drivers, and many of
those problems are caused by reentering the device stack, it would seem
logical that Microsoft would want to provide us with a means of avoiding
these problems and/or having to invent dubious work arounds.

Thanks,
Joel

-----Original Message-----
From: Marc Sherman [mailto:xxxxx@bionetrix.com]
Sent: Tuesday, December 12, 2000 3:39 PM
To: File Systems Developers
Subject: [ntfsd] RE: railing on reentrancy (wasRe: thread local storage in
ker nel mode)

Joel,

I had a design that called ZwCreateFile() from my create dispatch and I
changed it due to the recursive create discussion on this list. What I do
now is “borrow” the create irp sent to me by the IO manager. I modify the
irp based on my needs and send it on using IoCallDriver(). After I’ve
finished “borrowing” the irp, I restore any values I’ve modified to the
original values (this works because I always return
STATUS_MORE_PROCESSING_REQUIRED from my completion routine which allows my
create dispatch routine to restore the values). I like this a whole lot
better because it avoids the re-entrancy problems caused by ZwCreateFile().
It’s been working great!

good luck,
Marc

-----Original Message-----
From: Smith, Joel [mailto:xxxxx@ntpsoftware.com]
Sent: Tuesday, December 12, 2000 11:08 AM
To: File Systems Developers
Subject: [ntfsd] railing on reentrancy (wasRe: thread local storage in ker
nel mode)

The more I think about, the more I disklike using ZwCreatFile at all
in this scenario. The issue of reentering the device stack during create
processing was discussed exhaustively a few months back (search for
recursive create in the ntfs archive) and I don’t want to re-open that can
of worms. It seems that my proposed implementation lends itself to one of
those nightmare scenarios where another file system filter, also reentrant
and processing creates in multiple threads (i.e. a heirarchical storage file
system filter that processes some creates in the context of a service thread
before switching back to the original context to forward the irp to the next
driver) is layered below me. The potential for deadlocks or endless loops
is obvious. I can anticipate many potential problems, but surely not them
all. Yes, the plugfests and such will weed out these potential conflicts,
but not everyone in the game can attend these events.

I think we can all agree that reentering the device stack while
processing a dispatch routine is a bad idea, frought with peril; It can be
done, but is hard to get right, and depends on everyone else in the stack
getting it right too. All irps generated within the stack should go
strictly down the stack. If all filter driver implementors obeyed this
simple maxim, stability of the os (when running multiple filters) would
surely be increased. In the interest of the precieved stability of
WindowsNT, why hasn’t someone documented how to roll create irps? I can
find a few implementations, but they are from developers who have obviously
reverse engineered some IO manager routines to this effect. With everyone
using ‘home grown’ techniques of rolling create irps, I’m not sure if we are
being self defeating in our attempts to resolve the reentrancy problems.
Can anyone help US out here?

-Joel

-----Original Message-----
From: Smith, Joel
Sent: Tuesday, December 12, 2000 9:38 AM
To: ‘File Systems Developers’
Subject: RE: [ntfsd] Re: thread local storage in kernel mode

As a means of detecting reentrancy; I am developing a file system
filter. I am building up a prefix table of paths for file’s in the system.
The prefix table entries point to a unique identifier (either an FCB or an
ID, I haven’t decided yet) so that I can quickly and unambiguously resolve
contexts assigned to directory nodes (regardless of filename format, long,
short, etc). In order to do this, I traverse each node of the path in
create, checking the prefix on each iteration to see if it is in the prefix
table. If the prefix is not in the prefix table, I open it, get the unique
id, and add an entry to the prefix table.

I am using ZwCreateFile to open each prefix from the create dispatch
routine. I considered rolling my own irp to avoid the reentrancy or
employing a reparse scheme, but decided the reentrant employment of
ZwCreateFile would be adequate and easiest to implement. It would be nice
to have a counter in TLS to detect the reentrancy for efficiency reasons;
without TLS, I have to maintain a table with proper synchronization. With
TLS, I would not have to acquire any synchronization primitives and would
not have to keep the counter in a table.

Note that none of this is implemented yet, so there may be holes in
the logic.

-Joel


You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Microsoft has a downwards only api in the new IFS kit (Whistler) but it
won’t do you much good if you need to support NT4 or Win2K.
I’ve never tried to ‘borrow’ a clean or create irp and change it into a
create irp. This should be at least as hard as ‘rolling your own’ create
irp, which is to say, don’t try it unless you really know what you are
doing.

-----Original Message-----
From: lixin [mailto:xxxxx@indefense.com]
Sent: Tuesday, May 29, 2001 3:45 PM
To: File Systems Developers
Subject: [ntfsd] Avoid Re-entrancy

Hello All,

I am very interested in reading the dialog between Marc and Joe. It
is certainly a very good ideal by “borrowing” or “hijacking” the IRP
to do your own create request. If Microsoft can provide the downwards
Create API, it will make our life much easier.

I’d like to ask a similar question, can I “borrow” or “hijack” a Clean or
Close IRP to do a Create task in Filter? Is there any one done this type of
work before? I am trying to avoid the re-entrance from Clean/Close to
Create dispatcher. Any suggestions?

Thanks in advance.
Lixin

/*

I actually do something similar with my last file system filter. In
that filter, i would always open the target of the create and do exactly
what you described (I called it ‘hijacking’ the irp). I agree, it works
well (though they’re are a few subtle things I had to work out, especially
with regards to relative creates and filename handling in NTFS).
With this new filter I am writing, though, I do not open the target of
the create, but I open each subdirectory leading up to the target file. For
example, for ‘C:\USERS\FRED\TEMP.TXT’ I open '', then ‘\USERS’, then
‘\USERS\FRED’. I considered using the ‘hijack’ technique here, too, and it
would probably work fine, but it gets a little hairy when dealing with
relative opens. For example, if the create is relative to C:\USERS\FRED, I
would have to NULL the RelatedFileObject for the first few iterations, then
restore it for the ‘real’ create. I do not know if this would work. Like
everything else, it is probably more complicated than it looks. I may try
it, because it would solve the problem with a proven technique. If we ever
get a FAQ for this mailing list, maybe a code snippet showing how to do this
would be nice. I’d be happy to provide my implementation.
My rail was basically a plea to those who can do something about it
(i.e. Microsoft) to either release a sample in the IFS kit or export a
function that allows file system filter implementers to perform ‘downward
only’ creates. Judging by some of the ‘home grown’ implementations of this,
it shouldn’t be very difficult. If CREATE handling is the #1 source of
interoperability problems between file system filter drivers, and many of
those problems are caused by reentering the device stack, it would seem
logical that Microsoft would want to provide us with a means of avoiding
these problems and/or having to invent dubious work arounds.

Thanks,
Joel

-----Original Message-----
From: Marc Sherman [mailto:xxxxx@bionetrix.com]
Sent: Tuesday, December 12, 2000 3:39 PM
To: File Systems Developers
Subject: [ntfsd] RE: railing on reentrancy (wasRe: thread local storage in
ker nel mode)

Joel,

I had a design that called ZwCreateFile() from my create dispatch and I
changed it due to the recursive create discussion on this list. What I do
now is “borrow” the create irp sent to me by the IO manager. I modify the
irp based on my needs and send it on using IoCallDriver(). After I’ve
finished “borrowing” the irp, I restore any values I’ve modified to the
original values (this works because I always return
STATUS_MORE_PROCESSING_REQUIRED from my completion routine which allows my
create dispatch routine to restore the values). I like this a whole lot
better because it avoids the re-entrancy problems caused by ZwCreateFile().
It’s been working great!

good luck,
Marc

-----Original Message-----
From: Smith, Joel [mailto:xxxxx@ntpsoftware.com]
Sent: Tuesday, December 12, 2000 11:08 AM
To: File Systems Developers
Subject: [ntfsd] railing on reentrancy (wasRe: thread local storage in ker
nel mode)

The more I think about, the more I disklike using ZwCreatFile at all
in this scenario. The issue of reentering the device stack during create
processing was discussed exhaustively a few months back (search for
recursive create in the ntfs archive) and I don’t want to re-open that can
of worms. It seems that my proposed implementation lends itself to one of
those nightmare scenarios where another file system filter, also reentrant
and processing creates in multiple threads (i.e. a heirarchical storage file
system filter that processes some creates in the context of a service thread
before switching back to the original context to forward the irp to the next
driver) is layered below me. The potential for deadlocks or endless loops
is obvious. I can anticipate many potential problems, but surely not them
all. Yes, the plugfests and such will weed out these potential conflicts,
but not everyone in the game can attend these events.

I think we can all agree that reentering the device stack while
processing a dispatch routine is a bad idea, frought with peril; It can be
done, but is hard to get right, and depends on everyone else in the stack
getting it right too. All irps generated within the stack should go
strictly down the stack. If all filter driver implementors obeyed this
simple maxim, stability of the os (when running multiple filters) would
surely be increased. In the interest of the precieved stability of
WindowsNT, why hasn’t someone documented how to roll create irps? I can
find a few implementations, but they are from developers who have obviously
reverse engineered some IO manager routines to this effect. With everyone
using ‘home grown’ techniques of rolling create irps, I’m not sure if we are
being self defeating in our attempts to resolve the reentrancy problems.
Can anyone help US out here?

-Joel

-----Original Message-----
From: Smith, Joel
Sent: Tuesday, December 12, 2000 9:38 AM
To: ‘File Systems Developers’
Subject: RE: [ntfsd] Re: thread local storage in kernel mode

As a means of detecting reentrancy; I am developing a file system
filter. I am building up a prefix table of paths for file’s in the system.
The prefix table entries point to a unique identifier (either an FCB or an
ID, I haven’t decided yet) so that I can quickly and unambiguously resolve
contexts assigned to directory nodes (regardless of filename format, long,
short, etc). In order to do this, I traverse each node of the path in
create, checking the prefix on each iteration to see if it is in the prefix
table. If the prefix is not in the prefix table, I open it, get the unique
id, and add an entry to the prefix table.

I am using ZwCreateFile to open each prefix from the create dispatch
routine. I considered rolling my own irp to avoid the reentrancy or
employing a reparse scheme, but decided the reentrant employment of
ZwCreateFile would be adequate and easiest to implement. It would be nice
to have a counter in TLS to detect the reentrancy for efficiency reasons;
without TLS, I have to maintain a table with proper synchronization. With
TLS, I would not have to acquire any synchronization primitives and would
not have to keep the counter in a table.

Note that none of this is implemented yet, so there may be holes in
the logic.

-Joel


You are currently subscribed to ntfsd as: xxxxx@ntpsoftware.com
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntfsd as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntfsd-$subst(‘Recip.MemberIDChar’)@lists.osr.com