IO with resource held?

NTFSD Folk:

Somebody asked me a question that I thought I knew the answer to, but wanted confirmation.

Is this a legal sequence (running at PASSIVE_LEVEL)?

FltAcquireResourceExclusive( &Resource );
FltCreateFile( … );
FltWriteFile( … );
FltClose();
FltReleaseResource( &Resource);

Ken

No. Assuming FltAcquireResourceExclusive blocks APCs like it should,
that sequence will call I/O with APCs disabled, which means if they
post, they will hang forever.

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Ken Cross
Sent: Sunday, June 10, 2007 10:42 AM
To: ntfsd redirect
Subject: [ntfsd] IO with resource held?

NTFSD Folk:

Somebody asked me a question that I thought I knew the answer to, but
wanted confirmation.

Is this a legal sequence (running at PASSIVE_LEVEL)?

FltAcquireResourceExclusive( &Resource );
FltCreateFile( … );
FltWriteFile( … );
FltClose();
FltReleaseResource( &Resource);

Ken


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

Well to be fair: the Flt resource acquisition APIs do enter critical
region, disabling kernel APCs. However, special kernel APCs are not
disabled. So there should be no hangs due to i/o completion.

However I would not recommend doing i/o holding locks exclusive unless
you are very sure that you will not be re-entered, when you issue that
i/o. What we have seen usually is that folks arbitrarily assume that
file systems don’t re-enter the stack. If you really don’t want to dig
into the specifics every time, it’s best to avoid this. Usually there
are simpler alternatives.

Ravi

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Sunday, June 10, 2007 7:24 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] IO with resource held?

No. Assuming FltAcquireResourceExclusive blocks APCs like it should,
that sequence will call I/O with APCs disabled, which means if they
post, they will hang forever.

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Ken Cross
Sent: Sunday, June 10, 2007 10:42 AM
To: ntfsd redirect
Subject: [ntfsd] IO with resource held?

NTFSD Folk:

Somebody asked me a question that I thought I knew the answer to, but
wanted confirmation.

Is this a legal sequence (running at PASSIVE_LEVEL)?

FltAcquireResourceExclusive( &Resource );
FltCreateFile( … );
FltWriteFile( … );
FltClose();
FltReleaseResource( &Resource);

Ken


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

I was not assuming anything about the implementation of these APIs. The
Zw APIs expressly allow the caller to provide an APC routine and that is
an ordinary kernel APC. *IF* these routines do not rely upon kernel
APCs internally (something I would think is inherently unsafe to assume)
then special kernel APCs would be sufficient.

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

I agree that that the spirit of your answer was right. Just wanted to make sure that nobody tries it out and goes 'hey it doesn’t hang even when posted, and therefore it’s safe to do this '…
thanks,
Ravi


From: xxxxx@lists.osr.com on behalf of Tony Mason
Sent: Mon 6/11/2007 4:59 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] IO with resource held?

I was not assuming anything about the implementation of these APIs. The
Zw APIs expressly allow the caller to provide an APC routine and that is
an ordinary kernel APC. *IF* these routines do not rely upon kernel
APCs internally (something I would think is inherently unsafe to assume)
then special kernel APCs would be sufficient.

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com http:</http:>


Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Yep. FWIW, I told her don’t do it. But it’s not obvious.

Certainly, user and normal kernel APCs are disabled by FltAcquireResourceExclusive, but special kernel APCs aren’t. So why would a
kernel-created I/O request not work? For fun, I tried it and it works fine with simple tests.

But I don’t think we can *guarantee* that it will never deadlock. Hence: unless some authority says it’s OK, it ain’t.

Too bad. AFAIK, there’s no synchronization mechanism that works at PASSIVE_LEVEL to let you synchronize access (i.e., put a kernel
thread in a wait state) for things like a file.

Ken

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Monday, June 11, 2007 7:59 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] IO with resource held?

I was not assuming anything about the implementation of these APIs. The
Zw APIs expressly allow the caller to provide an APC routine and that is
an ordinary kernel APC. *IF* these routines do not rely upon kernel
APCs internally (something I would think is inherently unsafe to assume)
then special kernel APCs would be sufficient.

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com


Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

I think this is worth clarifying, so I will have to continue this thread.
There are ways to synchronize i/o to a file stream over a single handle: synchronous i/o accomplishes exactly that, where the i/o manager uses an exclusive lock associated with that file object. But i/o is not filtering the FS as well, so no deadlocks occur.

To emphasize this again: the concern is not about i/o completion, the concern is the nature of the exclusive lock here.
If we assume that the create was directed below the filter (which is a necessary condition), and the resource was tied to the specific file object it will be safe, as long as the filter - which is also seeing other i/o go by - doesn’t attempt to re-aquire the same resource exclusively again. So while one can get this right, it requires a lot of careful coding & testing. This is why the generic advice would always be to avoid holding locks across i/o.

Ravi


From: xxxxx@lists.osr.com on behalf of Ken Cross
Sent: Tue 6/12/2007 6:47 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] IO with resource held?

Yep. FWIW, I told her don’t do it. But it’s not obvious.

Certainly, user and normal kernel APCs are disabled by FltAcquireResourceExclusive, but special kernel APCs aren’t. So why would a
kernel-created I/O request not work? For fun, I tried it and it works fine with simple tests.

But I don’t think we can *guarantee* that it will never deadlock. Hence: unless some authority says it’s OK, it ain’t.

Too bad. AFAIK, there’s no synchronization mechanism that works at PASSIVE_LEVEL to let you synchronize access (i.e., put a kernel
thread in a wait state) for things like a file.

Ken

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Monday, June 11, 2007 7:59 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] IO with resource held?

I was not assuming anything about the implementation of these APIs. The
Zw APIs expressly allow the caller to provide an APC routine and that is
an ordinary kernel APC. *IF* these routines do not rely upon kernel
APCs internally (something I would think is inherently unsafe to assume)
then special kernel APCs would be sufficient.

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com http:</http:>


Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com


Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

Thanks, Ravi, but I’m not sure I exactly follow you.

I have 2 threads happily running along, threads A and B, both of which need to write to a file, say C:\abc.dat.

They’re not sharing a handle or FO or anything – they’re sharing the file. When thread A needs to write to the file, it needs to
synchronize access so that thread B doesn’t try to write at the same time, and vice versa.

I was looking for a kernel-supported mechanism to synchronize access, so that if A is writing to the file, B will be made to wait
until it’s done. The alternative is to have A lock the file and when B gets denied access, it sits and polls until it can get it.
Yuck.

There are no kernel-based synchronization objects that work at PASSIVE_LEVEL, are there?

Ken


From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Ravisankar Pudipeddi
Sent: Thursday, June 14, 2007 11:30 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] IO with resource held?

I think this is worth clarifying, so I will have to continue this thread.
There are ways to synchronize i/o to a file stream over a single handle: synchronous i/o accomplishes exactly that, where the i/o
manager uses an exclusive lock associated with that file object. But i/o is not filtering the FS as well, so no deadlocks occur.

To emphasize this again: the concern is not about i/o completion, the concern is the nature of the exclusive lock here.
If we assume that the create was directed below the filter (which is a necessary condition), and the resource was tied to the
specific file object it will be safe, as long as the filter - which is also seeing other i/o go by - doesn’t attempt to re-aquire
the same resource exclusively again. So while one can get this right, it requires a lot of careful coding & testing. This is why the
generic advice would always be to avoid holding locks across i/o.

Ravi


From: xxxxx@lists.osr.com on behalf of Ken Cross
Sent: Tue 6/12/2007 6:47 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] IO with resource held?

Yep. FWIW, I told her don’t do it. But it’s not obvious.

Certainly, user and normal kernel APCs are disabled by FltAcquireResourceExclusive, but special kernel APCs aren’t. So why would a
kernel-created I/O request not work? For fun, I tried it and it works fine with simple tests.

But I don’t think we can *guarantee* that it will never deadlock. Hence: unless some authority says it’s OK, it ain’t.

Too bad. AFAIK, there’s no synchronization mechanism that works at PASSIVE_LEVEL to let you synchronize access (i.e., put a kernel
thread in a wait state) for things like a file.

Ken

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Monday, June 11, 2007 7:59 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] IO with resource held?

I was not assuming anything about the implementation of these APIs. The
Zw APIs expressly allow the caller to provide an APC routine and that is
an ordinary kernel APC. *IF* these routines do not rely upon kernel
APCs internally (something I would think is inherently unsafe to assume)
then special kernel APCs would be sufficient.

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com http:</http:>


Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com


Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com


Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

There are a lot of things at work here, and it is always a bit tricky to
explain how to do this right over email:

You can always synchronize access for A & B and other participating
threads that want serialized access. You can use any exclusive lock for
that, and locks can be acquired at PASSIVE_LEVEL. You do not need to
raise to APC_LEVEL if that’s what you are implying. In fact if the
requirement is just what you say below, you don’t even need to disable
kernel APCs by entering critical region. The only thing you may have to
watch out for is to not block in a non-cancellable way on the lock (i.e.
you ensure that you can abandon if the thread is being terminated).

But if you want to block access for everybody who is accessing the file
as well, in a filter, you can run into problems: obviously you will need
to disable kernel APCs, so that you wouldn’t be suspended because you
are in the mainline path for all i/o and you would freeze the system if
suspended. Secondly you don’t want to deadlock in your filter through
re-entrant i/o even if that doesn’t pertain to the file you are
serializing.

In a nutshell: participative serializing of access to a file is easy.
Enforcing serialized access to a file from anybody (even threads that
don’t acquire the synchronization you want) is hard.

Ravi

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Ken Cross
Sent: Thursday, June 14, 2007 8:29 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] IO with resource held?

Thanks, Ravi, but I’m not sure I exactly follow you.

I have 2 threads happily running along, threads A and B, both of which
need to write to a file, say C:\abc.dat.

They’re not sharing a handle or FO or anything – they’re sharing the
file. When thread A needs to write to the file, it needs to synchronize
access so that thread B doesn’t try to write at the same time, and vice
versa.

I was looking for a kernel-supported mechanism to synchronize access, so
that if A is writing to the file, B will be made to wait until it’s
done. The alternative is to have A lock the file and when B gets denied
access, it sits and polls until it can get it. Yuck.

There are no kernel-based synchronization objects that work at
PASSIVE_LEVEL, are there?

Ken


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Ravisankar
Pudipeddi
Sent: Thursday, June 14, 2007 11:30 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] IO with resource held?

I think this is worth clarifying, so I will have to continue this
thread.

There are ways to synchronize i/o to a file stream over a single handle:
synchronous i/o accomplishes exactly that, where the i/o manager uses an
exclusive lock associated with that file object. But i/o is not
filtering the FS as well, so no deadlocks occur.

To emphasize this again: the concern is not about i/o completion, the
concern is the nature of the exclusive lock here.

If we assume that the create was directed below the filter (which is a
necessary condition), and the resource was tied to the specific file
object it will be safe, as long as the filter - which is also seeing
other i/o go by - doesn’t attempt to re-aquire the same resource
exclusively again. So while one can get this right, it requires a lot of
careful coding & testing. This is why the generic advice would always be
to avoid holding locks across i/o.

Ravi


From: xxxxx@lists.osr.com on behalf of Ken Cross
Sent: Tue 6/12/2007 6:47 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] IO with resource held?

Yep. FWIW, I told her don’t do it. But it’s not obvious.

Certainly, user and normal kernel APCs are disabled by
FltAcquireResourceExclusive, but special kernel APCs aren’t. So why
would a
kernel-created I/O request not work? For fun, I tried it and it works
fine with simple tests.

But I don’t think we can *guarantee* that it will never deadlock.
Hence: unless some authority says it’s OK, it ain’t.

Too bad. AFAIK, there’s no synchronization mechanism that works at
PASSIVE_LEVEL to let you synchronize access (i.e., put a kernel
thread in a wait state) for things like a file.

Ken

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Monday, June 11, 2007 7:59 PM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] IO with resource held?

I was not assuming anything about the implementation of these APIs. The
Zw APIs expressly allow the caller to provide an APC routine and that is
an ordinary kernel APC. *IF* these routines do not rely upon kernel
APCs internally (something I would think is inherently unsafe to assume)
then special kernel APCs would be sufficient.

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com http:</http:>


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com


Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com