very stuck with a bug - very long and involved

OSR_Community_User · June 16, 2000, 1:25pm

Acquring the resource works only because it happens to be the one FAT
uses on that path. Other FS may require more/different synchronization.
Reaching in and grabbing sychronization resources by hand is never going
to be the right thing to do.

-----Original Message-----
From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
Sent: Friday, June 16, 2000 8:43 AM
To: File Systems Developers
Subject: [ntfsd] RE: very stuck with a bug - very long and involved

RE: [ntfsd] RE: very stuck with a bug - very long and involved>Please,
no
one implement/ship the one that pre-acquires the resource. This

will not be correct for general filesystems.

What if the caller will also set TopLevelIrp to FSP_TOP_LEVEL_IRP?
Will it still be wrong? Looks like Mm does exactly this when requesting
file
growth from CreateFileMapping/NtCreateSection.

Max

You are currently subscribed to ntfsd as: xxxxx@exchange.microsoft.com
To unsubscribe send a blank email to $subst(‘Email.Unsub’)

OSR_Community_User · June 16, 2000, 6:19pm

Surely, I’m speaking on Max Lyadvinsky’s suggestion to preacquire the FCB
MainResource exclusively before calling ZwQueryInformationFile.
Daniel from MS said that this is dirty. So, I suggested to use the Max’s
idea
but with setting TopLevelIrp to the appropriate value to let FSD know that
resources are acquired.
Will it still be dirty way? Am I right that calling the FSD with resources
preacquired is the capability reserved for MS (for other kernel components)
only?

Max

----- Original Message -----
From: Pavel Hrdina
To: File Systems Developers
Sent: Friday, June 16, 2000 8:14 PM
Subject: [ntfsd] RE: very stuck with a bug - very long and involved

> But this is only when the file is already preacquired by
> FsRtlAcquireFileExclusive. Without pre-acquision of
> FSD resources this must be considered as too dirty method.
>
> Paul
>
> > -----Pùvodní zpráva-----
> > Od: Maxim S. Shatskih [SMTP:xxxxx@storagecraft.com]
> > Odesláno: 16. èervna 2000 17:43
> > Komu: File Systems Developers
> > Pøedmìt: [ntfsd] RE: very stuck with a bug - very long and involved
> >
> > RE: [ntfsd] RE: very stuck with a bug - very long and involved>Please,
no
> > one implement/ship the one that pre-acquires the resource. This
> > >will not be correct for general filesystems.
> >
> > What if the caller will also set TopLevelIrp to FSP_TOP_LEVEL_IRP?
> > Will it still be wrong? Looks like Mm does exactly this when requesting
> > file
> > growth from CreateFileMapping/NtCreateSection.
> >
> > Max
> >
> >
> > —
> > You are currently subscribed to ntfsd as: xxxxx@sodatsw.cz
> > To unsubscribe send a blank email to $subst(‘Email.Unsub’)
>
> —
> You are currently subscribed to ntfsd as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to $subst(‘Email.Unsub’)
>

OSR_Community_User · June 16, 2000, 6:41pm

RE: [ntfsd] RE: very stuck with a bug - very long and involvedBTW, Daniel,
is there any document from MS on how to use FCB resources
(except FASTFAT source, surely)?
Is there any MS’s guidelines for FSD writers on locking?

For instance - is it OK to acquire PagingIoResource shared on page reads
and advance valid data length without acquiring PagingIoResource and thus
possibly with a collision with page read in progress? Looks like FASTFAT
does so (incrementing valid data length - MainResource exclusively,
truncating valid data length - both PagingIoResource and MainResource
exclusively. Looks like this is the only place in FASTFAT where
PagingIoResource is taken exclusively).

Am I right that the idea is that PagingIoResource must be used only for
truncations?

Max

----- Original Message -----
From: Daniel Lovinger
To: File Systems Developers
Sent: Friday, June 16, 2000 9:14 PM
Subject: [ntfsd] RE: very stuck with a bug - very long and involved

Acquring the resource works only because it happens to be the one FAT uses
on that path. Other FS may require more/different synchronization. Reaching
in and grabbing sychronization resources by hand is never going to be the
right thing to do.
-----Original Message-----
From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
Sent: Friday, June 16, 2000 8:43 AM
To: File Systems Developers
Subject: [ntfsd] RE: very stuck with a bug - very long and involved

RE: [ntfsd] RE: very stuck with a bug - very long and involved>Please, no
one implement/ship the one that pre-acquires the resource. This

will not be correct for general filesystems.
What if the caller will also set TopLevelIrp to FSP_TOP_LEVEL_IRP?
Will it still be wrong? Looks like Mm does exactly this when requesting file
growth from CreateFileMapping/NtCreateSection.
Max

You are currently subscribed to ntfsd as: xxxxx@exchange.microsoft.com
To unsubscribe send a blank email to $subst(‘Email.Unsub’)

OSR_Community_User · June 19, 2000, 3:01pm

It will never be OK to acquire a filesystem’s resources on its behalf.

Valid data length is synchronized going by the main resource. Truncation
requires both the main and paging io resources, the idea being that any
number of paging io’s can be outstanding UNLESS the file is being
truncated. If that didn’t happen we could race a paging IO with clusters
being reallocated to other files.

-----Original Message-----
From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
Sent: Friday, June 16, 2000 3:38 PM
To: File Systems Developers
Subject: [ntfsd] RE: very stuck with a bug - very long and involved

RE: [ntfsd] RE: very stuck with a bug - very long and involvedBTW,
Daniel,
is there any document from MS on how to use FCB resources
(except FASTFAT source, surely)?
Is there any MS’s guidelines for FSD writers on locking?

For instance - is it OK to acquire PagingIoResource shared on page reads
and advance valid data length without acquiring PagingIoResource and
thus
possibly with a collision with page read in progress? Looks like FASTFAT
does so (incrementing valid data length - MainResource exclusively,
truncating valid data length - both PagingIoResource and MainResource
exclusively. Looks like this is the only place in FASTFAT where
PagingIoResource is taken exclusively).

Am I right that the idea is that PagingIoResource must be used only for
truncations?

Max

----- Original Message -----
From: Daniel Lovinger
To: File Systems Developers
Sent: Friday, June 16, 2000 9:14 PM
Subject: [ntfsd] RE: very stuck with a bug - very long and involved

Acquring the resource works only because it happens to be the one FAT
uses
on that path. Other FS may require more/different synchronization.
Reaching
in and grabbing sychronization resources by hand is never going to be
the
right thing to do.
-----Original Message-----
From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
Sent: Friday, June 16, 2000 8:43 AM
To: File Systems Developers
Subject: [ntfsd] RE: very stuck with a bug - very long and involved

RE: [ntfsd] RE: very stuck with a bug - very long and involved>Please,
no
one implement/ship the one that pre-acquires the resource. This

will not be correct for general filesystems.
What if the caller will also set TopLevelIrp to FSP_TOP_LEVEL_IRP?
Will it still be wrong? Looks like Mm does exactly this when requesting
file
growth from CreateFileMapping/NtCreateSection.
Max

You are currently subscribed to ntfsd as: xxxxx@exchange.microsoft.com
To unsubscribe send a blank email to $subst(‘Email.Unsub’)

OSR_Community_User · June 19, 2000, 4:03pm

RE: [ntfsd] RE: very stuck with a bug - very long and involved>number of
paging io’s can be outstanding UNLESS the file is being

truncated. If that didn’t happen we could race a paging IO with clusters
being

Then why PagingIoResource is not acquired exclusively on file
(ValidDataLength) growth? Are races between file growth and paging reads
(which must consult the ValidDataLength value) acceptable?

Max

OSR_Community_User · June 20, 2000, 3:46pm

This is actually separate from VDL.

Two cases:

* cached - no problem exists, page is either already in memory or paging
IO is bringing correct data off of the media
* noncached - this is the intriguing case

Noncached IO goes to media, and is distinct from write-through. If the
file is cached as noncached IO is proceeding, though, we must
flush/purge the cached section overhead - and this is irrespective of
VDL. At Windows 2000, and you will see this in FAT’s write path, there
is no synchronization to prevent pagefaults from racing with the
noncached write (which may extend VDL, but it doesn’t matter) *after*
the flush/purge has occurred. This was very much by design - the
decision was to avoid introducing serialization and leave this as a case
for the application to serialize. Note the required flush/purge: mixing
noncached write with cached operations is not cheap.

Due to some interesting irreconcilable cases this has been re-evaluated
… we will be forcing coherency as of Whistler. If you look at FAT, the
change suggests itself - continue holding paging IO exclusive until the
noncached write is complete.

-----Original Message-----
From: Maxim S. Shatskih [mailto:xxxxx@storagecraft.com]
Sent: Monday, June 19, 2000 1:00 PM
To: File Systems Developers
Subject: [ntfsd] RE: very stuck with a bug - very long and involved

RE: [ntfsd] RE: very stuck with a bug - very long and involved>number of
paging io’s can be outstanding UNLESS the file is being

truncated. If that didn’t happen we could race a paging IO with
clusters
being

Then why PagingIoResource is not acquired exclusively on file
(ValidDataLength) growth? Are races between file growth and paging reads
(which must consult the ValidDataLength value) acceptable?

Max

You are currently subscribed to ntfsd as: xxxxx@exchange.microsoft.com
To unsubscribe send a blank email to $subst(‘Email.Unsub’)

OSR_Community_User · June 20, 2000, 4:34pm

RE: [ntfsd] RE: very stuck with a bug - very long and involved>This is
actually separate from VDL.

Sorry for boring you, Daniel, I have the last question:

looking in MiWaitForInPageComplete assembly code discovered the following:
if IoPageRead resulted in STATUS_END_OF_FILE, MiWaitForInPageComplete
will zero all pages in question and proceed without any errors
otherwise, MiWaitForInPageComplete will zero the tail of the page
cluster - the addresses from (StartOfCluster + Iosb.Information) to
EndOfCluster.

Is it OK for the FSD to rely on this zeroing? Is it OK to fail the paging
read with STATUS_END_OF_FILE if it starts beyound the valid data length?
Is it OK to set the Irp->IoStatus.Information to (ValidDataLength -
StartOffset) in case if the paging read start lower than ValidDataLength,
but extends beyound?
Or it is necessary for the FSD to perform the zeroing?

Max

OSR_Community_User · June 20, 2000, 6:19pm

I’ve been reminded that I need to bracket all of this discussion of “X
synchronizes Y” with the explicit disclaimer that this is simply the
default synchronization model.

There are others, and this is why the “FastIO” callbacks exist. To
repeat, it will never be OK to acquire filesystem resources directly -
this is a guaranteed way to create additional problems. The FAT driver
is not demonstrating the only way to do things, simply a reasonable way,
and as filter writers you need to be very careful not to rely on
internal behavior of the filesystems distributed on the IFS Kit.

-----Original Message-----
From: Daniel Lovinger
Sent: Monday, June 19, 2000 11:51 AM
To: File Systems Developers
Subject: [ntfsd] RE: very stuck with a bug - very long and involved

It will never be OK to acquire a filesystem’s resources on its behalf.

Valid data length is synchronized going by the main resource. Truncation
requires both the main and paging io resources, the idea being that any
number of paging io’s can be outstanding UNLESS the file is being
truncated. If that didn’t happen we could race a paging IO with clusters
being reallocated to other files.

OSR_Community_User · June 21, 2000, 5:52am

But I think some correct method to (pre)acquire FSD resources exists.

I mean using of routines (mostly written yourself because
NTOSKRNL doesn’t export them) like a FsRtlAcquireFileExclusive,
FsRtlAcquireFileForCcFlush and FsRtlAcquireFileForModWrite.
(if the FSD uses other than standard method for synchronization
it has the opportunity to specify some callbacks in its FastIoDispatch)

Ultimately this is what for eg. MmCreateSection does and
it is guaranteed this way will always be most GENERIC.

Paul

-----P?vodn? zpr?va-----
Od: Daniel Lovinger [SMTP:xxxxx@Exchange.Microsoft.com]
Odesl?no: 20. ?ervna 2000 23:32
Komu: File Systems Developers
P?edm?t: [ntfsd] RE: very stuck with a bug - very long and involved

I’ve been reminded that I need to bracket all of this discussion of “X
synchronizes Y” with the explicit disclaimer that this is simply the
default synchronization model.

There are others, and this is why the “FastIO” callbacks exist. To repeat,
it will never be OK to acquire filesystem resources directly - this is a
guaranteed way to create additional problems. The FAT driver is not
demonstrating the only way to do things, simply a reasonable way, and as
filter writers you need to be very careful not to rely on internal
behavior of the filesystems distributed on the IFS Kit.

-----Original Message-----
From: Daniel Lovinger
Sent: Monday, June 19, 2000 11:51 AM
To: File Systems Developers
Subject: [ntfsd] RE: very stuck with a bug - very long and involved

It will never be OK to acquire a filesystem’s resources on its
behalf.

Valid data length is synchronized going by the main resource.
Truncation requires both the main and paging io resources, the idea being
that any number of paging io’s can be outstanding UNLESS the file is being
truncated. If that didn’t happen we could race a paging IO with clusters
being reallocated to other files.