Avoiding FAT32 Lazy Writer

NTFSD
1

Hi,

I encountered a mild “inconvenience” while working with FAT32 formatted storage devices:

In one of the projects I’m involved in, I implemented a file system inside a file whom I’m creating on the storage device (not that uncommon). The file itself occupies most of the usable space of the storage device and the user “sees” only my mounted file system.

When the user tries to safely remove the device, it triggers FAT’s lazy writer(s) and the remove operation takes forever (well, not forever, but much longer than I’d want it to take). In other case, if the user surprise removes the device, she sees the “delayed write failed” pop-up, indicating that the file wasn’t properly flushed.

I want to emphasize that I’m obligated to the “original” underlying file system of the storage device, hence if it comes formatted as FAT32, I can’t change it to something more “comfortable” like NTFS and correct my file’s valid size.

Due to the fact that I’m entirely controlling the used and unused sectors of this device (at least on systems having my drivers installed, etc.), can I somehow “disable” FAT’s LW, for this specific case (I’d happily have them working in the background when my file is already initialized, as I don’t want the user to entirely lose her information on surprise removal cases).

I tried zero-ing the file myself and it still takes too much time (and why wouldn’t it?), I don’t want to mimic surprise removal on safe removal (according to fastfat’s sample, the actual flush and cache purge are triggered on IRP_MN_QUERY_REMOVE_DEVICE), at least not in PnP terms and I’d rather prefer preserving its semantics (for example, according to fastfat’s sample, marking it as write protected will prevent the flush), but I really can’t think of something better.

If I could, somehow, “mark” this file so that it could avoid the flush, only in the scope of its initialization, it could be awesome.

I’d greatly appreciate any suggestions, workarounds and ideas.

Thanks in advance,
Leonid.

P.S.,
If I’m not mistaken, exFAT also employs LW, so although I can keep the “FAT”-ness of the device, re-formatting it to exFAT won’t help, right?

2

You’re doing cached I/O to your file, correct? If so, this would be the
expected behavior. FAT can’t let the drive be safely removed until the cache
is flushed and on surprise removal you’re going to get the pop up as the
lazy writes fail (to be pedantic, FAT doesn’t have a lazy writer…FAT
writes to the cache and the cache manager performs the lazy write process).
Even if you *could* somehow figure out how to turn the flush off on remove,
that would lead to data loss for the user.

Performing non-cached I/O to the file would probably make the delay go away,
but you might find the performance unacceptable under normal operation (you
would have to test to be sure). Do you ever send your own flushes to the
underlying file? You might need to do that periodically based on certain
events in order to avoid this long delay on remove. Making all of your I/O
write through might also solve the problem, but that has some of its own
issues as well (mostly performance related).

-scott

–
Scott Noone
Consulting Associate and Chief System Problem Analyst
OSR Open Systems Resources, Inc.
http://www.osronline.com

wrote in message news:xxxxx@ntfsd…

Hi,

I encountered a mild “inconvenience” while working with FAT32 formatted
storage devices:

In one of the projects I’m involved in, I implemented a file system inside a
file whom I’m creating on the storage device (not that uncommon). The file
itself occupies most of the usable space of the storage device and the user
“sees” only my mounted file system.

When the user tries to safely remove the device, it triggers FAT’s lazy
writer(s) and the remove operation takes forever (well, not forever, but
much longer than I’d want it to take). In other case, if the user surprise
removes the device, she sees the “delayed write failed” pop-up, indicating
that the file wasn’t properly flushed.

I want to emphasize that I’m obligated to the “original” underlying file
system of the storage device, hence if it comes formatted as FAT32, I can’t
change it to something more “comfortable” like NTFS and correct my file’s
valid size.

Due to the fact that I’m entirely controlling the used and unused sectors of
this device (at least on systems having my drivers installed, etc.), can I
somehow “disable” FAT’s LW, for this specific case (I’d happily have them
working in the background when my file is already initialized, as I don’t
want the user to entirely lose her information on surprise removal cases).

I tried zero-ing the file myself and it still takes too much time (and why
wouldn’t it?), I don’t want to mimic surprise removal on safe removal
(according to fastfat’s sample, the actual flush and cache purge are
triggered on IRP_MN_QUERY_REMOVE_DEVICE), at least not in PnP terms and I’d
rather prefer preserving its semantics (for example, according to fastfat’s
sample, marking it as write protected will prevent the flush), but I really
can’t think of something better.

If I could, somehow, “mark” this file so that it could avoid the flush, only
in the scope of its initialization, it could be awesome.

I’d greatly appreciate any suggestions, workarounds and ideas.

Thanks in advance,
Leonid.

P.S.,
If I’m not mistaken, exFAT also employs LW, so although I can keep the
“FAT”-ness of the device, re-formatting it to exFAT won’t help, right?

3

> I want to emphasize that I’m obligated to the “original” underlying file system of the storage device,

hence if it comes formatted as FAT32, I can’t change it to something more “comfortable” like NTFS
and correct my file’s valid size.

  1. PnP EjectionRelations or such can help
    OR
  2. write the disk (not FS) filter driver with tiny FAT parser in it. When the filter starts, it runs the FS parser over the new disk, looks for your special file, and, if it is present - takes its runlist and then redirects all requests from top against this runlist. Disk size queries must also be filters.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

4

Thanks for the answers.

Scott, I trigger the flush by setting the end of the file (in order for it to be made in the size of the whole storage device), hence on remove the FS tries to fill the whole file with zeros. Even if the user worked with the mounted FS, in most of the case the “valid size” of the file is still much smaller than its end of file. I do think that converting any I/O to non-cached I/O may greatly decrease the performance, which may be unacceptable as well (I’ll give it a try, though).

Actually, if I perform only non-cached I/O for this “initial instance” of the device, do you think that there won’t be any actual work during the flash (== because the cache will be empty), even though I did set its end of file to a greater size?

Maxim, are you suggesting me to ‘fake’ the EjectionRelations so that the file won’t support trivial safe remove? Unfortunately, although your second solution is plausible, it’ll be a bit of an overkill for this task, as I don’t have any disk filters (only virtual bus + bus enumerated devices, with a little help from a ‘legacy’ FS filter).

5

>Scott, I trigger the flush by setting the end of the file (in order for it

to be made in the size of the whole storage device), hence on >remove the
FS tries to fill the whole file with zeros.

Ah, my bad. I read your post too quickly and missed that we’re talking
specifically about a move of EOF causing the zeroes to be written. What
you’re really asking for in this case is a way to move VDL to avoid the
zeroes being written out to the cache. Worth nothing that this *does* pose a
potential security risk, the reason why we zero that data is so that garbage
disk data doesn’t become accessible from the file.

Starting in Win7, FAT does have support for setting
FileValidDataLengthInformation, which would probably do what you want. That
should extend the file but not actually zero it (see the docs for the Win32
API SetFileValidData).

-scott

–
Scott Noone
Consulting Associate and Chief System Problem Analyst
OSR Open Systems Resources, Inc.
http://www.osronline.com

wrote in message news:xxxxx@ntfsd…

Thanks for the answers.

Scott, I trigger the flush by setting the end of the file (in order for it
to be made in the size of the whole storage device), hence on remove the FS
tries to fill the whole file with zeros. Even if the user worked with the
mounted FS, in most of the case the “valid size” of the file is still much
smaller than its end of file. I do think that converting any I/O to
non-cached I/O may greatly decrease the performance, which may be
unacceptable as well (I’ll give it a try, though).

Actually, if I perform only non-cached I/O for this “initial instance” of
the device, do you think that there won’t be any actual work during the
flash (== because the cache will be empty), even though I did set its end of
file to a greater size?

Maxim, are you suggesting me to ‘fake’ the EjectionRelations so that the
file won’t support trivial safe remove? Unfortunately, although your second
solution is plausible, it’ll be a bit of an overkill for this task, as I
don’t have any disk filters (only virtual bus + bus enumerated devices, with
a little help from a ‘legacy’ FS filter).