I am trying to identify the file type by reading file header and matching the magic number from the header. For reading the file header I am using FltReadFile with flags FLTFL_IO_OPERATION_DO_NOT_UPDATE_BYTE_OFFSET | FLTFL_IO_OPERATION_NON_CACHED | FLTFL_IO_OPERATION_PAGING | FLTFL_IO_OPERATION_SYNCHRONOUS_PAGING. My requirement is that the read should be from disk and not from cache. Also it should not populate the cache.
This works fine except one scenario where FltReadFile call returns success but data in the buffer is garbage. If I retry the same second time it works well. Not sure why it returns garbage data first time and correct data second time? Because of this I am not able to identify the file type correctly.
The scenario is as follows…
An application downloads the file from cloud to local disk in some temp folder with some random name. It then moves the file to required location with original file name as that of cloud. After that our driver tries to identify the file type of this newly arrived file and fails because of above issue.
BTW I also tried just using FLTFL_IO_OPERATION_DO_NOT_UPDATE_BYTE_OFFSET | FLTFL_IO_OPERATION_NON_CACHED and it works fine. Still I want to know why FLTFL_IO_OPERATION_PAGING | FLTFL_IO_OPERATION_SYNCHRONOUS_PAGING are causing issues in this specific scenario?
> Still I want to know why FLTFL_IO_OPERATION_PAGING |
FLTFL_IO_OPERATION_SYNCHRONOUS_PAGING are causing issues in this specific
scenario?
Well to start with this will tell the FSD to ignore its caches. So you
might be reading from disk before anything was written. I am surprised that
you get garbage however, in my experience NTFS is pretty good at VDL
management (annoyingly so). Without those flags there is an implied flush
before the read gets to the disk.
You have a solution (IMO the correct one) so I’d suggest you move on, but if
you really care I would poke at the VDL with windbg and see. You might also
try making your read look more like it came from Mm (the most obvious being
to use an MDL not an address).
In general I avoid Paging reads and writes unless I have no other option.
Paging writes, which notwithstanding the previous I have used a bit, are
extraordinarily tricky to get right.
If the file was created and written then the data went into the cache. The cache policy for a file might be write-through as in case of removable media, in that case data on the disk and in the cache are synchronized before write IO completes. If the cache policy is not write-through the data in the cache is not synchronized with the data on the disk.
When FLTFL_IO_OPERATION_NON_CACHED w/o FLTFL_IO_OPERATION_PAGING IO was issued the file system driver synchronized the cache with the data on disk by calling CcFlushCache OR filled data from the cache which is okay as NON_CACHED flag is just an advise to a file system driver, it can ignore it.
When FLTFL_IO_OPERATION_PAGING IO was issued the file system driver read data from the disk, it is not allowed to call CcFlushCache when processing paging IO so no attempt to synchronize cache with the disk was made.
Slava, your explanation looks relevant as we never experienced this issue for files on removable media.
So now I have two solutions…
Use FLTFL_IO_OPERATION_NON_CACHED and do not use FLTFL_IO_OPERATION_PAGING. But then my initial requirements does not fulfill as IO may not go to disk and may populate cache as FLTFL_IO_OPERATION_NON_CACHED is just a request.
Keep my code as is, additionally flush data on disk by calling CcFlushCache whenever new file gets created and written completely. So irrespective of write-through policy data will get written on disk and when we read it for file type check we will get correct result.
Can anyone suggest which will be the better way to go?
Unless you own the cache (e.g., you are a file system or isolation filter) you cannot safely call CcFlushCache. But if all you want to do is make sure data gets flushed, FO_WRITE_THROUGH is enough - it won’t permit dirty cached data. Note that this will not *purge* the cache. You can try the IRP_MN_FLUSH_AND_PURGE and hope the underlying FSD supports it (the FAT sample doesn’t, for example.)