Hello all!
I want to cache temporarily all writes to specific files in memory (but returning STATUS_SUCCESS to requesters) and then flush them to the disk later. Am I getting in too much trouble?
Let me contextualize my problem: I have experience developing drivers for the storage stack, but none for FS (besides playing a little bit with minispy sample, but that's not something that I would call experience)
First of all, I have to say that this solution is targeted to only our enterprise systems, not to general public, so we have bigger control over system usage, and some of the not-so-great assumptions I made are kinda justified.
We have a full disk encryption solution that is a disk filter, and all it knows are disk blocks (offsets and length). I am currently working on a new feature that is a live-backup of the system volume (HarddiskVolume1), just like Volume Snapshot (VSS), but VSS is file-based, while our snapshot is a full encrypted mirror of C:\ partition.
Since my driver has absolutely no knowledge of files, I had to think about a strategy to not corrupt data/put my OS on an inconsistent state, because while my copy thread is mirroring C:\ from beginning to end (say 50 GB), system has to keep running, and new writes are coming all the time, even in portions of C:\ that I had already mirrored.
I developed a FIFO queue of sectors which are being written on C:\ and that were already backuped, in order to mirror them again to our backup later. For example: if my backup is at 50% progress (say 25 GB) and I create a new file on desktop (which may be roughly at 3 GB offset - in this particular example), I want my backup to have this new file.
I have to finish my backup sometime, but because storage cannot itself map sectors to files, I risk to corrupt data if I stop feeding my queue at the wrong time
(between 2 sectors that belong to the same file, or without updating NTFS Metadata that belonged to that file)
To avoid this, I developed the following criteria:
If my queue does not have any new members in 1 minute, it is safe to finish the backup process (obviously if it already passed 100% of 50 GB length). So I have a thread with a KTIMER that handles this, and the whole backup worked like a charm! 
The problem is that these tests were made on test VMs, which are very different from our production OSes, and because of numerous monitoring solutions, I am struggling to get a full minute without growing my queue. Even lowering my timer period to 25 seconds instead doesn't seem to solve it.
If I lower it more, I think the risk of data corruption outgrows the benefit of having a snapshot. I read somewhere that Cc flushes 1/8th of its cached pages every second, and besides not knowing if this period is true or not, I decided for 1 minute to get a safe margin.
When I played with minispy to track the most-writed files, most of them were .log files.
Some of them I can disable, but many others are from 3rd-party vendors, and it's not possible.
So I thought of caching myself these logs for this period on RAM, but returning SUCCESS to FS, effectively tricking it, and when my backup has finished, I will flush this memory to them.
Indeed there is a race condition in this approach, but I don't mind losing a line or two of the logs. I just don't want to mess with binary files (neither deadlocking my system, obviously). Is this idea possible/doable?
File size is not a problem, but rather its writing frequency.
I thought about redirecting these files to another volume (say D:) but I did some research about this, and it looked pretty complex (And unfortunately NameChanger sample just works inside the same volume), so I gave up.
If you have a different approach that could resolve my problem, I can give up on this caching/temp ramdisk idea as well, I just need a manner to put my system on "quiet mode" temporarily with these logs and then writing them transparently to the applications.
Any ideas? (And sorry if any of my questions were super FS beginner/101 haha)
PS: going to VSS is not an option, because encryption is a must, and you have to write custom APIs to integrate with it, to disable file writing temporarily (pretty much what I am trying to do) in order to correctly backup the files. MS stuff has it (like registry and SQL Server DBs) but we can't write them for every single application we run, and it's not possible with 3rd party tools we run