Virtualization with mapped file

NTFSD
1

Greetings everyone,

I am writing a file virtualization minifilter, using copy-on-write method. The driver works like this.
When applications try to change some files, it will be redirected to COWSTORAGE instead. The process
is as follow:

  1. Original file will be copied to COWSTORAGE.
  2. File operation will be issued there and then return with status FLT_PREOP_COMPLETE.
  3. File marked as changed.
    (For Async operation, the operation will be pended and completed later.)

This work with IRP operation but it doesn’t work for memory mapped files.

A file on C:\Test.txt contains “aaaa”. A user mode application open the file and write “1234” to the file.
Since the user mode apps changed the file, the file will be copied into C:\COWSTORAGE\Test.txt and the operation
performed there. Now the file on COWSTORAGE contains “1234”. Using the same file handle, the user mode app
create file mapping with it. When reading the memory pMem returned by MapViewOfFile, it shows “aaaa” instead of “1234”.

Now, I’m stuck with this. HOW to redirect a mapped file operation, to map the file on COWSTORAGE instead
of the actual file?

Further testing, when changing the memory contents pMem[0] = ‘x’, the memory mapped file will display
“xaaa” in its buffer while with FileRead, the buffer will contains “1234”. Of course this will happens since the memory is
mapped to C:\Test.txt while FileRead/FileWrite operation is mapped to C:\COWSTORAGE\Test.txt.

Now the file contents is out of sync. Issuing file write with different contents still doesn’t change the memory
mapped file which will still with “xaaa” as it contents. When closing all handles and stopping the minifilter,
I can see that the original file C:\Test.txt now contains “xaaa” while the file on COWSTORAGE now contains
“1234” or whatever last IRP_MJ_WRITE issued to it. It work flawlessly when using notepad. When opening
the file with notepad, made some changes and then save it, the file C:\Test.txt will remains untouched and
all changes can be viewed on C:\COWSTORAGE\Test.txt.

How to solve this problem and/or what is the correct method to solve this?
Thanks.

Syahmi

2

From your email, I am not clear about how the create for original file was
redirected to your COW file by the driver. To figure out what is going on,
try to see how the driver is missing the create for the file object passed
in the paging I/O.

There are two ways to achieve COW in the storage, at volume/disk level and
at FS level. Implementing it at FS level is non trivial, primarily because
of a rich mechanism that NTFS has. At volume level it is much simpler to
implement, VSS providers is a good example.

Also see,
http://msdn.microsoft.com/en-us/library/aa940813(v=winembedded.5).aspx

HTH,
Johri

On Mon, Jun 10, 2013 at 2:40 PM, Muhammad Syahmi wrote:

> Greetings everyone,
>
> I am writing a file virtualization minifilter, using copy-on-write method.
> The driver works like this.
> When applications try to change some files, it will be redirected to
> COWSTORAGE instead. The process
> is as follow:
> 1. Original file will be copied to COWSTORAGE.
> 2. File operation will be issued there and then return with status
> FLT_PREOP_COMPLETE.
> 3. File marked as changed.
> (For Async operation, the operation will be pended and completed later.)
>
> This work with IRP operation but it doesn’t work for memory mapped files.
>
> A file on C:\Test.txt contains “aaaa”. A user mode application open the
> file and write “1234” to the file.
> Since the user mode apps changed the file, the file will be copied into
> C:\COWSTORAGE\Test.txt and the operation
> performed there. Now the file on COWSTORAGE contains “1234”. Using the
> same file handle, the user mode app
> create file mapping with it. When reading the memory pMem returned by
> MapViewOfFile, it shows “aaaa” instead of “1234”.
>
> Now, I’m stuck with this. HOW to redirect a mapped file operation, to map
> the file on COWSTORAGE instead
> of the actual file?
>
> Further testing, when changing the memory contents pMem[0] = ‘x’, the
> memory mapped file will display
> “xaaa” in its buffer while with FileRead, the buffer will contains “1234”.
> Of course this will happens since the memory is
> mapped to C:\Test.txt while FileRead/FileWrite operation is mapped to
> C:\COWSTORAGE\Test.txt.
>
> Now the file contents is out of sync. Issuing file write with different
> contents still doesn’t change the memory
> mapped file which will still with “xaaa” as it contents. When closing all
> handles and stopping the minifilter,
> I can see that the original file C:\Test.txt now contains “xaaa” while the
> file on COWSTORAGE now contains
> “1234” or whatever last IRP_MJ_WRITE issued to it. It work flawlessly when
> using notepad. When opening
> the file with notepad, made some changes and then save it, the file
> C:\Test.txt will remains untouched and
> all changes can be viewed on C:\COWSTORAGE\Test.txt.
>
> How to solve this problem and/or what is the correct method to solve this?
> Thanks.
>
> Syahmi
>
> —
> NTFSD is sponsored by OSR
>
> OSR is hiring!! Info at http://www.osr.com/careers
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

3

Hi Johri,

Thanks for the input. FBWF is our temporary solution until our own driver is ready. We are looking for FS level right now and not on sector/volume level.

Each file will get its stream context and stream handle context during file create. When IRP_MJ_WRITE (or any IRP that changes the data) comes, the PreWrite will copy the file to COWSTORAGE and set FileChanges = TRUE on Stream context. This flag tells the driver that the file has been modified. Then all next subsequent IRP will be redirected to new file which resides on COWSTORAGE. This works perfectly for normal file operation, but for file mapping it is not.

The user mode application below might explain the issue.

//
//? File is created normally. The driver will create stream context/stream handle context for this
//? file. The file already exists on current folder, but doesn’t exists yet on COWSTORAGE.
//? File contains “aaaa”.
//

??? hFile = CreateFile( TEXT(“fileop_test.txt”), GENERIC_READ | GENERIC_WRITE,
??? ??? FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
??? ??? NULL,
??? ??? OPEN_EXISTING,
??? ??? FILE_ATTRIBUTE_NORMAL,
??? ??? NULL);

??? if (INVALID_HANDLE_VALUE != hFile) {

??? ??? HANDLE hFileMap;

//
//?? Now we are writing the file using WriteFile API. This will emit the IRP_MJ_WRITE callback.
//?? My driver will perform file copy to COWSTORAGE folder on PreWrite. Then roll own IRP
//?? and issue the IRP_MJ_WRITE to new file on COWSTORAGE instead.
//
??? ??? printf(“Writing file: 1234\n”);
??? ??? WriteFile( hFile, “1234\0”, 5, &dwWritten, NULL );

??? ??? SetFilePointer( hFile, 0, 0, FILE_BEGIN );

//
//?? HERE the interesting part. All the IRP operation now will be redirected to the new file on COWSTORAGE but
//?? the CreateFileMapping API isn’t. It will use the file on current folder instead.
//
??? ??? printf(“Creating File Map\n”);
??? ??? hFileMap = CreateFileMapping( hFile, NULL, PAGE_READWRITE, 0, 5, NULL );

??? ??? if (INVALID_HANDLE_VALUE != hFileMap)
??? ??? {
??? ??? ??? printf(“Mapping file view\n”);
??? ??? ??? char* pMem = (char*) MapViewOfFile( hFileMap,
??? ??? ??? ??? FILE_MAP_WRITE,
??? ??? ??? ??? 0,
??? ??? ??? ??? 0,
??? ??? ??? ??? 5 );

??? ??? ??? if (pMem)
??? ??? ??? {
??? ??? ??? ??? char ReadBuff[32];

??? ??? ??? ??? printf(“Change memory: [0] = ‘x’\n”);

??? ??? ??? ??? __try {

??? ??? ??? ??? ??? pMem[0] = ‘x’;

??? ??? ??? ??? } __except (EXCEPTION_EXECUTE_HANDLER) {

??? ??? ??? ??? ??? printf(“Exception occurred!\n”);
??? ??? ??? ??? }

??? ??? ??? ??? printf(“Reading file: Expected = x234\n”);
??? ??? ??? ??? _getch();

??? ??? ??? ??? ReadFile( hFile, ReadBuff, 5, &dwWritten, NULL );
??? ??? ??? ??? printf(" Map : %s\n", pMem);??? ??? // Show: xaaa (Incorrect)
??? ??? ??? ??? printf(" Read: %s\n", ReadBuff);??? // Show: x234 (This is desired result for both Map and Read)

??? ??? ??? ??? //
??? ??? ??? ??? //
??? ??? ??? ??? //

??? ??? ??? ??? printf(“Setting file pointer to 0\n”);
??? ??? ??? ??? SetFilePointer( hFile, 0, 0, FILE_BEGIN );

//
//? Write the file again!
//
??? ??? ??? ??? printf(“Writing ‘4321’ to file\n”);
??? ??? ??? ??? WriteFile( hFile, “4321\0”, 5, &dwWritten, NULL );

??? ??? ??? ??? SetFilePointer( hFile, 0, 0, FILE_BEGIN );

??? ??? ??? ??? printf(“Reading file: Expected = 4321\n”);

??? ??? ??? ??? ReadFile( hFile, ReadBuff, 5, &dwWritten, NULL );
??? ??? ??? ??? printf(" Map : %s\n", pMem); // Still shows “xaaa”. Expected result is 4321.
??? ??? ??? ??? printf(" Read: %s\n", ReadBuff); // Shows 4321

??? ??? ??? ??? //
??? ??? ??? ??? //
??? ??? ??? ??? //

??? ??? ??? ??? printf(“Unmap view of file\n”);
??? ??? ??? ??? _getch();

??? ??? ??? ??? UnmapViewOfFile( pMem );
??? ??? ??? }

??? ??? ??? printf(“Closing hFileMap\n”);
??? ??? ??? _getch();

??? ??? ??? CloseHandle( hFileMap );
??? ??? }

??? ??? printf(“Closing hFile\n”);
??? ??? _getch();

??? ??? CloseHandle( hFile );
??? }

After running and stop the filter driver, I observed that the file contents on real path has changed to “xaaa”. This shouldn’t happens as it will defeat the purpose of this driver, the original file should remains untouched. On COWSTORAGE, the file content contains 4321.

I only got IRP_MJ_READ/WRITE or FASTIO_READ/WRITE for ReadFile/WriteFile operation. For CreateFileMapping, I only got IRP_MJ_XXX_FOR_SECTION_SYNCHRONIZATION but not IRP_MJ_READ since I believe the file already on system cache.

This question might already asked by someone before, sorry if I was repeating the question again.
How do I handle file mapping correctly?

How do I cancel the file mapping write operation?? It seems I can’t find equivalent API for this. I don’t want the file outside COWSTORAGE (the original file) got changed.

On PreWrite, I even do this… and then execute the test program above, uninstall the driver and rebooted the system, then opened the original file. The original contents has been changed to “xaaa”.

??? status = CtxFindOrCreateStreamContext( Data, FALSE, &StreamCtx, NULL );

??? if (!NT_SUCCESS( status ) || StreamCtx == NULL) {

??? ??? goto PreWriteCleanup;
??? }

??? CtxAcquireResourceShared( StreamCtx->Resource );
???
??? DataChanged = StreamCtx->Changes.DataChanged;

??? CtxReleaseResource( StreamCtx->Resource );

??? {
//
//? THIS
//
??? ??? Data->IoStatus.Status = STATUS_SUCCESS;
??? ??? Data->IoStatus.Information = Param->Write.Length;

??? ??? cbStatus = FLT_PREOP_COMPLETE;
??? ??? goto PreWriteCleanup;
??? }

I hope someone can shine some light on this one for me. Thanks.
Syahmi.

4

I don’t understand your step 2 and why you’re dealing with IRP_MJ_WRITE/memory_mapped operations.

Since you duplicate the original file to COWSTORAGE, I expect you return STATUS_REPARSE and I/O Manager opens your COWSTORAGE copy. All read/write operations are done on the copy, so why do you need a special handling of WRITE requests? I don’t need that and I’m doing the same thing as you. Or is COWSTORAGE a ramdisk, special storage, or something?

As for FWBW, I thought it’s XP Embedded feature only, based on sector-level + file-system driver and it shouldn’t be included in Home/Pro (= non-embedded OSes).

Petr

5

Hi Petr

I don’t understand your step 2 and why you’re dealing with IRP_MJ_WRITE/memory_mapped operations.

Since you duplicate the original file to COWSTORAGE, I expect you return STATUS_REPARSE and I/O Manager opens your COWSTORAGE copy. All read/write operations are done on the copy, so why do you need a special handling of WRITE requests? I don’t need that and I’m doing the same thing as you. Or is COWSTORAGE a ramdisk, special storage, or something?

My intention is to make a virtualization driver using copy-on-write approach.
When the files already on COWSTORAGE, of course I will simply return STATUS_REPARSE. As we know copying files is an expensive operation and will decrease performance drastically especially for larger files. So when the files isn’t yet on COWSTORAGE, i’ll just let it open the original files and copy it to COWSTORAGE only when there is write request, and then continuing performing the write operation on COWSTORAGE instead. COWSTORAGE is simply a directory on same volume containing modified files.

I found the correct term after posting, its “cache coherency” right? I searched over this subject on osr and its led me to SFO approach, but doesn’t know yet how to implement it. I already tried read/write only on paging io? path and non-cached operation, but the system still complaining about file corruption and application crash.

Syahmi