Hide minifilter write operations

Mickey_H · July 5, 2017, 10:46am

Hi,

I am pretty new to minifilter drivers so please forgive me if I am missing anything obvious.
I am trying to write a minifilter driver which handles my stub files. My stub files are empty files with the original logical size (sparse file with a hole).
When some application tries to access a stub, the minifilter driver intercepts the pre-read operation and my service starts to download the real file. Once file is downloaded, the pre-read can continue.

My problem is that “Windows defender” (MsMpEng.exe) detects my service writing to the file, and for some reason rewrites to this file when the service closes it.
This write operation by windows defender is problematic for my application.

I tried to avoid it by making the write call from the service to the minifilter directly (through filter communication port), in the minifilter I use “FltWriteFile”, but that didn’t help.
Windows defender still detects my minifilter write operations even though my minifilter’s altitude is lower than “WdFilter”.

Any idea why windows defender re-writes the entire file?
Any idea how I can hide my write operations from windows defender?

Thanks,
Mickey

Scott_Noone_OSR · July 5, 2017, 12:41pm

Can you print the call stack for the duplicate write that you’re seeing?

Also, how do you fill the read buffer once the file population is complete?
Do you really just pend the callback data and send it later or are you doing
something with the original read buffer?

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntfsd…

Hi,

I am pretty new to minifilter drivers so please forgive me if I am missing
anything obvious.
I am trying to write a minifilter driver which handles my stub files. My
stub files are empty files with the original logical size (sparse file with
a hole).
When some application tries to access a stub, the minifilter driver
intercepts the pre-read operation and my service starts to download the real
file. Once file is downloaded, the pre-read can continue.

My problem is that “Windows defender” (MsMpEng.exe) detects my service
writing to the file, and for some reason rewrites to this file when the
service closes it.
This write operation by windows defender is problematic for my application.

I tried to avoid it by making the write call from the service to the
minifilter directly (through filter communication port), in the minifilter I
use “FltWriteFile”, but that didn’t help.
Windows defender still detects my minifilter write operations even though my
minifilter’s altitude is lower than “WdFilter”.

Any idea why windows defender re-writes the entire file?
Any idea how I can hide my write operations from windows defender?

Thanks,
Mickey

Mickey_H · July 5, 2017, 3:58pm

Thank you for your quick reply.
I believe that Windows Defender behavior is not related to my changes.
I wrote some utility that overwrites a file with a specific string. I ran it while my filter was not installed on the machine (and my service is disabled).
I see in Process Monitor that windows defender rewrites my file after is gets closed:

10:39:53.1444234 PM xappend.exe 5524 CreateFile E:\Downloads\Mickey.jpg SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: None, AllocationSize: n/a, OpenResult: Opened
10:39:53.1444838 PM xappend.exe 5524 WriteFile E:\Downloads\Mickey.jpg SUCCESS Offset: 0, Length: 15, Priority: Normal
10:39:53.1445793 PM xappend.exe 5524 ReadFile E:\Downloads\Mickey.jpg SUCCESS Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
10:39:53.1573086 PM xappend.exe 5524 CloseFile E:\Downloads\Mickey.jpg SUCCESS

10:39:53.1588688 PM MsMpEng.exe 1112 CreateFileMapping E:\Downloads\Mickey.jpg FILE LOCKED WITH WRITERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READ|PAGE_NOCACHE
10:39:53.1588867 PM MsMpEng.exe 1112 QueryStandardInformationFile E:\Downloads\Mickey.jpg SUCCESS AllocationSize: 2,424,832, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False
10:39:53.1589683 PM MsMpEng.exe 1112 WriteFile E:\Downloads\Mickey.jpg SUCCESS Offset: 0, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
10:39:53.1594533 PM MsMpEng.exe 1112 ReadFile E:\Downloads\Mickey.jpg SUCCESS Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
10:39:53.1600757 PM MsMpEng.exe 1112 ReadFile E:\Downloads\Mickey.jpg SUCCESS Offset: 2,355,200, Length: 5,442, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
10:39:53.1629744 PM MsMpEng.exe 1112 ReadFile E:\Downloads\Mickey.jpg SUCCESS Offset: 61,440, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal

Regarding my read operation, I just pend the operation and resume it when the file is populated.

Scott_Noone_OSR · July 5, 2017, 5:15pm

You’re seeing a flush of cached data. Check the call stack for the write
from the context of the MsMpEng process, probably a CcFlushCache from the
file system.

This wouldn’t be Defender writing the file, it’s just that its data scan is
causing the file system to flush the cache.

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntfsd…

Thank you for your quick reply.
I believe that Windows Defender behavior is not related to my changes.
I wrote some utility that overwrites a file with a specific string. I ran it
while my filter was not installed on the machine (and my service is
disabled).
I see in Process Monitor that windows defender rewrites my file after is
gets closed:

10:39:53.1444234 PM xappend.exe 5524 CreateFile E:\Downloads\Mickey.jpg
SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: Open,
Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N,
ShareMode: None, AllocationSize: n/a, OpenResult: Opened
10:39:53.1444838 PM xappend.exe 5524 WriteFile E:\Downloads\Mickey.jpg
SUCCESS Offset: 0, Length: 15, Priority: Normal
10:39:53.1445793 PM xappend.exe 5524 ReadFile E:\Downloads\Mickey.jpg
SUCCESS Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O,
Synchronous Paging I/O, Priority: Normal
10:39:53.1573086 PM xappend.exe 5524 CloseFile E:\Downloads\Mickey.jpg
SUCCESS

10:39:53.1588688 PM MsMpEng.exe 1112 CreateFileMapping
E:\Downloads\Mickey.jpg FILE LOCKED WITH WRITERS SyncType:
SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READ|PAGE_NOCACHE
10:39:53.1588867 PM MsMpEng.exe 1112 QueryStandardInformationFile
E:\Downloads\Mickey.jpg SUCCESS AllocationSize: 2,424,832, EndOfFile:
2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False
10:39:53.1589683 PM MsMpEng.exe 1112 WriteFile E:\Downloads\Mickey.jpg
SUCCESS Offset: 0, Length: 4,096, I/O Flags: Non-cached, Paging I/O,
Synchronous Paging I/O, Priority: Normal
10:39:53.1594533 PM MsMpEng.exe 1112 ReadFile E:\Downloads\Mickey.jpg
SUCCESS Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O,
Synchronous Paging I/O, Priority: Normal
10:39:53.1600757 PM MsMpEng.exe 1112 ReadFile E:\Downloads\Mickey.jpg
SUCCESS Offset: 2,355,200, Length: 5,442, I/O Flags: Non-cached, Paging I/O,
Synchronous Paging I/O, Priority: Normal
10:39:53.1629744 PM MsMpEng.exe 1112 ReadFile E:\Downloads\Mickey.jpg
SUCCESS Offset: 61,440, Length: 32,768, I/O Flags: Non-cached, Paging I/O,
Synchronous Paging I/O, Priority: Normal

Regarding my read operation, I just pend the operation and resume it when
the file is populated.

Mickey_H · July 6, 2017, 5:32pm

You are right, thank you for your help.

I did some reading in order to understand how the cache manager works. In order to bypass it, I decided to use the FILE_FLAG_NO_BUFFERING flag when I call CreateFile in my service.
According to the documentation, when using this flag the cache manager should ignore my service write operations.

Now I do see that my service initiates non-cached IO requests, but I still have a problem.
The cache manager still writes the last 4K of the file after my service closes the file.
I suspect that this write operation is because of my service setting the end of file size (FILE_FLAG_NO_BUFFERING requires writing aligned buffers).
I use SetFileInformationByHandle in order to set the actual size.

This is what I see in Process Monitor:

11:40:34.5217380 PM xSrv.exe 3576 CreateFile E:\Downloads\test.jpg SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: Open, Options: No Buffering, Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, AllocationSize: n/a, OpenResult: Opened
11:40:34.5226546 PM xSrv.exe 3576 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 0, Length: 1,048,576, I/O Flags: Non-cached, Priority: Normal
11:40:34.5362022 PM xSrv.exe 3576 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 1,048,576, Length: 1,048,576, I/O Flags: Non-cached, Priority: Normal
11:40:34.5396911 PM xSrv.exe 3576 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,097,152, Length: 266,240, I/O Flags: Non-cached, Priority: Normal
11:40:34.5432151 PM xSrv.exe 3576 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCESS EndOfFile: 2,360,642
11:40:34.5433648 PM xSrv.exe 3576 ReadFile E:\Downloads\test.jpg SUCCESS Offset: 2,359,296, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Priority: Normal
11:40:34.5443753 PM xSrv.exe 3576 CloseFile E:\Downloads\test.jpg SUCCESS
11:40:34.7266040 PM DllHost.exe 4224 CloseFile E:\Downloads\test.jpg SUCCESS
11:40:34.7502330 PM MsMpEng.exe 1756 CreateFileMapping E:\Downloads\test.jpg FILE LOCKED WITH WRITERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READ|PAGE_NOCACHE
11:40:34.7502548 PM MsMpEng.exe 1756 QueryStandardInformationFile E:\Downloads\test.jpg SUCCESS AllocationSize: 2,424,832, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False
11:40:34.7503037 PM MsMpEng.exe 1756 WriteFile E:\Downloads\test.jpg CANCELLED Offset: 2,359,296, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal

Please note that the last write gets canceled because it causes a bug on my side.
The call stack for the last write:

0 FLTMGR.SYS FltDecodeParameters + 0x195a 0xfffff80d605146ca C:\Windows\System32\drivers\FLTMGR.SYS
1 FLTMGR.SYS FltDecodeParameters + 0x1508 0xfffff80d60514278 C:\Windows\System32\drivers\FLTMGR.SYS
2 FLTMGR.SYS FltDecodeParameters + 0x616 0xfffff80d60513386 C:\Windows\System32\drivers\FLTMGR.SYS
3 FLTMGR.SYS FltDecodeParameters + 0x3be 0xfffff80d6051312e C:\Windows\System32\drivers\FLTMGR.SYS
4 ntoskrnl.exe IoSynchronousPageWrite + 0x15c 0xfffff8029a564afc C:\Windows\system32\ntoskrnl.exe
5 ntoskrnl.exe PsGetProcessWin32Process + 0x44a 0xfffff8029a576c9a C:\Windows\system32\ntoskrnl.exe
6 ntoskrnl.exe PsChargeProcessNonPagedPoolQuota + 0xcb2 0xfffff8029a4d8112 C:\Windows\system32\ntoskrnl.exe
7 ntoskrnl.exe KeAcquireInStackQueuedSpinLock + 0x8003 0xfffff8029a512fd3 C:\Windows\system32\ntoskrnl.exe
8 ntoskrnl.exe CcSetParallelFlushFile + 0x440 0xfffff8029a4f971c C:\Windows\system32\ntoskrnl.exe
9 ntoskrnl.exe CcFlushCache + 0x16 0xfffff8029a51aad6 C:\Windows\system32\ntoskrnl.exe
10 ntoskrnl.exe FsRtlCreateSectionForDataScan + 0x297 0xfffff8029a4a4a87 C:\Windows\system32\ntoskrnl.exe
11 ntoskrnl.exe FsRtlCreateSectionForDataScan + 0x11f 0xfffff8029a4a490f C:\Windows\system32\ntoskrnl.exe
12 FLTMGR.SYS FltCreateSectionForDataScan + 0xec 0xfffff80d6054761c C:\Windows\System32\drivers\FLTMGR.SYS
13 WdFilter.sys WdFilter.sys + 0x19578 0xfffff80d5fd19578 C:\Windows\system32\drivers\WdFilter.sys
14 WdFilter.sys WdFilter.sys + 0x19030 0xfffff80d5fd19030 C:\Windows\system32\drivers\WdFilter.sys
15 FLTMGR.SYS FltDeleteStreamHandleContext + 0x146 0xfffff80d60518fb6 C:\Windows\System32\drivers\FLTMGR.SYS
16 FLTMGR.SYS FltRemoveOpenReparseEntry + 0x33f 0xfffff80d6054345f C:\Windows\System32\drivers\FLTMGR.SYS
17 FLTMGR.SYS FltDecodeParameters + 0x409 0xfffff80d60513179 C:\Windows\System32\drivers\FLTMGR.SYS
18 ntoskrnl.exe NtQueryInformationFile + 0xc20 0xfffff8029a8a8db0 C:\Windows\system32\ntoskrnl.exe

Any idea why the last 4k are written by the cache manager?

Slava_Imameev · July 6, 2017, 11:43pm

A file system driver might not call the Cache Manager to perform read-write. The Cache Manager doesn’t care about this flag.

The file is cached. File system might ignore non-cached flags for its own reasons ( and NTFS does this for some files ) or use cache to maintain file coherence in presence of memory mapped file sections.

Mickey_H · July 7, 2017, 8:12am

According to “Windows Internals” (Chapter 11, Cache Manager):
“Keep in mind that the cache manager is involved in file I/O only when a file is opened without the FILE_FLAG_NO_BUFFERING flag
and then read from or written to using the Windows I/O functions (for example, using the Windows ReadFile and WriteFile functions).”

I wrote a simple utility that writes zeros to an existing file.
It writes zeros in 4K buffers, and at the end it sets the size to be the original size of the file.
The first version of the utility opens the file without any special flags, the second version (xzero_no_buff) opens the file with the flag FILE_FLAG_NO_BUFFERING.

* First version Process Monitor output:

2:51:16.3944250 PM xzero.exe 6412 CreateFile E:\Downloads\test.jpg SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: None, AllocationSize: n/a, OpenResult: Opened
2:51:16.3944703 PM xzero.exe 6412 QueryStandardInformationFile E:\Downloads\test.jpg SUCCESS AllocationSize: 2,363,392, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False
2:51:16.3944834 PM xzero.exe 6412 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 0, Length: 4,096, Priority: Normal
2:51:16.3945298 PM xzero.exe 6412 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 4,096, Length: 4,096
…
2:51:16.4019936 PM xzero.exe 6412 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,355,200, Length: 4,096
2:51:16.4020009 PM xzero.exe 6412 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,359,296, Length: 4,096
2:51:16.4020285 PM xzero.exe 6412 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCESS EndOfFile: 2,360,642
2:51:16.4090436 PM xzero.exe 6412 CloseFile E:\Downloads\test.jpg SUCCESS
2:51:16.4094012 PM MsMpEng.exe 372 CreateFileMapping E:\Downloads\test.jpg FILE LOCKED WITH WRITERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READ|PAGE_NOCACHE
2:51:16.4094244 PM MsMpEng.exe 372 QueryStandardInformationFile E:\Downloads\test.jpg SUCCESS AllocationSize: 2,363,392, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False
2:51:17.0794587 PM System 4 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 0, Length: 1,048,576, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Very Low
2:51:17.0821091 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther
2:51:18.3802095 PM System 4 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 1,048,576, Length: 1,048,576, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Very Low
2:51:18.3829431 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther
2:51:19.7413443 PM System 4 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,097,152, Length: 266,240, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Very Low
2:51:19.7424212 PM System 4 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCESS EndOfFile: 2,360,642
2:51:19.7424416 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther

* Second version Process Monitor output:

3:04:27.8110033 PM xzero_no_buff.exe 7016 CreateFile E:\Downloads\test.jpg SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: Open, Options: No Buffering, Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: None, AllocationSize: n/a, OpenResult: Opened
3:04:27.8207308 PM xzero_no_buff.exe 7016 QueryStandardInformationFile E:\Downloads\test.jpg SUCCESS AllocationSize: 2,363,392, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False
3:04:27.8207705 PM xzero_no_buff.exe 7016 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 0, Length: 4,096, I/O Flags: Non-cached, Priority: Normal
3:04:27.8211333 PM xzero_no_buff.exe 7016 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 4,096, Length: 4,096, I/O Flags: Non-cached, Priority: Normal
…
3:04:27.9064179 PM xzero_no_buff.exe 7016 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,359,296, Length: 4,096, I/O Flags: Non-cached, Priority: Normal
3:04:27.9065758 PM xzero_no_buff.exe 7016 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCESS EndOfFile: 2,360,642
3:04:27.9066143 PM xzero_no_buff.exe 7016 ReadFile E:\Downloads\test.jpg SUCCESS Offset: 2,359,296, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Priority: Normal
3:04:27.9086864 PM xzero_no_buff.exe 7016 CloseFile E:\Downloads\test.jpg SUCCESS
3:04:27.9090498 PM MsMpEng.exe 372 CreateFileMapping E:\Downloads\test.jpg FILE LOCKED WITH WRITERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READ|PAGE_NOCACHE
3:04:27.9090725 PM MsMpEng.exe 372 QueryStandardInformationFile E:\Downloads\test.jpg SUCCESS AllocationSize: 2,363,392, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False
3:04:27.9091353 PM MsMpEng.exe 372 ReadFile E:\Downloads\test.jpg SUCCESS Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
…
3:04:30.8326217 PM System 4 CreateFileMapping C:\test\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther
3:04:30.8445425 PM System 4 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCESS EndOfFile: 2,359,296
3:04:30.8445975 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther
3:04:32.1532605 PM System 4 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,359,296, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Very Low
3:04:32.1566545 PM System 4 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCESS EndOfFile: 2,360,642
3:04:32.1567095 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther

As you can see, in the first version the entire file is written by the cache manager, while in the second version only the last 4KB frame is written.
I suspect that the last frame is being written because I set the file size.

How can I make the SetEndOfFileInformationFile a non-cached operation?

Scott_Noone_OSR · July 7, 2017, 9:27am

You don’t appear to have deleted the file between the two runs, note that in
your second version the file already has a non-zero AllocationSize and
EndOfFile. So it would appear that you’re mixing cached I/O and non-cached
I/O to the same file.

Try writing a new file out using non-cached I/O and see if you get different
results.

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntfsd…

According to “Windows Internals” (Chapter 11, Cache Manager):
“Keep in mind that the cache manager is involved in file I/O only when a
file is opened without the FILE_FLAG_NO_BUFFERING flag
and then read from or written to using the Windows I/O functions (for
example, using the Windows ReadFile and WriteFile functions).”

I wrote a simple utility that writes zeros to an existing file.
It writes zeros in 4K buffers, and at the end it sets the size to be the
original size of the file.
The first version of the utility opens the file without any special flags,
the second version (xzero_no_buff) opens the file with the flag
FILE_FLAG_NO_BUFFERING.

* First version Process Monitor output:

2:51:16.3944250 PM xzero.exe 6412 CreateFile E:\Downloads\test.jpg SUCCESS
Desired Access: Generic Write, Read Attributes, Disposition: Open, Options:
Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode:
None, AllocationSize: n/a, OpenResult: Opened
2:51:16.3944703 PM xzero.exe 6412 QueryStandardInformationFile
E:\Downloads\test.jpg SUCCESS AllocationSize: 2,363,392, EndOfFile:
2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False
2:51:16.3944834 PM xzero.exe 6412 WriteFile E:\Downloads\test.jpg SUCCESS
Offset: 0, Length: 4,096, Priority: Normal
2:51:16.3945298 PM xzero.exe 6412 WriteFile E:\Downloads\test.jpg SUCCESS
Offset: 4,096, Length: 4,096
…
2:51:16.4019936 PM xzero.exe 6412 WriteFile E:\Downloads\test.jpg SUCCESS
Offset: 2,355,200, Length: 4,096
2:51:16.4020009 PM xzero.exe 6412 WriteFile E:\Downloads\test.jpg SUCCESS
Offset: 2,359,296, Length: 4,096
2:51:16.4020285 PM xzero.exe 6412 SetEndOfFileInformationFile
E:\Downloads\test.jpg SUCCESS EndOfFile: 2,360,642
2:51:16.4090436 PM xzero.exe 6412 CloseFile E:\Downloads\test.jpg SUCCESS
2:51:16.4094012 PM MsMpEng.exe 372 CreateFileMapping E:\Downloads\test.jpg
FILE LOCKED WITH WRITERS SyncType: SyncTypeCreateSection, PageProtection:
PAGE_EXECUTE_READ|PAGE_NOCACHE
2:51:16.4094244 PM MsMpEng.exe 372 QueryStandardInformationFile
E:\Downloads\test.jpg SUCCESS AllocationSize: 2,363,392, EndOfFile:
2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False
2:51:17.0794587 PM System 4 WriteFile E:\Downloads\test.jpg SUCCESS Offset:
0, Length: 1,048,576, I/O Flags: Non-cached, Paging I/O, Synchronous Paging
I/O, Priority: Very Low
2:51:17.0821091 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS
SyncType: SyncTypeOther
2:51:18.3802095 PM System 4 WriteFile E:\Downloads\test.jpg SUCCESS Offset:
1,048,576, Length: 1,048,576, I/O Flags: Non-cached, Paging I/O, Synchronous
Paging I/O, Priority: Very Low
2:51:18.3829431 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS
SyncType: SyncTypeOther
2:51:19.7413443 PM System 4 WriteFile E:\Downloads\test.jpg SUCCESS Offset:
2,097,152, Length: 266,240, I/O Flags: Non-cached, Paging I/O, Synchronous
Paging I/O, Priority: Very Low
2:51:19.7424212 PM System 4 SetEndOfFileInformationFile
E:\Downloads\test.jpg SUCCESS EndOfFile: 2,360,642
2:51:19.7424416 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS
SyncType: SyncTypeOther

* Second version Process Monitor output:

3:04:27.8110033 PM xzero_no_buff.exe 7016 CreateFile E:\Downloads\test.jpg
SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: Open,
Options: No Buffering, Synchronous IO Non-Alert, Non-Directory File,
Attributes: N, ShareMode: None, AllocationSize: n/a, OpenResult: Opened
3:04:27.8207308 PM xzero_no_buff.exe 7016 QueryStandardInformationFile
E:\Downloads\test.jpg SUCCESS AllocationSize: 2,363,392, EndOfFile:
2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False
3:04:27.8207705 PM xzero_no_buff.exe 7016 WriteFile E:\Downloads\test.jpg
SUCCESS Offset: 0, Length: 4,096, I/O Flags: Non-cached, Priority: Normal
3:04:27.8211333 PM xzero_no_buff.exe 7016 WriteFile E:\Downloads\test.jpg
SUCCESS Offset: 4,096, Length: 4,096, I/O Flags: Non-cached, Priority:
Normal
…
3:04:27.9064179 PM xzero_no_buff.exe 7016 WriteFile E:\Downloads\test.jpg
SUCCESS Offset: 2,359,296, Length: 4,096, I/O Flags: Non-cached, Priority:
Normal
3:04:27.9065758 PM xzero_no_buff.exe 7016 SetEndOfFileInformationFile
E:\Downloads\test.jpg SUCCESS EndOfFile: 2,360,642
3:04:27.9066143 PM xzero_no_buff.exe 7016 ReadFile E:\Downloads\test.jpg
SUCCESS Offset: 2,359,296, Length: 4,096, I/O Flags: Non-cached, Paging I/O,
Priority: Normal
3:04:27.9086864 PM xzero_no_buff.exe 7016 CloseFile E:\Downloads\test.jpg
SUCCESS
3:04:27.9090498 PM MsMpEng.exe 372 CreateFileMapping E:\Downloads\test.jpg
FILE LOCKED WITH WRITERS SyncType: SyncTypeCreateSection, PageProtection:
PAGE_EXECUTE_READ|PAGE_NOCACHE
3:04:27.9090725 PM MsMpEng.exe 372 QueryStandardInformationFile
E:\Downloads\test.jpg SUCCESS AllocationSize: 2,363,392, EndOfFile:
2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False
3:04:27.9091353 PM MsMpEng.exe 372 ReadFile E:\Downloads\test.jpg SUCCESS
Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous
Paging I/O, Priority: Normal
…
3:04:30.8326217 PM System 4 CreateFileMapping C:\test\Downloads\test.jpg
SUCCESS SyncType: SyncTypeOther
3:04:30.8445425 PM System 4 SetEndOfFileInformationFile
E:\Downloads\test.jpg SUCCESS EndOfFile: 2,359,296
3:04:30.8445975 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS
SyncType: SyncTypeOther
3:04:32.1532605 PM System 4 WriteFile E:\Downloads\test.jpg SUCCESS Offset:
2,359,296, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous
Paging I/O, Priority: Very Low
3:04:32.1566545 PM System 4 SetEndOfFileInformationFile
E:\Downloads\test.jpg SUCCESS EndOfFile: 2,360,642
3:04:32.1567095 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS
SyncType: SyncTypeOther

As you can see, in the first version the entire file is written by the cache
manager, while in the second version only the last 4KB frame is written.
I suspect that the last frame is being written because I set the file size.

How can I make the SetEndOfFileInformationFile a non-cached operation?

Slava_Imameev · July 7, 2017, 3:07pm

This description is not complete. It doesn’t tell that the Cache Manager is not involved if the flag is set.

A file system driver makes a decision on file caching and can ignore FILE_NO_INTERMEDIATE_BUFFERING.

That is why I advocate for opening/sharing Windows kernel sources. Natural languages are not precise.

You can’t make this for all possible file system driver implementations. A file system driver might decide to initialize a cache for non-paging End-Of-File operation. You can find this in FastFAT sources when a cache is initialized if a data section for a mapped file exists. Look at FatSetEndOfFileInfo .

Mickey_H · July 7, 2017, 6:10pm

My utility writes zeros to an existing jpg file. Between the tests I replace the zeroed file from the first run with the original file (This is why in both runs I have non-zero allocation size).
The utility writes 4KB frames and at the end it sets the file size as the original size (using SetFileInformationByHandle).

If the utility doesn’t set the size at the end (only writes 4KB frames and closes the file), I do NOT see the last frame being written by the Cache Manager.
I repeated the test several times and never saw the last frame written in this version of the utility.

Scott_Noone_OSR · July 11, 2017, 3:54pm

How do you replace the file? Do you dismount the volume between test runs?

Also, please post the call stack for the write at the end.

-scott
OSR
@OSRDrivers

wrote in message news:xxxxx@ntfsd…

My utility writes zeros to an existing jpg file. Between the tests I replace
the zeroed file from the first run with the original file (This is why in
both runs I have non-zero allocation size).
The utility writes 4KB frames and at the end it sets the file size as the
original size (using SetFileInformationByHandle).

If the utility doesn’t set the size at the end (only writes 4KB frames and
closes the file), I do NOT see the last frame being written by the Cache
Manager.
I repeated the test several times and never saw the last frame written in
this version of the utility.