FloppyMedia is a bit that means the entire image needs to be copied to pagefile-backed memory, instead of being backed by the image file on disk. Its original purpose was to allow programs on CDs or floppies to run after their media has been ejected. You can also link the program with /swaprun:net option and the system will copy the image to the pagefile when it is run from network.
In your case the image is getting copied to the pagefile because the memory manager thinks there might be user writeable references to the image file (e.g. writable handles, sections or mapped views), and it wants to avoid the possibility of an executable being modified after it's been loaded. If the file is actually opened read-only, Vista and win7 can detect that and avoid the copy, but I believe XP can't do that because it lacks some of the required plumbing between filesystems and the memory manager.
Executables backed by the pagefile should still share physical pages between processes (unless a process tries to modify a page, in which case it will get a private copy).
I agree with Alex and others who suggested using non-cached IO here. It seems like the simplest and safest approach.
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@vmware.com
Sent: Tuesday, October 04, 2011 4:15 AM
To: Windows File Systems Devs Interest List
Subject: RE:[ntfsd] FltReadFile from file system filter driver and memory usage...
Hello Pavel,
You were absolutely correct with your analysis! With my filter driver loaded, FloppyMedia bit is indeed set. I am attaching relevant windbg output later in this post for one of the core file ntdll.dll.
Given this, I have few follow-up questions.
- What is this FloppyMedia bit? Is there any documentation on this? I searched google and cannot find anything relevant.
- When FloppyMedia bit is set, does this mean that all the processes are getting separate physical pages for all the shared DLLs viz. kernel32.dll, ntdll.dll etc and hence causing per process memory shoot-up? The various counters and tools seem to be misleading in this regard e.g. vmmap tool reports that the shared DLL pages as shared whereas actually they are not. Instead, vmmap reports the additional memory attributed towards page tables.
- The file object (ntdll in this case) on which I am making FltReadFile call is opened in readonly mode. Hence, there is no question of file being modified. Given this, why does Windows still give a copy of the pages for all processes and not share it?
- I observed this behavior on Windows XP and Windows 2003. However, on Vista, I don't see FloppyMedia bit being set and hence I don't see the memory shoot up either. Does this mean, that, this is an issue which is fixed in Vista and above?
- What options do I have to solve this issue? I can think of following options.
a. Always do a non-cached read. I believe this will have performance impact and would like to avoid it.
b. Do a cached read only if the image is mapped. I can probably do this by checking SectionObjectPointers->ImageSectionObject != NULL? However, it won't buy me much because I need to read the file only on first PostOpCreate for the file and not subsequently. Hence, effectively, I will end up doing non-cached read for all image files.
c. Is there a way to detect in advance if the file being opened is a image file apart from looking at file extension and if so map it as image before initiating a read on it?
d. Can I use FltReadFile with FLTFL_IO_OPERATION_PAGING|FLTFL_IO_OPERATION_SYNCHRONOUS_PAGING? This also seems to solve the memory shoot up issue.
Below is the windbg output for ntdll.dll.
Without my filter driver loaded, the relevant windbg output shows the following.
kd> !memusage
...
...
Control Valid Standby Dirty Shared Locked PageTables name ...
...
85e13280 0 4 0 0 0 0 mapped_file( ntdll.dll )
8619f2f8 340 420 0 316 0 0 mapped_file( ntdll.dll )
...
...
kd> !ca 85e13280
ControlArea @ 85e13280
Segment e1539598 Flink 85c80eac Blink 85e13974
Section Ref 0 Pfn Ref 1 Mapped Views 0
User Ref 0 WaitForDel 0 Flush Count 0
File Object 85e13310 ModWriteCount 0 System Views 0
WritableRefs 0
Flags (80) File
File: \WINDOWS\system32\ntdll.dll
Segment @ e1539598
Type nt!_MAPPED_FILE_SEGMENT not found.
kd> !ca 8619f2f8
ControlArea @ 8619f2f8
Segment e1008188 Flink 00000000 Blink 00000000
Section Ref 1 Pfn Ref be Mapped Views 1b
User Ref 1c WaitForDel 0 Flush Count 0
File Object 86010b00 ModWriteCount 0 System Views 0
WritableRefs 0
Flags (81000a0) Image File DebugSymbolsLoaded Accessed
File: \WINDOWS\system32\ntdll.dll
Segment @ e1008188
ControlArea 8619f2f8 BasedAddress 7c800000
Total Ptes c0
WriteUserRef 0 SizeOfSegment c0000
Committed 0 PTE Template 8619f3b000000420
Based Addr 7c800000 Image Base 0
Image Commit 6 Image Info e10087c8
ProtoPtes e10081c8
Subsection 1 @ 8619f330
ControlArea 8619f2f8 Starting Sector 0 Number Of Sectors 2
Base Pte e10081c8 Ptes In Subsect 1 Unused Ptes 0
Flags 11 Sector Offset 0 Protection 1
Subsection 2 @ 8619f350
ControlArea 8619f2f8 Starting Sector 2 Number Of Sectors 42e
Base Pte e10081d0 Ptes In Subsect 86 Unused Ptes 0
Flags 31 Sector Offset 0 Protection 3
Subsection 3 @ 8619f370
ControlArea 8619f2f8 Starting Sector 430 Number Of Sectors 1b
Base Pte e1008600 Ptes In Subsect 6 Unused Ptes 0
Flags 51 Sector Offset 0 Protection 5
Subsection 4 @ 8619f390
ControlArea 8619f2f8 Starting Sector 44b Number Of Sectors 172
Base Pte e1008630 Ptes In Subsect 2f Unused Ptes 0
Flags 11 Sector Offset 0 Protection 1
Subsection 5 @ 8619f3b0
ControlArea 8619f2f8 Starting Sector 5bd Number Of Sectors 1a
Base Pte e10087a8 Ptes In Subsect 4 Unused Ptes 0
Flags 11 Sector Offset 0 Protection 1
With my filter driver loaded, the relevant windbg output shows the following.
kd> !memusage
...
...
Control Valid Standby Dirty Shared Locked PageTables name ...
...
861a06c8 760 0 0 748 0 0 mapped_file( ntdll.dll )
85c5aca0 0 252 0 0 0 0 mapped_file( ntdll.dll )
...
...
kd> !ca 861a06c8
ControlArea @ 861a06c8
Segment e1437990 Flink 00000000 Blink 00000000
Section Ref 1 Pfn Ref be Mapped Views 1a
User Ref 1b WaitForDel 0 Flush Count 0
File Object 8605a858 ModWriteCount 0 System Views 0
WritableRefs 0
Flags (81040a0) Image File FloppyMedia DebugSymbolsLoaded Accessed
File: \WINDOWS\system32\ntdll.dll
Segment @ e1437990
ControlArea 861a06c8 BasedAddress 7c800000
Total Ptes c0
WriteUserRef 0 SizeOfSegment c0000
Committed 0 PTE Template 861a078000000420
Based Addr 7c800000 Image Base 0
Image Commit 6 Image Info e1437fd0
ProtoPtes e14379d0
Subsection 1 @ 861a0700
ControlArea 861a06c8 Starting Sector 0 Number Of Sectors 2
Base Pte e14379d0 Ptes In Subsect 1 Unused Ptes 0
Flags 11 Sector Offset 0 Protection 1
Subsection 2 @ 861a0720
ControlArea 861a06c8 Starting Sector 2 Number Of Sectors 42e
Base Pte e14379d8 Ptes In Subsect 86 Unused Ptes 0
Flags 31 Sector Offset 0 Protection 3
Subsection 3 @ 861a0740
ControlArea 861a06c8 Starting Sector 430 Number Of Sectors 1b
Base Pte e1437e08 Ptes In Subsect 6 Unused Ptes 0
Flags 51 Sector Offset 0 Protection 5
Subsection 4 @ 861a0760
ControlArea 861a06c8 Starting Sector 44b Number Of Sectors 172
Base Pte e1437e38 Ptes In Subsect 2f Unused Ptes 0
Flags 11 Sector Offset 0 Protection 1
Subsection 5 @ 861a0780
ControlArea 861a06c8 Starting Sector 5bd Number Of Sectors 1a
Base Pte e1437fb0 Ptes In Subsect 4 Unused Ptes 0
Flags 11 Sector Offset 0 Protection 1
kd> !ca 85c5aca0
ControlArea @ 85c5aca0
Segment e14053c8 Flink 85f23c7c Blink 85d9077c
Section Ref 0 Pfn Ref 3f Mapped Views 0
User Ref 0 WaitForDel 0 Flush Count 0
File Object 8605a858 ModWriteCount 0 System Views 0
WritableRefs 0
Flags (8008080) File WasPurged Accessed
File: \WINDOWS\system32\ntdll.dll
Segment @ e14053c8
Type nt!_MAPPED_FILE_SEGMENT not found.
Thanks.
-Prasad
NTFSD is sponsored by OSR
For our schedule of debugging and file system seminars visit:
To unsubscribe, visit the List Server section of OSR Online at ListServer/Forum