File Count - ZwQueryDirectoryFile and caching

Naer_Chang · August 24, 2007, 10:42am

Hi:

I have been reading for two days about NTFS and all possible ways of enumerating files. My purpose is to be able to quickly count how many files there are.

Findings (tested on XP SP2 - 1gig RAM):

I observed that after running an enumeration with ZwQueryDirectoryFile (or FileFindFirst) once subsequent runs are more than 10 times faster. In one instance I had 72204 file which took only 1984ms (less than 2 sec). In the first run the actual time is 81094ms

Now if you use Disk Defragmenter and analyze (dfrgntfs.exe) any of your NTFS partition you can find that it’s consistently fast (it gives you total file count). What does dfrgntfs.exe do that we don’t know about?

Question:

Is there a trick to get the total number of files per partition very quickly like dfrgntfs.exe?
If there is some kind of caching with Windows (I am interested in 2000, XP, Vista) about file system information. How does it work? How is it related to system memory? How long can it last?

Thanks.
Naer

OSR_Community_User · August 24, 2007, 11:15am

I would imagine that it is getting the information by processing the
file system metadata. As far as defrag, there is a documented user mode
api (FSCTL) for defragmentation; this is what I would imagine that it
uses, although I don’t think that the information you seek can be
obtained with it.

Search MSDN/SDK help for any of these, all of which are used with
DeviceIoControl:

FSCTL_GET_VOLUME_BITMAP
FSCTL_GET_RETRIEVAL_POINTERS
FSCTL_MOVE_FILE
FSCTL_READ_MFT_RECORD

I believe that this is what’s going on, but I can’t say that I have ever
played around with these, and I don’t think that these particular
(FSCTL_) will do exactly what you wish, but it is a place to start.

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@yahoo.ca
Sent: Friday, August 24, 2007 10:44
To: Windows System Software Devs Interest List
Subject: [ntdev] File Count - ZwQueryDirectoryFile and caching

Hi:

I have been reading for two days about NTFS and all possible ways of
enumerating files. My purpose is to be able to quickly count how many
files there are.

Findings (tested on XP SP2 - 1gig RAM):

I observed that after running an enumeration with ZwQueryDirectoryFile
(or FileFindFirst) once subsequent runs are more than 10 times faster.
In one instance I had 72204 file which took only 1984ms (less than 2
sec). In the first run the actual time is 81094ms

Now if you use Disk Defragmenter and analyze (dfrgntfs.exe) any of your
NTFS partition you can find that it’s consistently fast (it gives you
total file count). What does dfrgntfs.exe do that we don’t know about?

Question:

Is there a trick to get the total number of files per partition very
quickly like dfrgntfs.exe?
If there is some kind of caching with Windows (I am interested in
2000, XP, Vista) about file system information. How does it work? How is
it related to system memory? How long can it last?

Thanks.
Naer

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

OSR_Community_User · August 24, 2007, 11:25am

I forgot something. FSCTL_READ_MFT_RECORD will do exactly what you
wish, I think, but it is, to the best of my knowledge, completely
undocumented.

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Martin O’Brien
Sent: Friday, August 24, 2007 11:15
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] File Count - ZwQueryDirectoryFile and caching

I would imagine that it is getting the information by processing the
file system metadata. As far as defrag, there is a documented user mode
api (FSCTL) for defragmentation; this is what I would imagine that it
uses, although I don’t think that the information you seek can be
obtained with it.

Search MSDN/SDK help for any of these, all of which are used with
DeviceIoControl:

FSCTL_GET_VOLUME_BITMAP
FSCTL_GET_RETRIEVAL_POINTERS
FSCTL_MOVE_FILE
FSCTL_READ_MFT_RECORD

I believe that this is what’s going on, but I can’t say that I have ever
played around with these, and I don’t think that these particular
(FSCTL_) will do exactly what you wish, but it is a place to start.

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@yahoo.ca
Sent: Friday, August 24, 2007 10:44
To: Windows System Software Devs Interest List
Subject: [ntdev] File Count - ZwQueryDirectoryFile and caching

Hi:

I have been reading for two days about NTFS and all possible ways of
enumerating files. My purpose is to be able to quickly count how many
files there are.

Findings (tested on XP SP2 - 1gig RAM):

I observed that after running an enumeration with ZwQueryDirectoryFile
(or FileFindFirst) once subsequent runs are more than 10 times faster.
In one instance I had 72204 file which took only 1984ms (less than 2
sec). In the first run the actual time is 81094ms

Now if you use Disk Defragmenter and analyze (dfrgntfs.exe) any of your
NTFS partition you can find that it’s consistently fast (it gives you
total file count). What does dfrgntfs.exe do that we don’t know about?

Question:

Is there a trick to get the total number of files per partition very
quickly like dfrgntfs.exe?
If there is some kind of caching with Windows (I am interested in
2000, XP, Vista) about file system information. How does it work? How is
it related to system memory? How long can it last?

Thanks.
Naer

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Naer_Chang · August 24, 2007, 5:33pm

I don’t see my reply…

I get STATUS_INVALID_DEVICE_REQUEST from NtFsControlFile. I open the volume “\\.\c:” with CreateFile which was successful. Do I need to use ZwCreateFille?

HANDLE hFile = CreateFile( _T(“\\.\c:”),
GENERIC_READ, FILE_SHARE_READ|FILE_SHARE_WRITE,
NULL, OPEN_EXISTING,
0, 0 );

Naer_Chang · August 24, 2007, 5:38pm

DWORD dwStatus = 0;
IO_STATUS_BLOCK ioStatus;
ULONGLONG startVcn = 0;
PGET_RETRIEVAL_DESCRIPTOR fileMappings;
BYTE *RecordMap = new BYTE [1039];

pfn_NtFsControlFile( hFile, 0, NULL, NULL, &ioStatus,
FSCTL_READ_MFT_RECORD,
&startVcn, sizeof( startVcn ),
RecordMap,
sizeof(RecordMap) )

Question:

Is there a specific struct that I need to use to receive the data? I used BYTE because I saw a Delphi example doing that.

http://delphi.newswhat.com/geoxml/forumhistorythread?groupname=borland.public.delphi.thirdpartytools.general&messageid=454aa40f$xxxxx@newsgroups.borland.com

OSR_Community_User · August 24, 2007, 6:20pm

Yes, although perhaps not a closed C structure per se; in either case, I
have no idea of what it looks like.

mm

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@yahoo.ca
Sent: Friday, August 24, 2007 17:40
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] File Count - ZwQueryDirectoryFile and caching

DWORD dwStatus = 0;
IO_STATUS_BLOCK ioStatus;
ULONGLONG startVcn = 0;
PGET_RETRIEVAL_DESCRIPTOR fileMappings;
BYTE *RecordMap = new BYTE [1039];

pfn_NtFsControlFile( hFile, 0, NULL, NULL, &ioStatus,
FSCTL_READ_MFT_RECORD,
&startVcn, sizeof( startVcn ),
RecordMap,
sizeof(RecordMap) )

Question:

Is there a specific struct that I need to use to receive the data? I
used BYTE because I saw a Delphi example doing that.

http://delphi.newswhat.com/geoxml/forumhistorythread?groupname=borland.p
ublic.delphi.thirdpartytools.general&messageid=454aa40f$xxxxx@newsgroups.bor
land.com

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

David_J_Craig · August 24, 2007, 6:23pm

What is the size_t value for sizeof(RecordMap)? You only want 4 bytes of
data? Eight if x64?

–
David J. Craig
Engineer, Sr. Staff Software Systems
Broadcom Corporation

wrote in message news:xxxxx@ntdev…
> DWORD dwStatus = 0;
> IO_STATUS_BLOCK ioStatus;
> ULONGLONG startVcn = 0;
> PGET_RETRIEVAL_DESCRIPTOR fileMappings;
> BYTE *RecordMap = new BYTE [1039];
>
> pfn_NtFsControlFile( hFile, 0, NULL, NULL, &ioStatus,
> FSCTL_READ_MFT_RECORD,
> &startVcn, sizeof( startVcn ),
> RecordMap,
> sizeof(RecordMap) )
>
> Question:
>
> 1) Is there a specific struct that I need to use to receive the data? I
> used BYTE because I saw a Delphi example doing that.
>
> http://delphi.newswhat.com/geoxml/forumhistorythread?groupname=borland.public.delphi.thirdpartytools.general&messageid=454aa40f$xxxxx@newsgroups.borland.com
>

Naer_Chang · August 27, 2007, 9:42pm

Size is set to BYTE[1039] (don’t ask why I copied from somewhere). Is there a struct I need to use (they usually require them so the data coming out is useful)?

There is another flag, FSCTL_GET_NTFS_FILE_RECORD, that supposedly also can query MFT. I tried but it doesn’t work either. Is there a way to use this flag to enum all the files on a partition (how to find the first file reference)?

Could someone provide a working example with these flag? I am attempting to read the records through these flags. I know you can do it by reading the disk directly (and reference the linux site for a complete NTFS reference) but I don’t want to go that route for obvious reasons - there is an example too at codeproject.com (undelete project).