Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Home NTFSD
Before Posting...
Please check out the Community Guidelines in the Announcements and Administration Category.

More Info on Driver Writing and Debugging


The free OSR Learning Library has more than 50 articles on a wide variety of topics about writing and debugging device drivers and Minifilters. From introductory level to advanced. All the articles have been recently reviewed and updated, and are written using the clear and definitive style you've come to expect from OSR over the years.


Check out The OSR Learning Library at: https://www.osr.com/osr-learning-library/


Reading the NTFS MFT from user mode as a file (so it gets cached)

DonpedroDonpedro Member Posts: 3

Hello,

I'd like to optimize very specific scenarios where almost every directory is enumerated on an NTFS volume (on an HDD). When an application spends a considerable time reading a file with inefficient access patterns, a cheap trick can be utilized to speed it up: reading the whole file sequentially in big chunks beforehand into memory, so it gets cached by the system. Then the application's reads will be fulfilled from memory by the cache manager, so it's going to be really fast.

I thought I would employ the same "trick" for this problem, as well. However, it doesn't seem like that the MFT can be accessed easily. Every code I could find on the internet opens a handle to the volume itself and performs raw reads on it. This works great in the sense that you can actually read the MFT or anything else on the drive, but it seems like that no caching is performed by the cache manager for these reads. I speculate this is because it's not opened as a regular file.

Is there a way to either open the MFT as a regular file or somehow perform reads on it in big chunks in a way that will populate the file cache with its content?

Comments

  • vagrantvagrant Member Posts: 8

    I'm also interested in this, I wanted to read the MFT attributes, it doesn't seem to be quick way to do it, as far as I know you have to use FSCTL_GET_NTFS_VOLUME_DATA and then parse the sectors individually

  • Scott_Noone_(OSR)Scott_Noone_(OSR) Administrator Posts: 3,260

    NTFS effectively treats the MFT as a regular file and uses the Cache Manager to cache its contents. The file is available to user mode as "C:\$Mft" but NTFS restricts you to only getting FILE_READ_ATTRIBUTES access.

    When you read directly from the volume you bypass all of this caching because Windows caches on a file level. So, no, there is no supported way to do what you're asking.

    -scott
    OSR

  • DonpedroDonpedro Member Posts: 3

    Thanks Scott, I'm sorry to hear that.

    there is no supported way to do what you're asking.

    Is there an unsupported way, though? This is not something that we would like to ship in an end-user product or something, so if there's a way that's prone to breaking, that would still be acceptable.

    If the answer is still no, would this be possible from a software driver? If so, could you give me some pointers?

  • Scott_Noone_(OSR)Scott_Noone_(OSR) Administrator Posts: 3,260

    The quasi-documented FSCTL_FILE_PREFETCH lets you prefetch the metadata for a file or the contents of a directory. This really requires that you know in advance what it is you want to prefetch though (easy for the prefetcher because it's just speeding up the next time).

    If you wanted a very large hammer presumably you could try loading a driver and call MmPrefetchPages yourself on the MFT. I haven't ever tried this myself but it seems reasonably safe in that you're not accessing the MFT in a way that might conflict with NTFS' usage. The downside is that it only puts the pages into the standby state (instead of active) so it's possible that for a sufficiently large MFT the start of the MFT could be paged out by the time you reached the end.

    Note that just having the MFT resident isn't necessarily going to speed up your directory scans as any sufficiently large directory will not be resident in the MFT.

    -scott
    OSR

  • DonpedroDonpedro Member Posts: 3

    As for FSCTL_FILE_PREFETCH, I think I'm better off with using NtQueryDirectory directly with a big enough buffer size (FindFirstFile is stubborn and uses a buffer size of 0x1000 even if I pass FIND_FIRST_EX_LARGE_FETCH, and only uses a bigger, fixed buffer of 0x10000 for subsequent FindNextFile's). This speeds up the enumeration of larger directories, but the gains are not that significant.

    As for MmPrefetchPages, I see that the READ_LIST structure (second parameter) should contain pointer(s) to FILE_OBJECT(s). Could you give me some guidance on how to acquire a FILE_OBJECT (ZwOpenFile etc. give me a regular HANDLE). Even if there is a way to get one, won't the system prevent my driver to get one for the MFT specifically (as it does in user mode)?

  • Scott_Noone_(OSR)Scott_Noone_(OSR) Administrator Posts: 3,260

    @Donpedro said:
    As for FSCTL_FILE_PREFETCH, I think I'm better off with using NtQueryDirectory directly with a big enough buffer size (FindFirstFile is stubborn and uses a buffer size of 0x1000 even if I pass FIND_FIRST_EX_LARGE_FETCH, and only uses a bigger, fixed buffer of 0x10000 for subsequent FindNextFile's). This speeds up the enumeration of larger directories, but the gains are not that significant.

    The point of the prefetch is that it brings the file/directory into memory using an efficient I/O pattern. Then when someone actually enumerates the directory it's resident in memory and you avoid the page faults. You need to eat the disk I/O either way though (either up front or on demand).

    As for MmPrefetchPages, I see that the READ_LIST structure (second parameter) should contain pointer(s) to FILE_OBJECT(s). Could you give me some guidance on how to acquire a FILE_OBJECT (ZwOpenFile etc. give me a regular HANDLE). Even if there is a way to get one, won't the system prevent my driver to get one for the MFT specifically (as it does in user mode)?

    ZwOpenFile + ObReferenceObjectByHandle. MmPrefetchPages does not require any specific access to the FILE_OBJECT so the fact that you can't get FILE_READ_DATA doesn't matter.

    -scott
    OSR

Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Upcoming OSR Seminars
OSR has suspended in-person seminars due to the Covid-19 outbreak. But, don't miss your training! Attend via the internet instead!
Kernel Debugging 30 Mar 2020 OSR Seminar Space
Developing Minifilters 15 Jun 2020 LIVE ONLINE
Writing WDF Drivers 22 June 2020 LIVE ONLINE
Internals & Software Drivers 28 Sept 2020 Dulles, VA