Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Before Posting...
Please check out the Community Guidelines in the Announcements and Administration Category.

Allocation of reparsed file name buffers - memory fragmentation considerations

Dejan_MaksimovicDejan_Maksimovic Member - All Emails Posts: 244
edited January 19 in NTFSD

Hi,

I am testing a filter, which would redirect literally all file
opens (this is just a test, nothing specific in mind). And I am worried about the possibility of memory fragmentation
after some number of file name string buffers get allocated and
deallocated, which will happen, and is only a matter of time.

Now, aside form allocating, say 10K, buffers when the driver loads,
and using those, is there any mechanism which would allow me to
replace the file names, and still not get memory fragmentations, at
some point?
10K string will get eaten within minutes, if not seconds, and even
if I allocate only 256 BYTES (which I would often require), that is
not a small amount of memory.

The second idea was to allocate 4K aligned buffers only, hoping
that would simply lead to less fragmentation, but a lookaside list
would be the best approach. I just don't see any present in the API.

Does IoReplaceFileObjectName use lookaside lists internally, by any chance?

Kind regards, Dejan.

Post edited by Peter_Viscarola_(OSR) on

Comments

  • Dejan_MaksimovicDejan_Maksimovic Member - All Emails Posts: 244

    I have no idea why the styling appeared ;)

  • Gabriel_BerceaGabriel_Bercea Member - All Emails Posts: 482

    Don't worry too much about that as the memory manager behind the scenes is not so oblivious to this and does its best to keep things in check.
    You could use lookaside lists for one, but in general there are allocations happening all the time in the kernel.
    I would worry more about performance rather than fragmentation in your case.

    Cheers,
    Gabriel

  • Dejan_MaksimovicDejan_Maksimovic Member - All Emails Posts: 244
    via Email
    Not so. We have seen memory fragmentation in the past, and since we
    need to support legacy OSes (XP and 7), we need to consider the memory
    fragmentation. It happens, and it is far from trivial.

    I already know I need to worry about it, I just do not know the best
    mitigation ;)

    On 1/19/19, Gabriel_Bercea wrote:
    > OSR https://community.osr.com/
    > Gabriel_Bercea commented on Allocation of reparsed file name buffers -
    > memory fragmentation considerations
    >
    > Don't worry too much about that as the memory manager behind the scenes is
    > not so oblivious to this and does its best to keep things in check.
    >
    > You could use lookaside lists for one, but in general there are allocations
    > happening all the time in the kernel.
    >
    > I would worry more about performance rather than fragmentation in your
    > case.
    >
    > --
    > Reply to this email directly or follow the link below to check it out:
    > https://community.osr.com/discussion/comment/292218#Comment_292218
    >
    > Check it out:
    > https://community.osr.com/discussion/comment/292218#Comment_292218
    >
  • Peter_Viscarola_(OSR)Peter_Viscarola_(OSR) Administrator Posts: 7,507

    I have no idea why the styling appeared

    I have a one word answer: Markdown.

    Like Mr. Bercea, I am having trouble believing allocating and reallocating file name strings is going to cause you fragmentation problems. But if you insist this is an issue... how about writing your own allocator? At startup, allocate a big chunk of pool... and the handle the string allocation yourself from within the chunk?? I don’t see this being a great idea, but...

    Peter

    Peter Viscarola
    OSR
    @OSRDrivers

  • Dejan_MaksimovicDejan_Maksimovic Member - All Emails Posts: 244
    via Email
    That's the only thing I have at the moment, but however big that
    allocation is, it will run out at some point. Reallocating a new
    batch, when the old is near completion is the best I can do
    afterwards.

    This is under the presumption that there are a LOT of redirections (sa
    I mentioned in the tune of 10K a minute possibly).

    I know there were fragmentation issues when we allocated Flt contexts
    (esseentially file name buffers with a few flags) without lookaside
    lists. It was a metter of time only before any machine would run out
    of memory, due to fragmentation, even though there is technically
    enough free memory.

    On 1/19/19, Peter_Viscarola_(OSR)
    wrote:
    > OSR https://community.osr.com/
    > Peter_Viscarola_(OSR) commented on Allocation of reparsed file name buffers
    > - memory fragmentation considerations
    >
    >> I have no idea why the styling appeared
    >
    > I have a one word answer: Markdown.
    >
    > Like Mr. Bercea, I am having trouble believing allocating and reallocating
    > file name strings is going to cause you fragmentation problems. But if you
    > insist this is an issue... how about writing your own allocator? At
    > startup, allocate a big chunk of pool... and the handle the string
    > allocation yourself from within the chunk?? I don’t see this being a great
    > idea, but...
    >
    > Peter
    >
    > --
    > Reply to this email directly or follow the link below to check it out:
    > https://community.osr.com/discussion/comment/292226#Comment_292226
    >
    > Check it out:
    > https://community.osr.com/discussion/comment/292226#Comment_292226
    >
  • Gabriel_BerceaGabriel_Bercea Member - All Emails Posts: 482

    But if you insist this is an issue... how about writing your own allocator?

    And that's fun as well. I've written one just for "fun" for a kernel library I have which has all sorts of utility functions and functionalities. It's not trivial, but also not the most difficult task you'll ever have.

    Again though, you have to realize that according to what you are saying:

    I am testing a filter, which would redirect literally all file opens

    That's not a lot of opens. Especially if you actually free the memory after the open :smile:
    All of those file opens have name allocates associated with them already. You are implying that allocating another Unicode string for each one during the Create path, fragment your memory. Hard to believe, unless you forget to free the memory. Can you test your driver with verifier and unload it to check for leaks ?
    There is something more fragmenting than file system activity and that is registry actvity. There are by far more reg api/sec going on. Just open procmon and allow for all the reg activity to flood in and all FS activity and compare. I did one of these experiments for you.
    Procmon running for 30 seconds on idle system: 364 filesystem events, 6909 reg events.
    Procmon running for 30 seconds on system doing random activity: 20k filesystem Creates/Opens, 84k RegOpens/Creates
    You can try that yourself.

    And for each one (reg open/create) imagine you need to create a context of your own and then maybe you want to store the name as well. And perhaps you want to create a reg virtualization filter so you would need to allocate the "target key name" as well. Those filters would be impossible to write, but somehow they exist. The only issue that can come up from such an active callback is performance.

    I would advise using verifier and make sure you're not leaking memory.

    Gabriel.

    Cheers,
    Gabriel

  • Dejan_MaksimovicDejan_Maksimovic Member - All Emails Posts: 244
    via Email
    You are hunting in the wrong area :)

    When I allocate contexts, they DO fragment the memory, without any
    leaks. Leaks would NOT fragment the memory, as they remain in a used
    space.
    We had this confirmed by Microsoft kernel guys, it is not a random
    thought, leak or something of sorts.
    We mitigated that one easily by allocating contexts from a lookaside
    list. Simple, and easily doable.

    A file name buffer, when used internally by us only, is also allocated
    from a lookaside list, thus the fragmentation is not an issue.

    The REPARSE buffers are another thing. We can only allocate them - we
    do not free them, and we cannot make them appear in any of our
    lookasides.


    Now, you are considering 30 minutes of activity only - how about 30
    weeks? This needs to run on a server, and also on machines that do not
    restart at all for a month at least.


    So, any thoughts on how to minimize the fragmentation?

    On 1/21/19, Gabriel_Bercea wrote:
    > OSR https://community.osr.com/
    > Gabriel_Bercea commented on Allocation of reparsed file name buffers -
    > memory fragmentation considerations
    >
    >> But if you insist this is an issue... how about writing your own
    >> allocator?
    >
    > And that's fun as well. I've written one just for "fun" for a kernel library
    > I have which has all sorts of utility functions and functionalities. It's
    > not trivial, but also not the most difficult task you'll ever have.
    >
    > Again though, you have to realize that according to what you are saying:
    >
    >> I am testing a filter, which would redirect literally all file opens
    >
    > That's not a lot of opens. Especially if you actually free the memory after
    > the open
    >
    > All of those file opens have name allocates associated with them already.
    > You are implying that allocating another Unicode string for each one during
    > the Create path, fragment your memory. Hard to believe, unless you forget to
    > free the memory. Can you test your driver with verifier and unload it to
    > check for leaks ?
    >
    > There is something more fragmenting than file system activity and that is
    > registry actvity. There are by far more reg api/sec going on. Just open
    > procmon and allow for all the reg activity to flood in and all FS activity
    > and compare. I did one of these experiments for you.
    >
    > Procmon running for 30 seconds on idle system: 364 filesystem events, 6909
    > reg events.
    >
    > Procmon running for 30 seconds on system doing random activity: 20k
    > filesystem Creates/Opens, 84k RegOpens/Creates
    >
    > You can try that yourself.
    >
    > And for each one (reg open/create) imagine you need to create a context of
    > your own and then maybe you want to store the name as well. And perhaps you
    > want to create a reg virtualization filter so you would need to allocate the
    > "target key name" as well. Those filters would be impossible to write, but
    > somehow they exist. The only issue that can come up from such an active
    > callback is performance.
    >
    > I would advise using verifier and make sure you're not leaking memory.
    >
    > Gabriel.
    >
    > --
    > Reply to this email directly or follow the link below to check it out:
    > https://community.osr.com/discussion/comment/292242#Comment_292242
    >
    > Check it out:
    > https://community.osr.com/discussion/comment/292242#Comment_292242
    >
  • Gabriel_BerceaGabriel_Bercea Member - All Emails Posts: 482

    The REPARSE buffers are another thing. We can only allocate them - we do not free them, and we cannot make them appear in any of our lookasides.

    Are you sure about that ?
    First of all if you don't free these buffers that you say are very high on the activity for months on end I assure you the fragmentation is not your worry but rather the memory capacity.
    Secondly, of course you can make everything fit in a lookaside.
    As @Peter_Viscarola_(OSR) suggested one way would be to write your own allocator.

    The allocator could behind the scene, depending on the size you want to allocate would allocate from an internal pool of lookaside lists and free accordingly.

    Very cruel skeleton idea comes to mind:

    PVOID MyCustomAlloc(ULONG Size)
    {
    PVOID mem = ExAllocateFromLookaside(Get_Right_Lookaside(Size + sig_size));
    //write a signature so you know where to free from
    WriteSignature(mem);
    return mem+sig_size;
    }

    VOID MyCustomFree(mem)
    {
    PVOID mem_header = mem-sig_size;
    ExFreeToLookaside(mem_header, Get_RIghtLookaside(*((PULONG)mem_header));
    }

    You now initialize a number of looksides in driverEntry varying from the low end size of a memory allocation to a high end size of the memory allocations this pool of lookasides could serve as fragmentation protection.
    But then again, fragmentation protection is useless in your case as you say you don't free the buffers, so what's the point of using lookasides anyway.

    Can you explain why reparse buffes cannot be freed ? Also, let's assume the fragmentation never occurs, if you never free them how do you cope with limited memory ?

    Cheers,
    Gabriel

  • Scott_Noone_(OSR)Scott_Noone_(OSR) Administrator Posts: 3,198

    If you step through IoReplaceFileObjectName you'll see it does not use a lookaside list. However, it is smart enough to use the existing buffer if the new name is <= the max buffer length of the old name. It also appears to overallocate the buffer if it's below certain thresholds (though I'm not exactly sure why the numbers there were chosen).

    It would be very difficult for the OS to change the way name buffers are allocated at this point and maintain backwards compatibility.

    -scott
    OSR

  • Dejan_MaksimovicDejan_Maksimovic Member - All Emails Posts: 244
    via Email
    Do you know what a reparse file name buffer is?

    There are no leaks... please don't dillute the topic into a wrong direction.
  • Dejan_MaksimovicDejan_Maksimovic Member - All Emails Posts: 244
    via Email
    Just so we are clear - those buffers cannot be freed by us! They are
    freed by the I/O manager, so we cannot put them back into a lookaside.

    Unless IoReplaceFileObjectName uses lookasides itself, fragmentation
    will occur :(

    On 1/23/19, Dejan Maksimovic wrote:
    > Do you know what a reparse file name buffer is?
    >
    > There are no leaks... please don't dillute the topic into a wrong
    > direction.
    >
    > --
    >
    >
    > Kind regards, Dejan.
    >
Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Upcoming OSR Seminars
Writing WDF Drivers 21 Oct 2019 OSR Seminar Space & ONLINE
Internals & Software Drivers 18 Nov 2019 Dulles, VA
Kernel Debugging 30 Mar 2020 OSR Seminar Space
Developing Minifilters 27 Apr 2020 OSR Seminar Space & ONLINE