Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Home NTFSD
Before Posting...
Please check out the Community Guidelines in the Announcements and Administration Category.

More Info on Driver Writing and Debugging


The free OSR Learning Library has more than 50 articles on a wide variety of topics about writing and debugging device drivers and Minifilters. From introductory level to advanced. All the articles have been recently reviewed and updated, and are written using the clear and definitive style you've come to expect from OSR over the years.


Check out The OSR Learning Library at: https://www.osr.com/osr-learning-library/


memory management question

OSR_Community_UserOSR_Community_User Member Posts: 110,217
Hello,
Say you have a driver that needs to manage many small structures of
fixed size such that the driver may have several megabytes worth of these
structures allocated at any given time from paged pool. Since the
structures are allocated from paged pool, I can imagine that it would be
beneficial to allocate the structures from a contiguous heap of memory to
preserve 'locality of reference': one page fault will bring in tens or
hundreds of these structures reducing overall paging of the system as these
structures are accessed.
Is it worth the effort of doing some of your own memory management
or will the VMM do an adequate job of preserving this locality of reference
(or does it even try)? Looking at the DDK documentation, it seems that
'zones' were designed to address this issue, however, they have been
declared obsolete. The documentation suggests 'lookaside lists', although
lookaside lists seem to address a different need than did zones; lookaside
lists seem not to be designed to reduce fragmentation and preserve locality
of reference as I've described.
So, Will lookaside lists help here? Should I use zones to reduce
fragmentation? Or should I just assume the VMM will do some work to help
out with fragmentation? Any suggestions?
BTW are there any guidelines as to how much address space a well
mannered driver should consume? Given that I'm sharing my 2GB with everyon
else now (and just a fraction of that for the pools), I'm wondering how much
is too much. Is several megabytes of of paged pool too much? I figure if
its there, I may as well use it - but that's coming from the perspective of
a user mode developer!

Thanks,
Joel

Comments

  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    You might want to look into using the zone memory routines.

    ExInitializeZone();
    ExExtendZone();
    ExAllocateFromZone();
    ExFreeToZone();
    ExIsFullZone();


    -----Original Message-----
    From: Smith, Joel [mailto:[email protected]]
    Sent: Thursday, April 27, 2000 10:08 AM
    To: File Systems Developers
    Subject: [ntfsd] memory management question



    Hello,
    Say you have a driver that needs to manage many small structures of
    fixed size such that the driver may have several megabytes worth of these
    structures allocated at any given time from paged pool. Since the
    structures are allocated from paged pool, I can imagine that it would be
    beneficial to allocate the structures from a contiguous heap of memory to
    preserve 'locality of reference': one page fault will bring in tens or
    hundreds of these structures reducing overall paging of the system as these
    structures are accessed.

    Is it worth the effort of doing some of your own memory management
    or will the VMM do an adequate job of preserving this locality of reference
    (or does it even try)? Looking at the DDK documentation, it seems that
    'zones' were designed to address this issue, however, they have been
    declared obsolete. The documentation suggests 'lookaside lists', although
    lookaside lists seem to address a different need than did zones; lookaside
    lists seem not to be designed to reduce fragmentation and preserve locality
    of reference as I've described.

    So, Will lookaside lists help here? Should I use zones to reduce
    fragmentation? Or should I just assume the VMM will do some work to help
    out with fragmentation? Any suggestions?

    BTW are there any guidelines as to how much address space a well
    mannered driver should consume? Given that I'm sharing my 2GB with everyon
    else now (and just a fraction of that for the pools), I'm wondering how much
    is too much. Is several megabytes of of paged pool too much? I figure if
    its there, I may as well use it - but that's coming from the perspective of
    a user mode developer!

    Thanks,
    Joel
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    Why does the W2K DDK say the zone routines are obsolete and suggest we use
    look-aside lists? As I see it, these two sub-allocators serve two separate
    purposes. Zone routines are useful for allocating a large number of fixed
    size small blocks. Look-aside lists are useful for allocating larger blocks
    where there is a substantial difference between the high and low watermarks.
    I don't see that either sub-allocator is always a suitable replacement for
    the other.

    -----Original Message-----
    From: Bob Depelteau [mailto:[email protected]]
    Sent: Thursday, April 27, 2000 9:25 AM
    To: File Systems Developers
    Subject: [ntfsd] RE: memory management question


    You might want to look into using the zone memory routines.

    ExInitializeZone();
    ExExtendZone();
    ExAllocateFromZone();
    ExFreeToZone();
    ExIsFullZone();


    -----Original Message-----
    From: Smith, Joel [mailto:[email protected]]
    Sent: Thursday, April 27, 2000 10:08 AM
    To: File Systems Developers
    Subject: [ntfsd] memory management question



    Hello,
    Say you have a driver that needs to manage many small structures of
    fixed size such that the driver may have several megabytes worth of these
    structures allocated at any given time from paged pool. Since the
    structures are allocated from paged pool, I can imagine that it would be
    beneficial to allocate the structures from a contiguous heap of memory to
    preserve 'locality of reference': one page fault will bring in tens or
    hundreds of these structures reducing overall paging of the system as these
    structures are accessed.

    Is it worth the effort of doing some of your own memory management
    or will the VMM do an adequate job of preserving this locality of reference
    (or does it even try)? Looking at the DDK documentation, it seems that
    'zones' were designed to address this issue, however, they have been
    declared obsolete. The documentation suggests 'lookaside lists', although
    lookaside lists seem to address a different need than did zones; lookaside
    lists seem not to be designed to reduce fragmentation and preserve locality
    of reference as I've described.

    So, Will lookaside lists help here? Should I use zones to reduce
    fragmentation? Or should I just assume the VMM will do some work to help
    out with fragmentation? Any suggestions?

    BTW are there any guidelines as to how much address space a well
    mannered driver should consume? Given that I'm sharing my 2GB with everyon
    else now (and just a fraction of that for the pools), I'm wondering how much
    is too much. Is several megabytes of of paged pool too much? I figure if
    its there, I may as well use it - but that's coming from the perspective of
    a user mode developer!

    Thanks,
    Joel
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    Zones are not interlocked - they require external synchronization - and
    can't shrink. Build your own out of SLISTs.

    You should also have a development-cycle switch to turn it off and use
    regular pool. This will let you get the most out of using the verifier
    against the driver.

    -----Original Message-----
    From: Rob Fuller [mailto:[email protected]]
    Sent: Thursday, April 27, 2000 7:43 AM
    To: File Systems Developers
    Subject: [ntfsd] RE: memory management question


    Why does the W2K DDK say the zone routines are obsolete and suggest we
    use look-aside lists? As I see it, these two sub-allocators serve two
    separate purposes. Zone routines are useful for allocating a large
    number of fixed size small blocks. Look-aside lists are useful for
    allocating larger blocks where there is a substantial difference between
    the high and low watermarks. I don't see that either sub-allocator is
    always a suitable replacement for the other.

    -----Original Message-----
    From: Bob Depelteau [mailto:[email protected]]
    Sent: Thursday, April 27, 2000 9:25 AM
    To: File Systems Developers
    Subject: [ntfsd] RE: memory management question


    You might want to look into using the zone memory routines.

    ExInitializeZone();
    ExExtendZone();
    ExAllocateFromZone();
    ExFreeToZone();
    ExIsFullZone();


    -----Original Message-----
    From: Smith, Joel [mailto:[email protected]]
    Sent: Thursday, April 27, 2000 10:08 AM
    To: File Systems Developers
    Subject: [ntfsd] memory management question



    Hello,
    Say you have a driver that needs to manage many small structures
    of fixed size such that the driver may have several megabytes worth of
    these structures allocated at any given time from paged pool. Since the
    structures are allocated from paged pool, I can imagine that it would be
    beneficial to allocate the structures from a contiguous heap of memory
    to preserve 'locality of reference': one page fault will bring in tens
    or hundreds of these structures reducing overall paging of the system as
    these structures are accessed.

    Is it worth the effort of doing some of your own memory
    management or will the VMM do an adequate job of preserving this
    locality of reference (or does it even try)? Looking at the DDK
    documentation, it seems that 'zones' were designed to address this
    issue, however, they have been declared obsolete. The documentation
    suggests 'lookaside lists', although lookaside lists seem to address a
    different need than did zones; lookaside lists seem not to be designed
    to reduce fragmentation and preserve locality of reference as I've
    described.

    So, Will lookaside lists help here? Should I use zones to
    reduce fragmentation? Or should I just assume the VMM will do some work
    to help out with fragmentation? Any suggestions?

    BTW are there any guidelines as to how much address space a well
    mannered driver should consume? Given that I'm sharing my 2GB with
    everyon else now (and just a fraction of that for the pools), I'm
    wondering how much is too much. Is several megabytes of of paged pool
    too much? I figure if its there, I may as well use it - but that's
    coming from the perspective of a user mode developer!

    Thanks,
    Joel
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    I think you're right. A good example of this is Lpc susbsystem which still
    uses zones for his tasks.

    > -----P?vodn? zpr?va-----
    > Od: Rob Fuller [SMTP:[email protected]]
    > Odesl?no: 27. dubna 2000 16:43
    > Komu: File Systems Developers
    > P?edm?t: [ntfsd] RE: memory management question
    >
    > Why does the W2K DDK say the zone routines are obsolete and suggest we use
    > look-aside lists? As I see it, these two sub-allocators serve two
    > separate purposes. Zone routines are useful for allocating a large number
    > of fixed size small blocks. Look-aside lists are useful for allocating
    > larger blocks where there is a substantial difference between the high and
    > low watermarks. I don't see that either sub-allocator is always a
    > suitable replacement for the other.
    >
    > -----Original Message-----
    > From: Bob Depelteau [mailto:[email protected]]
    > Sent: Thursday, April 27, 2000 9:25 AM
    > To: File Systems Developers
    > Subject: [ntfsd] RE: memory management question
    >
    >
    > You might want to look into using the zone memory routines.
    >
    > ExInitializeZone();
    > ExExtendZone();
    > ExAllocateFromZone();
    > ExFreeToZone();
    > ExIsFullZone();
    >
    >
    > -----Original Message-----
    > From: Smith, Joel [mailto:[email protected]]
    > Sent: Thursday, April 27, 2000 10:08 AM
    > To: File Systems Developers
    > Subject: [ntfsd] memory management question
    >
    >
    >
    > Hello,
    > Say you have a driver that needs to manage many
    > small structures of fixed size such that the driver may have several
    > megabytes worth of these structures allocated at any given time from paged
    > pool. Since the structures are allocated from paged pool, I can imagine
    > that it would be beneficial to allocate the structures from a contiguous
    > heap of memory to preserve 'locality of reference': one page fault will
    > bring in tens or hundreds of these structures reducing overall paging of
    > the system as these structures are accessed.
    >
    > Is it worth the effort of doing some of your own
    > memory management or will the VMM do an adequate job of preserving this
    > locality of reference (or does it even try)? Looking at the DDK
    > documentation, it seems that 'zones' were designed to address this issue,
    > however, they have been declared obsolete. The documentation suggests
    > 'lookaside lists', although lookaside lists seem to address a different
    > need than did zones; lookaside lists seem not to be designed to reduce
    > fragmentation and preserve locality of reference as I've described.
    >
    > So, Will lookaside lists help here? Should I use
    > zones to reduce fragmentation? Or should I just assume the VMM will do
    > some work to help out with fragmentation? Any suggestions?
    >
    > BTW are there any guidelines as to how much address
    > space a well mannered driver should consume? Given that I'm sharing my
    > 2GB with everyon else now (and just a fraction of that for the pools), I'm
    > wondering how much is too much. Is several megabytes of of paged pool too
    > much? I figure if its there, I may as well use it - but that's coming
    > from the perspective of a user mode developer!
    >
    > Thanks,
    > Joel
    >
Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Upcoming OSR Seminars
OSR has suspended in-person seminars due to the Covid-19 outbreak. But, don't miss your training! Attend via the internet instead!
Writing WDF Drivers 7 Dec 2020 LIVE ONLINE
Internals & Software Drivers 25 Jan 2021 LIVE ONLINE
Developing Minifilters 8 March 2021 LIVE ONLINE