memory management question

NTFSD
1

Hello,
Say you have a driver that needs to manage many small structures of
fixed size such that the driver may have several megabytes worth of these
structures allocated at any given time from paged pool. Since the
structures are allocated from paged pool, I can imagine that it would be
beneficial to allocate the structures from a contiguous heap of memory to
preserve ‘locality of reference’: one page fault will bring in tens or
hundreds of these structures reducing overall paging of the system as these
structures are accessed.
Is it worth the effort of doing some of your own memory management
or will the VMM do an adequate job of preserving this locality of reference
(or does it even try)? Looking at the DDK documentation, it seems that
‘zones’ were designed to address this issue, however, they have been
declared obsolete. The documentation suggests ‘lookaside lists’, although
lookaside lists seem to address a different need than did zones; lookaside
lists seem not to be designed to reduce fragmentation and preserve locality
of reference as I’ve described.
So, Will lookaside lists help here? Should I use zones to reduce
fragmentation? Or should I just assume the VMM will do some work to help
out with fragmentation? Any suggestions?
BTW are there any guidelines as to how much address space a well
mannered driver should consume? Given that I’m sharing my 2GB with everyon
else now (and just a fraction of that for the pools), I’m wondering how much
is too much. Is several megabytes of of paged pool too much? I figure if
its there, I may as well use it - but that’s coming from the perspective of
a user mode developer!

Thanks,
Joel

2

You might want to look into using the zone memory routines.

ExInitializeZone();
ExExtendZone();
ExAllocateFromZone();
ExFreeToZone();
ExIsFullZone();

-----Original Message-----
From: Smith, Joel [mailto:xxxxx@ntpsoftware.com]
Sent: Thursday, April 27, 2000 10:08 AM
To: File Systems Developers
Subject: [ntfsd] memory management question

Hello,
Say you have a driver that needs to manage many small structures of
fixed size such that the driver may have several megabytes worth of these
structures allocated at any given time from paged pool. Since the
structures are allocated from paged pool, I can imagine that it would be
beneficial to allocate the structures from a contiguous heap of memory to
preserve ‘locality of reference’: one page fault will bring in tens or
hundreds of these structures reducing overall paging of the system as these
structures are accessed.

Is it worth the effort of doing some of your own memory management
or will the VMM do an adequate job of preserving this locality of reference
(or does it even try)? Looking at the DDK documentation, it seems that
‘zones’ were designed to address this issue, however, they have been
declared obsolete. The documentation suggests ‘lookaside lists’, although
lookaside lists seem to address a different need than did zones; lookaside
lists seem not to be designed to reduce fragmentation and preserve locality
of reference as I’ve described.

So, Will lookaside lists help here? Should I use zones to reduce
fragmentation? Or should I just assume the VMM will do some work to help
out with fragmentation? Any suggestions?

BTW are there any guidelines as to how much address space a well
mannered driver should consume? Given that I’m sharing my 2GB with everyon
else now (and just a fraction of that for the pools), I’m wondering how much
is too much. Is several megabytes of of paged pool too much? I figure if
its there, I may as well use it - but that’s coming from the perspective of
a user mode developer!

Thanks,
Joel

3

Why does the W2K DDK say the zone routines are obsolete and suggest we use
look-aside lists? As I see it, these two sub-allocators serve two separate
purposes. Zone routines are useful for allocating a large number of fixed
size small blocks. Look-aside lists are useful for allocating larger blocks
where there is a substantial difference between the high and low watermarks.
I don’t see that either sub-allocator is always a suitable replacement for
the other.

-----Original Message-----
From: Bob Depelteau [mailto:xxxxx@OTG.com]
Sent: Thursday, April 27, 2000 9:25 AM
To: File Systems Developers
Subject: [ntfsd] RE: memory management question

You might want to look into using the zone memory routines.

ExInitializeZone();
ExExtendZone();
ExAllocateFromZone();
ExFreeToZone();
ExIsFullZone();

-----Original Message-----
From: Smith, Joel [mailto:xxxxx@ntpsoftware.com]
Sent: Thursday, April 27, 2000 10:08 AM
To: File Systems Developers
Subject: [ntfsd] memory management question

Hello,
Say you have a driver that needs to manage many small structures of
fixed size such that the driver may have several megabytes worth of these
structures allocated at any given time from paged pool. Since the
structures are allocated from paged pool, I can imagine that it would be
beneficial to allocate the structures from a contiguous heap of memory to
preserve ‘locality of reference’: one page fault will bring in tens or
hundreds of these structures reducing overall paging of the system as these
structures are accessed.

Is it worth the effort of doing some of your own memory management
or will the VMM do an adequate job of preserving this locality of reference
(or does it even try)? Looking at the DDK documentation, it seems that
‘zones’ were designed to address this issue, however, they have been
declared obsolete. The documentation suggests ‘lookaside lists’, although
lookaside lists seem to address a different need than did zones; lookaside
lists seem not to be designed to reduce fragmentation and preserve locality
of reference as I’ve described.

So, Will lookaside lists help here? Should I use zones to reduce
fragmentation? Or should I just assume the VMM will do some work to help
out with fragmentation? Any suggestions?

BTW are there any guidelines as to how much address space a well
mannered driver should consume? Given that I’m sharing my 2GB with everyon
else now (and just a fraction of that for the pools), I’m wondering how much
is too much. Is several megabytes of of paged pool too much? I figure if
its there, I may as well use it - but that’s coming from the perspective of
a user mode developer!

Thanks,
Joel

4

Zones are not interlocked - they require external synchronization - and
can’t shrink. Build your own out of SLISTs.

You should also have a development-cycle switch to turn it off and use
regular pool. This will let you get the most out of using the verifier
against the driver.

-----Original Message-----
From: Rob Fuller [mailto:xxxxx@NSISW.COM]
Sent: Thursday, April 27, 2000 7:43 AM
To: File Systems Developers
Subject: [ntfsd] RE: memory management question

Why does the W2K DDK say the zone routines are obsolete and suggest we
use look-aside lists? As I see it, these two sub-allocators serve two
separate purposes. Zone routines are useful for allocating a large
number of fixed size small blocks. Look-aside lists are useful for
allocating larger blocks where there is a substantial difference between
the high and low watermarks. I don’t see that either sub-allocator is
always a suitable replacement for the other.

-----Original Message-----
From: Bob Depelteau [mailto:xxxxx@OTG.com]
Sent: Thursday, April 27, 2000 9:25 AM
To: File Systems Developers
Subject: [ntfsd] RE: memory management question

You might want to look into using the zone memory routines.

ExInitializeZone();
ExExtendZone();
ExAllocateFromZone();
ExFreeToZone();
ExIsFullZone();

-----Original Message-----
From: Smith, Joel [mailto:xxxxx@ntpsoftware.com]
Sent: Thursday, April 27, 2000 10:08 AM
To: File Systems Developers
Subject: [ntfsd] memory management question

Hello,
Say you have a driver that needs to manage many small structures
of fixed size such that the driver may have several megabytes worth of
these structures allocated at any given time from paged pool. Since the
structures are allocated from paged pool, I can imagine that it would be
beneficial to allocate the structures from a contiguous heap of memory
to preserve ‘locality of reference’: one page fault will bring in tens
or hundreds of these structures reducing overall paging of the system as
these structures are accessed.

Is it worth the effort of doing some of your own memory
management or will the VMM do an adequate job of preserving this
locality of reference (or does it even try)? Looking at the DDK
documentation, it seems that ‘zones’ were designed to address this
issue, however, they have been declared obsolete. The documentation
suggests ‘lookaside lists’, although lookaside lists seem to address a
different need than did zones; lookaside lists seem not to be designed
to reduce fragmentation and preserve locality of reference as I’ve
described.

So, Will lookaside lists help here? Should I use zones to
reduce fragmentation? Or should I just assume the VMM will do some work
to help out with fragmentation? Any suggestions?

BTW are there any guidelines as to how much address space a well
mannered driver should consume? Given that I’m sharing my 2GB with
everyon else now (and just a fraction of that for the pools), I’m
wondering how much is too much. Is several megabytes of of paged pool
too much? I figure if its there, I may as well use it - but that’s
coming from the perspective of a user mode developer!

Thanks,
Joel

5

I think you’re right. A good example of this is Lpc susbsystem which still
uses zones for his tasks.

-----P?vodn? zpr?va-----
Od: Rob Fuller [SMTP:xxxxx@NSISW.COM]
Odesl?no: 27. dubna 2000 16:43
Komu: File Systems Developers
P?edm?t: [ntfsd] RE: memory management question

Why does the W2K DDK say the zone routines are obsolete and suggest we use
look-aside lists? As I see it, these two sub-allocators serve two
separate purposes. Zone routines are useful for allocating a large number
of fixed size small blocks. Look-aside lists are useful for allocating
larger blocks where there is a substantial difference between the high and
low watermarks. I don’t see that either sub-allocator is always a
suitable replacement for the other.

-----Original Message-----
From: Bob Depelteau [mailto:xxxxx@OTG.com]
Sent: Thursday, April 27, 2000 9:25 AM
To: File Systems Developers
Subject: [ntfsd] RE: memory management question

You might want to look into using the zone memory routines.

ExInitializeZone();
ExExtendZone();
ExAllocateFromZone();
ExFreeToZone();
ExIsFullZone();

-----Original Message-----
From: Smith, Joel [mailto:xxxxx@ntpsoftware.com]
Sent: Thursday, April 27, 2000 10:08 AM
To: File Systems Developers
Subject: [ntfsd] memory management question

Hello,
Say you have a driver that needs to manage many
small structures of fixed size such that the driver may have several
megabytes worth of these structures allocated at any given time from paged
pool. Since the structures are allocated from paged pool, I can imagine
that it would be beneficial to allocate the structures from a contiguous
heap of memory to preserve ‘locality of reference’: one page fault will
bring in tens or hundreds of these structures reducing overall paging of
the system as these structures are accessed.

Is it worth the effort of doing some of your own
memory management or will the VMM do an adequate job of preserving this
locality of reference (or does it even try)? Looking at the DDK
documentation, it seems that ‘zones’ were designed to address this issue,
however, they have been declared obsolete. The documentation suggests
‘lookaside lists’, although lookaside lists seem to address a different
need than did zones; lookaside lists seem not to be designed to reduce
fragmentation and preserve locality of reference as I’ve described.

So, Will lookaside lists help here? Should I use
zones to reduce fragmentation? Or should I just assume the VMM will do
some work to help out with fragmentation? Any suggestions?

BTW are there any guidelines as to how much address
space a well mannered driver should consume? Given that I’m sharing my
2GB with everyon else now (and just a fraction of that for the pools), I’m
wondering how much is too much. Is several megabytes of of paged pool too
much? I figure if its there, I may as well use it - but that’s coming
from the perspective of a user mode developer!

Thanks,
Joel