I have a function that allocates a structure using ExAllocatePool at the
beginning of the function and then frees that memory using ExFreePool at the
end of the function.
Development claims:
2.The function is called a lot - making allocation\deallocation from it not
a good idea for a product that should run for months without restart.
Is this true? Is memory management in Windows so bad that
allocating/deallocating memory in this way will fragment memory severely and
degrade performance over time? Am I the only one who allocates/deallocates
memory this way? Aren’t any of the other driver’s allocating/deallocating
memory? Won’t they also cause memory fragmentation over time? This claim
does not seem reasonable to me.
Best Regards,
Joe
My experience says that it (2) is true. Frequent alloc/free from the
pool leads to significant performance degradation of the system overall.
What I do in my driver to address that is using set of lookaside lists
as an alternative source of memory allocation. Since memory that you’re
allocating is short living there is a great chance that lookaside allocs
are not going to consume a lot of memory. So, what I do (in some
details) is that I have 5 lookaside lists: for 64, 128, 256, 512 and
1024 bytes chunks of memory. In my main alloc routine I look at the size
of memory needed and allocate it from the appropriate list (basically,
from that alloc routine I return a memory descriptor that consist of a
pointer to the allocated memory and lookaside list index so on “free” I
know which list to use). If required memory size exceeds the max
lookaside alloc size then I allocate memory from the regular pool.
May be it’s not the best technique but simple and efficient enough (to
me)
Regards,
Vladimir
-----Original Message-----
From: Joe Stivaletta [mailto:xxxxx@rcn.com]
Sent: Monday, August 09, 2004 7:39 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Memory Fragmentation
I have a function that allocates a structure using ExAllocatePool at the
beginning of the function and then frees that memory using ExFreePool at
the
end of the function.
Development claims:
2.The function is called a lot - making allocation\deallocation from it
not
a good idea for a product that should run for months without restart.
Is this true? Is memory management in Windows so bad that
allocating/deallocating memory in this way will fragment memory severely
and
degrade performance over time? Am I the only one who
allocates/deallocates
memory this way? Aren’t any of the other driver’s
allocating/deallocating
memory? Won’t they also cause memory fragmentation over time? This
claim
does not seem reasonable to me.
Best Regards,
Joe
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
You are currently subscribed to ntdev as:
xxxxx@borland.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
Is there any particular reason you’re not allocating the structure on
the stack, e.g. as a local to the function?
Chuck
----- Original Message -----
From: “Joe Stivaletta”
Newsgroups: ntdev
To: “Windows System Software Devs Interest List”
Sent: Monday, August 09, 2004 9:38 PM
Subject: [ntdev] Memory Fragmentation
> I have a function that allocates a structure using ExAllocatePool at
the
> beginning of the function and then frees that memory using ExFreePool
at the
> end of the function.
>
> Development claims:
> 2.The function is called a lot - making allocation\deallocation from
it not
> a good idea for a product that should run for months without restart.
>
> Is this true? Is memory management in Windows so bad that
> allocating/deallocating memory in this way will fragment memory
severely and
> degrade performance over time? Am I the only one who
allocates/deallocates
> memory this way? Aren’t any of the other driver’s
allocating/deallocating
> memory? Won’t they also cause memory fragmentation over time? This
claim
> does not seem reasonable to me.
>
> Best Regards,
> Joe
That depends.
First, yes frequent allocations of small chunks from the pools (NonPaged or
Paged) could result in fragmentation. However, the OS provides a lookaside
list for small size allocations to prevent this. On XP I am not sure the
size, but on Win2K I believe anything equal to or smaller than 256 bytes is
allocated from this lookaside list.
If you have frequent allocations that are larger than this, then you should
try to create your own lookaside list. If the memory chunks are all the
same size, or you know the maximum size, this is easy. This is where
Windows differs from most other operating systems and which leads to its
flakey behavior. Windows pushes WAY to much complexity to the driver
developer. You have to understand a great deal about the operation of the
OS in order to create a driver which plays nicely in the system. I think it
is unfortunate, but that is how it is.
Back to the problem at hand. If you are executing numerous memory
allocations during the execution of your driver you likely are not
architecting your driver correctly. Memory is never guaranteed to be
accessible (resource rebalance in the OS can make the pools temporarily
inaccesible). Thus you should try to allocate as much of the memory that
your driver will need at init time or in the device extension (or miniport
adapter context for NDIS). In this way your driver does not have to worry
about failing memory allocations during operation and this removes a great
deal of error checking code. Even with lookaside lists your allocations can
fail which is why I try to allocate my memory up front if possible. Just my
$0.02 BTW.
Unfortunately you can’t specify a start size for the lookaside lists which I
find to be rather odd given the fact that the pools can be inaccesible at
times. This is the one area where I have to agree with you that Windows is
kind of screwed up. I don’t know how you can limit access to the memory,
and yet give the developer no way to hedge against there not being memory
available. Again, this is why I try to allocate everything up front. Not
always possible with DMA and such though. Then again, resource rebalance is
a rare occurance, and I have never had pool fragmentation problems. Just
try to protect the best you can.
–
Bill McKenzie
Software Engineer - Prism 802.11 Wireless Solutions
Conexant Systems, Inc.
“Joe Stivaletta” wrote in message news:xxxxx@ntdev…
> I have a function that allocates a structure using ExAllocatePool at the
> beginning of the function and then frees that memory using ExFreePool at
the
> end of the function.
>
> Development claims:
> 2.The function is called a lot - making allocation\deallocation from it
not
> a good idea for a product that should run for months without restart.
>
> Is this true? Is memory management in Windows so bad that
> allocating/deallocating memory in this way will fragment memory severely
and
> degrade performance over time? Am I the only one who
allocates/deallocates
> memory this way? Aren’t any of the other driver’s allocating/deallocating
> memory? Won’t they also cause memory fragmentation over time? This
claim
> does not seem reasonable to me.
>
> Best Regards,
> Joe
>
>
>
There is a general article covering this subject from Microsoft:
http://www.microsoft.com/whdc/driver/perform/mem-alloc.mspx
Regards,
Volker
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of Joe Stivaletta
Sent: Monday, August 09, 2004 4:39 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] Memory Fragmentation
I have a function that allocates a structure using
ExAllocatePool at the
beginning of the function and then frees that memory using
ExFreePool at the
end of the function.
Development claims:
2.The function is called a lot - making
allocation\deallocation from it not
a good idea for a product that should run for months without restart.
Is this true? Is memory management in Windows so bad that
allocating/deallocating memory in this way will fragment
memory severely and
degrade performance over time? Am I the only one who
allocates/deallocates
memory this way? Aren’t any of the other driver’s
allocating/deallocating
memory? Won’t they also cause memory fragmentation over time?
This claim
does not seem reasonable to me.
Best Regards,
Joe
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
You are currently
subscribed to ntdev as: xxxxx@baslerweb.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
Yes, the memory block can get rather large; much too large for the stack.
“Chuck Batson” wrote in message
news:xxxxx@ntdev…
> Is there any particular reason you’re not allocating the structure on
> the stack, e.g. as a local to the function?
>
> Chuck
>
> ----- Original Message -----
> From: “Joe Stivaletta”
> Newsgroups: ntdev
> To: “Windows System Software Devs Interest List”
> Sent: Monday, August 09, 2004 9:38 PM
> Subject: [ntdev] Memory Fragmentation
>
>
> > I have a function that allocates a structure using ExAllocatePool at
> the
> > beginning of the function and then frees that memory using ExFreePool
> at the
> > end of the function.
> >
> > Development claims:
> > 2.The function is called a lot - making allocation\deallocation from
> it not
> > a good idea for a product that should run for months without restart.
> >
> > Is this true? Is memory management in Windows so bad that
> > allocating/deallocating memory in this way will fragment memory
> severely and
> > degrade performance over time? Am I the only one who
> allocates/deallocates
> > memory this way? Aren’t any of the other driver’s
> allocating/deallocating
> > memory? Won’t they also cause memory fragmentation over time? This
> claim
> > does not seem reasonable to me.
> >
> > Best Regards,
> > Joe
>
>
Is the use of this function serialized (i.e. do you ensure in your
driver that only one thread at a time can be calling it, or does the
function ensure that only one thread at a time is within it)? If so you
could consider allocating the memory either during PNP start or on your
first create (and freeing it in remove or the last close request).
The pool does become framented over long periods of time (just like with
any memory allocator that can’t reorganize live allocations). However
you would be more concerned (I imagine) with the possibility that there
won’t be enough contiguous space within the fragmented pool for your
“large” memory block, where smaller allocations could still succeed. If
this happens and the memory manager has no room to grow the pool, your
allocation will fail (and you’ll have to cleanup and return an error to
your client - your driver shouldn’t crash when the system runs low on
memory).
-p
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Joe Stivaletta
Sent: Monday, August 09, 2004 10:45 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Memory Fragmentation
Yes, the memory block can get rather large; much too large for the
stack.
“Chuck Batson” wrote in message
news:xxxxx@ntdev…
> Is there any particular reason you’re not allocating the structure on
> the stack, e.g. as a local to the function?
>
> Chuck
>
> ----- Original Message -----
> From: “Joe Stivaletta”
> Newsgroups: ntdev
> To: “Windows System Software Devs Interest List”
> Sent: Monday, August 09, 2004 9:38 PM
> Subject: [ntdev] Memory Fragmentation
>
>
> > I have a function that allocates a structure using ExAllocatePool at
> the
> > beginning of the function and then frees that memory using
> > ExFreePool
> at the
> > end of the function.
> >
> > Development claims:
> > 2.The function is called a lot - making allocation\deallocation from
> it not
> > a good idea for a product that should run for months without
restart.
> >
> > Is this true? Is memory management in Windows so bad that
> > allocating/deallocating memory in this way will fragment memory
> severely and
> > degrade performance over time? Am I the only one who
> allocates/deallocates
> > memory this way? Aren’t any of the other driver’s
> allocating/deallocating
> > memory? Won’t they also cause memory fragmentation over time?
This
> claim
> > does not seem reasonable to me.
> >
> > Best Regards,
> > Joe
>
>
—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
You are currently subscribed to ntdev as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
(1) Let me add one more caveat to this. The display drivers have a memory
mangement routines in them and they use their own. Just look at a sample
display driver and see have they keep track the their Device Bitmaps that
they store them offscreen memory. They do use Lookaside list but just keep
track of the memory by “brute”. This is proably much more complicated than
what you need but it is memory management. Note that Microsoft will be
supplying Video Memory Mangement routine in Longhorn. So you may wish to
look at the Longhorn DDK as well. A very fascinating subject. Memory
managment. I must say I like the simplicity the multiple Lookaside lists.
Thanks
William Michael Jones
“Joe Stivaletta” wrote in message news:xxxxx@ntdev…
> I have a function that allocates a structure using ExAllocatePool at the
> beginning of the function and then frees that memory using ExFreePool at
the
> end of the function.
>
> Development claims:
> 2.The function is called a lot - making allocation\deallocation from it
not
> a good idea for a product that should run for months without restart.
>
> Is this true? Is memory management in Windows so bad that
> allocating/deallocating memory in this way will fragment memory severely
and
> degrade performance over time? Am I the only one who
allocates/deallocates
> memory this way? Aren’t any of the other driver’s allocating/deallocating
> memory? Won’t they also cause memory fragmentation over time? This
claim
> does not seem reasonable to me.
>
> Best Regards,
> Joe
>
>
>
I was testing the driver for a separate problem. Since I was running in a
checked version of Windows Server2003 with Driver Verifier, MM threw an
exception due to a page fault. When I analyzed the problem, I discovered
that our routine was calling a system function which resides in pageable
memory at IRQL=2. Upon investigation, I discovered our function uses a
spinlock to act as a critical section while scanning the memory resource
using that pageable function. My proposal was to make a temporary copy of
the array within the spinlock, so that the function could scan a copy of the
array outside of the spinlock. This array can be accessed from other
threads on other processors.
“Peter Wieland” wrote in message
news:xxxxx@ntdev…
Is the use of this function serialized (i.e. do you ensure in your
driver that only one thread at a time can be calling it, or does the
function ensure that only one thread at a time is within it)? If so you
could consider allocating the memory either during PNP start or on your
first create (and freeing it in remove or the last close request).
The pool does become framented over long periods of time (just like with
any memory allocator that can’t reorganize live allocations). However
you would be more concerned (I imagine) with the possibility that there
won’t be enough contiguous space within the fragmented pool for your
“large” memory block, where smaller allocations could still succeed. If
this happens and the memory manager has no room to grow the pool, your
allocation will fail (and you’ll have to cleanup and return an error to
your client - your driver shouldn’t crash when the system runs low on
memory).
-p
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Joe Stivaletta
Sent: Monday, August 09, 2004 10:45 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Memory Fragmentation
Yes, the memory block can get rather large; much too large for the
stack.
“Chuck Batson” wrote in message
news:xxxxx@ntdev…
> Is there any particular reason you’re not allocating the structure on
> the stack, e.g. as a local to the function?
>
> Chuck
>
> ----- Original Message -----
> From: “Joe Stivaletta”
> Newsgroups: ntdev
> To: “Windows System Software Devs Interest List”
> Sent: Monday, August 09, 2004 9:38 PM
> Subject: [ntdev] Memory Fragmentation
>
>
> > I have a function that allocates a structure using ExAllocatePool at
> the
> > beginning of the function and then frees that memory using
> > ExFreePool
> at the
> > end of the function.
> >
> > Development claims:
> > 2.The function is called a lot - making allocation\deallocation from
> it not
> > a good idea for a product that should run for months without
restart.
> >
> > Is this true? Is memory management in Windows so bad that
> > allocating/deallocating memory in this way will fragment memory
> severely and
> > degrade performance over time? Am I the only one who
> allocates/deallocates
> > memory this way? Aren’t any of the other driver’s
> allocating/deallocating
> > memory? Won’t they also cause memory fragmentation over time?
This
> claim
> > does not seem reasonable to me.
> >
> > Best Regards,
> > Joe
>
>
—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
You are currently subscribed to ntdev as: xxxxx@windows.microsoft.com
To unsubscribe send a blank email to xxxxx@lists.osr.com
> Development claims:
2.The function is called a lot - making allocation\deallocation from it not
a good idea for a product that should run for months without restart.
Is this true? Is memory management in Windows so bad that
allocating/deallocating memory in this way will fragment memory severely and
degrade performance over time?
Look yourself. Windows pool manager is a binary buddy allocator, which also
uses the per-CPU lookaside lists for the standard size blocks. From this, you
can make your own conclusions about the possible performance penalty (very
low).
More so, such an approach is explicitly recommended to reduce the stack usage.
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
The DirectDraw chapter of the DDK documents the video memory allocator
routines.
Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
----- Original Message -----
From: “Michael Jones”
Newsgroups: ntdev
To: “Windows System Software Devs Interest List”
Sent: Monday, August 09, 2004 10:36 PM
Subject: Re:[ntdev] Memory Fragmentation
> (1) Let me add one more caveat to this. The display drivers have a memory
> mangement routines in them and they use their own. Just look at a sample
> display driver and see have they keep track the their Device Bitmaps that
> they store them offscreen memory. They do use Lookaside list but just keep
> track of the memory by “brute”. This is proably much more complicated than
> what you need but it is memory management. Note that Microsoft will be
> supplying Video Memory Mangement routine in Longhorn. So you may wish to
> look at the Longhorn DDK as well. A very fascinating subject. Memory
> managment. I must say I like the simplicity the multiple Lookaside lists.
>
> Thanks
> William Michael Jones
>
> “Joe Stivaletta” wrote in message news:xxxxx@ntdev…
> > I have a function that allocates a structure using ExAllocatePool at the
> > beginning of the function and then frees that memory using ExFreePool at
> the
> > end of the function.
> >
> > Development claims:
> > 2.The function is called a lot - making allocation\deallocation from it
> not
> > a good idea for a product that should run for months without restart.
> >
> > Is this true? Is memory management in Windows so bad that
> > allocating/deallocating memory in this way will fragment memory severely
> and
> > degrade performance over time? Am I the only one who
> allocates/deallocates
> > memory this way? Aren’t any of the other driver’s allocating/deallocating
> > memory? Won’t they also cause memory fragmentation over time? This
> claim
> > does not seem reasonable to me.
> >
> > Best Regards,
> > Joe
> >
> >
> >
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com