memory management

Here’s the problem. Our driver needs to allocate a pretty big piece of
memory (like 8Mb). Allocating it in non paged pool is not a very good idea
(or is it?). Can I allocate memory from paged pool and lock it somehow when
I running, say, on DISPATCH_LEVEL? I did not find a definite and easy to
understand answer in DDK.

Alexey Logachyov
xxxxx@vba.com.by
VirusBlokAda Ltd.
http://www.vba.com.by


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Depends … if it must be contiguous then it has to be NPP. If not then
allocate it from paged pool. Create an MDL for it and lock it down.

Gary G. Little
Broadband Storage, Inc.
xxxxx@broadstor.com
xxxxx@inland.net

-----Original Message-----
From: Alexey Logachyov [mailto:xxxxx@vba.com.by]
Sent: Thursday, January 24, 2002 11:46 AM
To: NT Developers Interest List
Subject: [ntdev] memory management

Here’s the problem. Our driver needs to allocate a pretty big piece of
memory (like 8Mb). Allocating it in non paged pool is not a very good idea
(or is it?). Can I allocate memory from paged pool and lock it somehow when
I running, say, on DISPATCH_LEVEL? I did not find a definite and easy to
understand answer in DDK.

Alexey Logachyov
xxxxx@vba.com.by
VirusBlokAda Ltd.
http://www.vba.com.by


You are currently subscribed to ntdev as: xxxxx@broadstor.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Hello,

If you definitely want to access this memory on dispatch level it must be
allocated from non-paged pool or this memory must be locked before you come
to dispatch level if allocated from paged pool. There is no way to lock
pages from paged pool while being on dispatch level (these pages can be in
the swap file at that moment).

Allocating from paged pool is preferable, but you will have to lock parts of
your buffer (which are expected to be accessed on dispatch level) before you
come dispatch. If this is not possible in your model you’d better use
nonpaged pool.

Best Regards,
Vadim
http://www.ntndis.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of Alexey Logachyov
Sent: Thursday, January 24, 2002 10:46 PM
To: NT Developers Interest List
Subject: [ntdev] memory management

Here’s the problem. Our driver needs to allocate a pretty big piece of
memory (like 8Mb). Allocating it in non paged pool is not a very good idea
(or is it?). Can I allocate memory from paged pool and lock it somehow when
I running, say, on DISPATCH_LEVEL? I did not find a definite and easy to
understand answer in DDK.

Alexey Logachyov
xxxxx@vba.com.by
VirusBlokAda Ltd.
http://www.vba.com.by


You are currently subscribed to ntdev as: xxxxx@pcausa.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

8MB is not that big to worry too much about. And anyway no, you can’t lock
down paged pool at DISPATCH_LEVEL as the locking down is likely to cause a
page fault. Contiguous memory for DMA requires nonpaged pool.

-----Original Message-----
From: Alexey Logachyov [mailto:xxxxx@vba.com.by]
Sent: Thursday, January 24, 2002 2:46 PM
To: NT Developers Interest List
Subject: [ntdev] memory management

Here’s the problem. Our driver needs to allocate a pretty big
piece of memory (like 8Mb). Allocating it in non paged pool
is not a very good idea (or is it?). Can I allocate memory
from paged pool and lock it somehow when I running, say, on
DISPATCH_LEVEL? I did not find a definite and easy to
understand answer in DDK.

Alexey Logachyov
xxxxx@vba.com.by
VirusBlokAda Ltd.
http://www.vba.com.by


You are currently subscribed to ntdev as:
xxxxx@stratus.com To unsubscribe send a blank email to
leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

“Alexey Logachyov” wrote in message news:xxxxx@ntdev…
>
> Here’s the problem. Our driver needs to allocate a pretty big piece of
> memory (like 8Mb). Allocating it in non paged pool is not a very good idea
> (or is it?).
>

It depends on what you want to use the memory FOR.

It is not true that allocating the memory from non-paged pool assures you
that it’s contiguous (sorry, Gary – it’s ALMOST always true, but not
ALWAYS).

If you need the memory to be physically contiguous (no contiguous for DMA,
but really physically contiguous), you call
MmAllocateContiguousMemorySpecifyCache(…); Note that this even allows you
to specify min and max physical address boundaries.

If you need the memory to be logically contiguous for DMA use, you
call ->AllocateCommonBuffer(…), which returns you both a logical address
suitable for DMA and a kernel virtual address.

If you MUST have non-cached memory, you can call
MmAllocateNonCachedMemory();

If you don’t need either one (it doesn’t need to be contiguous, you don’t
need a mapping for DMA, it can be cached… in other words, you just need a
big honkin’ buffer) you can try several alternatives to see which works best
according to your system’s configuration and workload:

1) Allocate from non-paged pool

2) Create a section of the required size and lock it (all or piece by piece)
using an MDL

3) For XP/.NET, MmAllocatePagesForMdl(…)

Hope that helps,

Peter
OSR


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

I must be missing something - what’s the difference between allocating
something from the NPP and allocating something from the PP but locking it
down? Does the latter buy you something?

  • Dennis

Dennis Merrill
Embedded Systems Engineer
Thermo Electron Corporation
Spectroscopy Division

-----Original Message-----
From: Gary Little [mailto:xxxxx@Broadstor.com]
Sent: Thursday, January 24, 2002 2:32 PM
To: NT Developers Interest List
Subject: [ntdev] RE: memory management

Depends … if it must be contiguous then it has to be NPP. If not then
allocate it from paged pool. Create an MDL for it and lock it down.

Gary G. Little
Broadband Storage, Inc.
xxxxx@broadstor.com
xxxxx@inland.net

-----Original Message-----
From: Alexey Logachyov [mailto:xxxxx@vba.com.by]
Sent: Thursday, January 24, 2002 11:46 AM
To: NT Developers Interest List
Subject: [ntdev] memory management

Here’s the problem. Our driver needs to allocate a pretty big piece of
memory (like 8Mb). Allocating it in non paged pool is not a very good idea
(or is it?). Can I allocate memory from paged pool and lock it somehow when
I running, say, on DISPATCH_LEVEL? I did not find a definite and easy to
understand answer in DDK.

Alexey Logachyov
xxxxx@vba.com.by
VirusBlokAda Ltd.
http://www.vba.com.by


You are currently subscribed to ntdev as: xxxxx@broadstor.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: xxxxx@thermonicolet.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

RE: [ntdev] RE: memory management wrote in
message news:xxxxx@ntdev…
> I must be missing something - what’s the difference between allocating
something from the NPP and allocating
> something from the PP but locking it down? Does the latter buy you
something?
>

Well, there’s a limit to the maximum size of the nonPaged pool… depending
on how you allocate the memory, the pages can come from different places.
If I’m not mistaken, for example, non-paged pool pages can come from large
pages (good for performance, but possibly shared with other O/S code).
Paged pool pages do not.

I’m NOT a fan of allocating paged pool pages and locking them down. If God
intended for those pool pages to be locked, he would have put them in the
nonpaged pool in the first place :slight_smile:

Peter
OSR


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Dennis Yes … It doesn’t waste NPP if you don’t need contiguous memory.

Peter … I knew that … just didn’t say it. NPP is not necessarily
contiguous but contiguous memory is NPP.

(At least that is how I understand it, though DMA contiguous may be the
qualifier on the use of NPP.)

Gary G. Little
Broadband Storage, Inc.
xxxxx@broadstor.com
xxxxx@inland.net

-----Original Message-----
From: xxxxx@thermonicolet.com [mailto:xxxxx@thermonicolet.com]
Sent: Thursday, January 24, 2002 2:13 PM
To: NT Developers Interest List
Subject: [ntdev] RE: memory management

I must be missing something - what’s the difference between allocating
something from the NPP and allocating something from the PP but locking it
down? Does the latter buy you something?

  • Dennis

Dennis Merrill
Embedded Systems Engineer
Thermo Electron Corporation
Spectroscopy Division

-----Original Message-----
From: Gary Little [ mailto:xxxxx@Broadstor.com
mailto:xxxxx ]
Sent: Thursday, January 24, 2002 2:32 PM
To: NT Developers Interest List
Subject: [ntdev] RE: memory management
Depends … if it must be contiguous then it has to be NPP. If not then
allocate it from paged pool. Create an MDL for it and lock it down.
Gary G. Little
Broadband Storage, Inc.
xxxxx@broadstor.com
xxxxx@inland.net
-----Original Message-----
From: Alexey Logachyov [mailto:xxxxx@vba.com.by mailto:xxxxx]
Sent: Thursday, January 24, 2002 11:46 AM
To: NT Developers Interest List
Subject: [ntdev] memory management
Here’s the problem. Our driver needs to allocate a pretty big piece of
memory (like 8Mb). Allocating it in non paged pool is not a very good idea
(or is it?). Can I allocate memory from paged pool and lock it somehow when
I running, say, on DISPATCH_LEVEL? I did not find a definite and easy to
understand answer in DDK.
Alexey Logachyov
xxxxx@vba.com.by
VirusBlokAda Ltd.
http://www.vba.com.by http:


You are currently subscribed to ntdev as: xxxxx@broadstor.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntdev as: xxxxx@thermonicolet.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntdev as: xxxxx@broadstor.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com</http:></mailto:xxxxx></mailto:xxxxx>

RE: [ntdev] RE: memory managementWe do not need all the memory all the time.
Depending on what is happening we some of its part while the rest can be
paged out if system need memory.

Alexey Logachyov
xxxxx@vba.com.by
VirusBlokAda ltd.
http://www.vba.com.by

----- Original Message -----
From: xxxxx@thermonicolet.com
To: NT Developers Interest List
Sent: Friday, January 25, 2002 12:13 AM
Subject: [ntdev] RE: memory management

I must be missing something - what’s the difference between allocating
something from the NPP and allocating something from the PP but locking it
down? Does the latter buy you something?

  • Dennis

Dennis Merrill
Embedded Systems Engineer
Thermo Electron Corporation
Spectroscopy Division

-----Original Message-----
From: Gary Little [mailto:xxxxx@Broadstor.com]
Sent: Thursday, January 24, 2002 2:32 PM
To: NT Developers Interest List
Subject: [ntdev] RE: memory management
Depends … if it must be contiguous then it has to be NPP. If not then
allocate it from paged pool. Create an MDL for it and lock it down.
Gary G. Little
Broadband Storage, Inc.
xxxxx@broadstor.com
xxxxx@inland.net
-----Original Message-----
From: Alexey Logachyov [mailto:xxxxx@vba.com.by]
Sent: Thursday, January 24, 2002 11:46 AM
To: NT Developers Interest List
Subject: [ntdev] memory management
Here’s the problem. Our driver needs to allocate a pretty big piece of
memory (like 8Mb). Allocating it in non paged pool is not a very good idea
(or is it?). Can I allocate memory from paged pool and lock it somehow when
I running, say, on DISPATCH_LEVEL? I did not find a definite and easy to
understand answer in DDK.
Alexey Logachyov
xxxxx@vba.com.by
VirusBlokAda Ltd.
http://www.vba.com.by


You are currently subscribed to ntdev as: xxxxx@broadstor.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntdev as: xxxxx@thermonicolet.com
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

You are currently subscribed to ntdev as: xxxxx@vba.com.by
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

I do not need contiguos piece of memory. I just want to allocate memory for
internal use (I mean not for hardware use like DMA or something) and let
some parts be paged out when system needs more memory. So, I need some big
buffer where I can lock some block before use (this will actually happen
very rare as far as I run at high IRQL very rarely). I feel that MDL can
help me but I did not find good explanation on how it works in DDK. Could
someone write basic steps for doing that?

Alexey Logachyov
xxxxx@vba.com.by
VirusBlokAda ltd.
http://www.vba.com.by

----- Original Message -----
From: “Peter Viscarola”
Newsgroups: ntdev
To: “NT Developers Interest List”
Sent: Thursday, January 24, 2002 11:53 PM
Subject: [ntdev] Re: memory management

> “Alexey Logachyov” wrote in message news:xxxxx@ntdev…
> >
> > Here’s the problem. Our driver needs to allocate a pretty big piece of
> > memory (like 8Mb). Allocating it in non paged pool is not a very good
idea
> > (or is it?).
> >
>
> It depends on what you want to use the memory FOR.
>
> It is not true that allocating the memory from non-paged pool assures you
> that it’s contiguous (sorry, Gary – it’s ALMOST always true, but not
> ALWAYS).
>
> If you need the memory to be physically contiguous (no contiguous for DMA,
> but really physically contiguous), you call
> MmAllocateContiguousMemorySpecifyCache(…); Note that this even allows
you
> to specify min and max physical address boundaries.
>
> If you need the memory to be logically contiguous for DMA use, you
> call ->AllocateCommonBuffer(…), which returns you both a logical address
> suitable for DMA and a kernel virtual address.
>
> If you MUST have non-cached memory, you can call
> MmAllocateNonCachedMemory();
>
> If you don’t need either one (it doesn’t need to be contiguous, you don’t
> need a mapping for DMA, it can be cached… in other words, you just need
a
> big honkin’ buffer) you can try several alternatives to see which works
best
> according to your system’s configuration and workload:
>
> 1) Allocate from non-paged pool
>
> 2) Create a section of the required size and lock it (all or piece by
piece)
> using an MDL
>
> 3) For XP/.NET, MmAllocatePagesForMdl(…)
>
> Hope that helps,
>
> Peter
> OSR
>
>
>
>
>
> —
> You are currently subscribed to ntdev as: xxxxx@vba.com.by
> To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

> It is not true that allocating the memory from non-paged pool assures you

that it’s contiguous (sorry, Gary – it’s ALMOST always true, but not
ALWAYS).

IIRC NPP is divided to primary pool and expansion pool.
Primary pool is limited in size, looks like these pages are for NPP only, and not used for other allocations like process working
sets.
This memory is physically contiguous. Pages are allocated in physically contiguous chunks. They are also virtually contiguous due to
(0x0000000 physical - 0x80000000 virtual) mapping of 512MB which is always present in NT.
The addresses of the primary pool are like 0x8xxxxxxx.

Expansion pool uses any pages - not necessary contiguous - and then maps them to be virtually contiguous (for a multi-page
allocation) using system PTEs. These allocations are not physically contiguous at all, and have addresses like 0xfxxxxxxx.

Max


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

> IIRC NPP is divided to primary pool and expansion pool.

Primary pool is limited in size, looks like these pages are for NPP only, and
not used for other allocations like process working
sets.
This memory is physically contiguous. Pages are allocated in physically
contiguous chunks. They are also virtually contiguous due to
(0x0000000 physical - 0x80000000 virtual) mapping of 512MB which is always
present in NT.
The addresses of the primary pool are like 0x8xxxxxxx.

There was a posting from J.Hanrahan recently, where he said:

I hesitate to mention this, but there IS a workaround… but it applies
only on systems where the “large page” feature is available and used by
the OS, and furthermore only on systems with 512 MB or less physical
memory, and where the /3GB boot option is not in use. In such cases, all
of physical memory is already mapped into virtual address space, starting
with physical 0 = virtual 0x80000000, through physical 0x1FFFFFFF (512MB)
= virtual 0x9FFFFFFF. So if you have the physical address of something in
the first 512MB, it’s trivial to compute and use the equivalent virtual
address, IF the above criteria are all met. The OS uses this to avoid
double-mapping buffers to perform the “virtual map register” trick, for
example. ALso, if you try MmGetSystemAddressForMdl on an MDL that
describes a buffer of just one page, you’ll find that the resulting
address is in this range, inspecting the debugger’s !pte output will
reveal that it’s just a PDE with the “large page” bit set, and the virtual
address returned will NOT be unmapped when you free the MDL.

Now, no way would I build this knowledge into production code. It isn’t
supported for use outside the OS; these facts were learned by inspection.
And we have no idea if all those criteria will remain constant or even
relevant.


Do You Yahoo!?
Great stuff seeking new owners in Yahoo! Auctions!
http://auctions.yahoo.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com

Hi All,

I am developing an application under Win NT / 2K which requires more than
128 MB memory.This memory needs to be locked, but can be non-contiguous. It
is not
required while processing the interrupts.

From what I understood through MSDN & this list ,there is limitation on
maximum size of
non paged pool size pages. ( upto 128MB memory )

If system memory is sufficient,is there any workaround to allocate memory
more than 128MB ?

Thanks in advance,
Sundeep


----- Original Message -----
From: Peter Viscarola
Newsgroups: ntdev
To: NT Developers Interest List
Sent: Friday, January 25, 2002 4:50 AM
Subject: [ntdev] Re: memory management

> RE: [ntdev] RE: memory management wrote in
> message news:xxxxx@ntdev…
> > I must be missing something - what’s the difference between allocating
> something from the NPP and allocating
> > something from the PP but locking it down? Does the latter buy you
> something?
> >
>
> Well, there’s a limit to the maximum size of the nonPaged pool…
depending
> on how you allocate the memory, the pages can come from different places.
> If I’m not mistaken, for example, non-paged pool pages can come from large
> pages (good for performance, but possibly shared with other O/S code).
> Paged pool pages do not.
>
> I’m NOT a fan of allocating paged pool pages and locking them down. If
God
> intended for those pool pages to be locked, he would have put them in the
> nonpaged pool in the first place :slight_smile:
>
> Peter
> OSR
>
>
>
>
>
>
> —
> You are currently subscribed to ntdev as: xxxxx@nital.stpp.soft.net
> To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com


You are currently subscribed to ntdev as: $subst(‘Recip.EmailAddr’)
To unsubscribe send a blank email to leave-ntdev-$subst(‘Recip.MemberIDChar’)@lists.osr.com