How to determine the maximum and currently-used paged pool size

There are documented interfaces to get notifications from the system that paged or non-paged pool are becoming depleted or have been replenished relative to system-defined high memory or low memory amounts. See ?Standard Event Objects? in the WDK documentation.

If the OP was really looking for hints that the system is low (or not) on pool memory, those should be used ? e.g. as a hint to free cached resources that aren?t critically needed for operation on low memory conditions.

  • S

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Prokash Sinha
Sent: Saturday, May 16, 2009 1:17 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] How to determine the maximum and currently-used paged pool size

Okay, I hate to see this goes on and on first because OP was really looking for global variables that would give him an indications within the next epoch (s)he is ready to go and grab or go and release the memory…

As few people pointed out there is no bullet proof method to do that, and I suspect OP also does not have that requirement. Otherwise OP would have go to the modeling it, rather than asking how it could be done by the driver.

For the sake of argument, lets say OP is in fact getting to know some estimated value within the next epoch ( let say within 10 ms on an average ). What are the approaches … ( For now ignore the UM & KM & HW )

(1) Well, differential equations is one approach. But we need a set of functionals that can easily form into DE and solve it by discretizing or whatever transform you can use. Big big question is what is the functional form to being with? I don’t have a clue …

(2) Well, it can be done by a simple statistical analysis: Mean and Sigma. Sure but what is the sample, what is the sample distribution. Again I sure don’t have a clue…

(3) What the problem signals to us?

A) We don’t know the shit, and it is random

B) What random? Random means nothing a functional form whose image domain is [0, 1]

C) Anything more? Of course there is some more … It is a continuous time and discrete state Random Process !.

D) That it?? Nope, this does not have a memory so (i) we don’t know when the next ( + or - ) consumptions are going to be. Oh, the inter arrival time is memory less. So it is exponential.

E) How about the payload ( + or - ) of the consumptions? Same thing as the along the time line.

Rest we talked about in this forum before … And quite unnecessary, until OP comes and say I want the model and say "Pro I know you did some mathematical modeling in the past, but sure hell you are an ugly model when I look at your face and compare with the pretty faces in the modeling business :slight_smile:

-pro
On Sat, May 16, 2009 at 12:17 PM, > wrote:
> I have every reason to believe that Calvin’s estimation based approach to determining the
> size of paged pool would yield a very satisfactory result. I suspect it would be not that much more
> state intensive nor complex than a good ‘peak/average’ estimator typical of network (or disk)
> throughput statistics - something found in kernel modes the world over (and in hardware too).

Please note that the things you have mentioned don’t require solving differential equations - all they need is a relatively simple statistical analysis like calculating mean and standard deviation. Therefore, Calvin’s post made me suspect that he would rather go for something significantly more complex than that
(for example, Fourier Transform)…

Anton Bassov


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

And that is exactly what the OP needs to follow and sing away “In the year 2020 …, in the year 3030 … in the year 5050 …”

-pro

Well, as if I’ve to sign something !
Prokash Sinha
http://prokash.squarespace.com
Success has many fathers, but failure is an orphan.

----- Original Message -----
From: Skywing
To: Windows System Software Devs Interest List
Sent: Saturday, May 16, 2009 5:41 PM
Subject: RE: [ntdev] How to determine the maximum and currently-used paged pool size

There are documented interfaces to get notifications from the system that paged or non-paged pool are becoming depleted or have been replenished relative to system-defined high memory or low memory amounts. See ?Standard Event Objects? in the WDK documentation.

If the OP was really looking for hints that the system is low (or not) on pool memory, those should be used ? e.g. as a hint to free cached resources that aren?t critically needed for operation on low memory conditions.

  • S

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Prokash Sinha
Sent: Saturday, May 16, 2009 1:17 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] How to determine the maximum and currently-used paged pool size

Okay, I hate to see this goes on and on first because OP was really looking for global variables that would give him an indications within the next epoch (s)he is ready to go and grab or go and release the memory…

As few people pointed out there is no bullet proof method to do that, and I suspect OP also does not have that requirement. Otherwise OP would have go to the modeling it, rather than asking how it could be done by the driver.

For the sake of argument, lets say OP is in fact getting to know some estimated value within the next epoch ( let say within 10 ms on an average ). What are the approaches … ( For now ignore the UM & KM & HW )

(1) Well, differential equations is one approach. But we need a set of functionals that can easily form into DE and solve it by discretizing or whatever transform you can use. Big big question is what is the functional form to being with? I don’t have a clue …

(2) Well, it can be done by a simple statistical analysis: Mean and Sigma. Sure but what is the sample, what is the sample distribution. Again I sure don’t have a clue…

(3) What the problem signals to us?

A) We don’t know the shit, and it is random

B) What random? Random means nothing a functional form whose image domain is [0, 1]

C) Anything more? Of course there is some more … It is a continuous time and discrete state Random Process !.

D) That it?? Nope, this does not have a memory so (i) we don’t know when the next ( + or - ) consumptions are going to be. Oh, the inter arrival time is memory less. So it is exponential.

E) How about the payload ( + or - ) of the consumptions? Same thing as the along the time line.

Rest we talked about in this forum before … And quite unnecessary, until OP comes and say I want the model and say "Pro I know you did some mathematical modeling in the past, but sure hell you are an ugly model when I look at your face and compare with the pretty faces in the modeling business :slight_smile:

-pro

On Sat, May 16, 2009 at 12:17 PM, wrote:

> I have every reason to believe that Calvin’s estimation based approach to determining the
> size of paged pool would yield a very satisfactory result. I suspect it would be not that much more
> state intensive nor complex than a good ‘peak/average’ estimator typical of network (or disk)
> throughput statistics - something found in kernel modes the world over (and in hardware too).

Please note that the things you have mentioned don’t require solving differential equations - all they need is a relatively simple statistical analysis like calculating mean and standard deviation. Therefore, Calvin’s post made me suspect that he would rather go for something significantly more complex than that
(for example, Fourier Transform)…

Anton Bassov


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

And make sure the algorithm has enough “damping” for the nominal input spectrum such that the output doesn’t oscillate, but at the same time, too much damping leads to awfully slow transient response-:slight_smile:

Calvin

— On Sat, 5/16/09, David R. Cattley wrote:

> From: David R. Cattley
> Subject: RE: [ntdev] How to determine the maximum and currently-used paged pool size
> To: “Windows System Software Devs Interest List”
> Date: Saturday, May 16, 2009, 3:22 PM
>
> Please note that the things you have mentioned don’t
> require solving
> differential equations? - all they need is a
> relatively simple statistical
> analysis like calculating mean and standard deviation.
>
>
> Someone can simplistically apply what they believe to be a
> good 1st order
> linear approximation to what they think is estimating the
> mean and/or std
> deviation.???The problem is that these
> ‘simple statistical analysis’ derived
> things do not always work that well for dynamic prediction,
> discrete data
> sets, and finite time (or limited time) observation.
>
> Sometimes they do, sometimes they do not.? Knowing
> when requires knowing
> more and choosing wisely.? Often the choice is to
> choose the simpler
> approach.? But picking it simply because it is simple
> (or the only one
> known) is not a ‘sound’ choice.? So it may well take a
> 3rd order Dif-E-Q
> solution to show that the residual value of the additional
> orders is
> contributing an insignificant set of terms to the
> solution.? But you can’t
> know that without asking the question and solving the
> model.? That was my
> only point.? Not that one or the other ‘answer’ was
> more meritorious.
>
> Thanks,
> -Dave
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars
> visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online
> at http://www.osronline.com/page.cfm?name=ListServer
>

>

I’m afraid no one will hire me to do kernel work, Calvin.
LOL.

Prokash Sinha
http://prokash.squarespace.com
Success has many fathers, but failure is an orphan.

If and only if your interviewer believes “this kind of thingy should have never been in kernel mode” -:))

Calvin Guan
Broadcom Corp.
Connecting Everything(r)

>If the OP was really looking for hints that the system is low (or not) on pool memory, those should be used

This seems to be most reasonable approach. The only problem is that it is just too boring for some folks here…

Anton Bassov

> We all know that you just love calculus, but how on Earth differential equations are related to the

KERNEL

Yes, especially the time derivative of the paged pool size across Windows versions.

:slight_smile:


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

>None of those is computational complex. My 98 Honda Accord even has such a close-loop control

system to meet the ridiculous emission regulation

I think most fuel injection computers are close-loop. The assembly called “lambda probe” is the sensor in this loop, and most modern cars have one (very easy to be destroyed by bad quality fuel).

People’s Republic of California.

So, California is communistic now?


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

Apologies for the massive delay in seeing this!

>Why do you believe that you need to do this?

I?m facing this excessive memory consumption problem in my FSD. For each opened file, this FSD maintains some amount of metadata in memory.
While catering to a test app that creates exorbitant number of files, it reaches to a point when it finds exhaustion of paged pool. At this point, FSD starts failing the IO operation and has to do appropriate roll back. However, rollback activity itself sometimes requires memory allocation.
So, I can?t really afford to end up in no-memory state?I have to know it beforehand that memory in on the verge of depletion and start failing the operations. I thought that knowing the maximum and available paged pool would have rescued me here?..basically, I was expecting (way too rich expectation seems) of the existence of a routine/variable which could give me something similar to what is given by !vm command.

>There are documented interfaces to get notifications from the system that paged or non-paged pool are becoming depleted or have been replenished relative to system-defined high memory or low memory amounts. See ?Standard Event Objects? in the WDK documentation.

Given my nescience to advanced mathematics, I find it plausible to walk on this path. I?ll see to it if it can be instrumental. Many thanks!

Preallocate enough memory while maintaining your metadata such that rollback never requires more allocations. Any rollback/cleanup path that requires an allocation to succeed is doomed to failure (and IMHO a poor design to begin with).

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@rediffmail.com
Sent: Sunday, May 17, 2009 11:57 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] How to determine the maximum and currently-used paged pool size

Apologies for the massive delay in seeing this!

>Why do you believe that you need to do this?

I?m facing this excessive memory consumption problem in my FSD. For each opened file, this FSD maintains some amount of metadata in memory.
While catering to a test app that creates exorbitant number of files, it reaches to a point when it finds exhaustion of paged pool. At this point, FSD starts failing the IO operation and has to do appropriate roll back. However, rollback activity itself sometimes requires memory allocation.
So, I can?t really afford to end up in no-memory state?I have to know it beforehand that memory in on the verge of depletion and start failing the operations. I thought that knowing the maximum and available paged pool would have rescued me here?..basically, I was expecting (way too rich expectation seems) of the existence of a routine/variable which could give me something similar to what is given by !vm command.

>There are documented interfaces to get notifications from the system that paged or non-paged pool are becoming depleted or have been replenished relative to system-defined high memory or low memory amounts. See ?Standard Event Objects? in the WDK documentation.

Given my nescience to advanced mathematics, I find it plausible to walk on this path. I?ll see to it if it can be instrumental. Many thanks!


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


However, rollback activity itself sometimes requires memory allocation.

This sounds like you have some form of transactional behavior and a
requirement to make progress (at least on the rollback) even in the event of
system resource exhaustion.

Two thing:

  • Your ‘prepare’ phase of your transaction ought to have acquired all the
    resources it needs to complete either a commit or a rollback. By exiting
    ‘prepare’ successfully, you are saying the transaction is ‘prepared’ to do
    its (sub-)part and complete either the ‘commit’ or ‘rollback’. That does
    not sound like the case here.

  • If you require a minimum subset of resources to guarantee forward progress
    in the event of system resource exhaustion, you might consider having those
    pre-allocated in reserve for just such an event. There are obvious and
    not-so-obvious challenges to managing those resources and serializing the
    usage of them in the event of persistent system resource exhaustion.

Good Luck,
Dave Cattley

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@rediffmail.com
Sent: Monday, May 18, 2009 2:57 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] How to determine the maximum and currently-used paged
pool size

Apologies for the massive delay in seeing this!

>Why do you believe that you need to do this?

I?m facing this excessive memory consumption problem in my FSD. For each
opened file, this FSD maintains some amount of metadata in memory.
While catering to a test app that creates exorbitant number of files, it
reaches to a point when it finds exhaustion of paged pool. At this point,
FSD starts failing the IO operation and has to do appropriate roll back.
However, rollback activity itself sometimes requires memory allocation.
So, I can?t really afford to end up in no-memory state?I have to know it
beforehand that memory in on the verge of depletion and start failing the
operations. I thought that knowing the maximum and available paged pool
would have rescued me here?..basically, I was expecting (way too rich
expectation seems) of the existence of a routine/variable which could give
me something similar to what is given by !vm command.

>There are documented interfaces to get notifications from the system that
paged or non-paged pool are becoming depleted or have been replenished
relative to system-defined high memory or low memory amounts. See ?Standard
Event Objects? in the WDK documentation.

Given my nescience to advanced mathematics, I find it plausible to walk on
this path. I?ll see to it if it can be instrumental. Many thanks!


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Pre-allocation can be done to ensure the roll back but still I do not want to end up in a situation when entire pool is exhausted, for it makes system almost dead.
Currently, even when the roll backs have been made, the test application (which creates 1 million+ folders in a tight loop) stops progressing and I get to see bountiful of these messages on debugger - SXS: BasepCreateActCtx() NtMapViewOfSection failed.

As for Standard event objects -
\KernelObjects\HighMemoryCondition, \KernelObjects\LowMemoryCondition
deal with physical memory.

And \KernelObjects\HighPagedPoolCondition, \KernelObjects\LowPagedPoolCondition
,to my dismay, do not exist on windows XP (pro-vista folks have another reason to cheer here)

There actually exists a kernel variable MmSizeOfPagedPoolInBytes which tells the maximum size of paged pool…but nothing really seem to exist for current usage.

I guess you know why current usage should not be provided. If it is a global
variable then the question is how frequently you will update it, and between
the last update and your or mine probe ( million things can happen quite
randomly ). So any such counter is flawed by its nature …

I think what Ken provided is a good technique to craft — pacing technique
internally, so that none of the transaction becomes half baked, and further
transaction can be on hold ( basically rejecting ) or speed up using some
backlogs…

Well, as if I’ve to sign something !
Prokash Sinha
http://prokash.squarespace.com
Success has many fathers, but failure is an orphan.

----- Original Message -----
From:
To: “Windows System Software Devs Interest List”
Sent: Tuesday, May 19, 2009 3:28 AM
Subject: RE:[ntdev] How to determine the maximum and currently-used paged
pool size

> Pre-allocation can be done to ensure the roll back but still I do not want
> to end up in a situation when entire pool is exhausted, for it makes
> system almost dead.
> Currently, even when the roll backs have been made, the test application
> (which creates 1 million+ folders in a tight loop) stops progressing and I
> get to see bountiful of these messages on debugger - SXS:
> BasepCreateActCtx() NtMapViewOfSection failed.
>
> As for Standard event objects -
> \KernelObjects\HighMemoryCondition, \KernelObjects\LowMemoryCondition
> deal with physical memory.
>
> And \KernelObjects\HighPagedPoolCondition,
> \KernelObjects\LowPagedPoolCondition
> ,to my dismay, do not exist on windows XP (pro-vista folks have another
> reason to cheer here)
>
> There actually exists a kernel variable MmSizeOfPagedPoolInBytes which
> tells the maximum size of paged pool…but nothing really seem to exist
> for current usage.
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer