Large Buffer in Device Extension ?

OSR_Community_User

Hello,

In my W2K Bus Driver, I need a buffer of size 128 M for each of the devices
on the bus. Can I make this 128 M buffer a part of the PDO
Device Extension of each of the devices on the Bus ?

Is it ok to have such large buffers in the PDO Device Extension ?
If not, what is the alternative ?

I will be accessing these buffers in a driver-created thread mostly.

Thanks for your help in advance.

Thanks
Puja

OSR_Community_User-35

M as in megabytes???

That sounds like a bad idea. There isn't that much nonpaged pool on
most (if any) systems. In fact, there aren't enough system PTEs to map
that much memory into kernel space on many systems. And there is a
ceiling to how many system PTEs you can configure the OS to allocate.
You will exceed that ceiling with only a few such device extensions.

You could allocate large amounts of user-space memory, lock it down, and
use MDLs to describe it all. But you should refrain from mapping large
amounts of it into kernel space at any given time.

-----------------------------------------------------------------------
Dave Cox
Hewlett-Packard Co.
HPSO/SSMO (Santa Barbara)
https://ecardfile.com/id/Dave+Cox


-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Monday, May 01, 2000 6:01 PM
To: NT Developers Interest List
Subject: [ntdev] Large Buffer in Device Extension ?


Hello,

In my W2K Bus Driver, I need a buffer of size 128 M for each of the devices
on the bus. Can I make this 128 M buffer a part of the PDO
Device Extension of each of the devices on the Bus ?

Is it ok to have such large buffers in the PDO Device Extension ?
If not, what is the alternative ?

I will be accessing these buffers in a driver-created thread mostly.

Thanks for your help in advance.

Thanks
Puja

---
You are currently subscribed to ntdev as: [email protected]
To unsubscribe send a blank email to $subst('Email.Unsub')

OSR_Community_User

Sorry, the Buffer Size I need is 128 K (NOT 128 M). It was a typo.
The question still remains, can I allocate a buffer of this size in the
PDO Device Extension ?

Thanks
Puja


On 05/02/00, "[email protected]" wrote:
> Hello,

In my W2K Bus Driver, I need a buffer of size 128 M for each of the
devices
on the bus. Can I make this 128 M buffer a part of the PDO
Device Extension of each of the devices on the Bus ?

Is it ok to have such large buffers in the PDO Device Extension ?
If not, what is the alternative ?

I will be accessing these buffers in a driver-created thread mostly.

Thanks for your help in advance.

Thanks
Puja

OSR_Community_User

So, are you saying that you will pass a size of 128K+ when creating the
device object?

I suspect this is OK. The device extension is allocated from NPP and as long
as the resources exist, it should work.

I would think a better mechanism would be to store a pointer in the DE and
then allocate your memory and store the pointer in the DE. I guess is is a
matter prefernece more than anything else.

Jamey

> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]]On Behalf Of [email protected]
> Sent: Tuesday, May 02, 2000 5:59 PM
> To: NT Developers Interest List
> Subject: [ntdev] Re: Large Buffer in Device Extension ?
>
>
>
> Sorry, the Buffer Size I need is 128 K (NOT 128 M). It was a typo.
> The question still remains, can I allocate a buffer of this size in the
> PDO Device Extension ?
>
> Thanks
> Puja
>
>
> On 05/02/00, "[email protected]" wrote:
> > Hello,
>
> In my W2K Bus Driver, I need a buffer of size 128 M for each of the
> devices
> on the bus. Can I make this 128 M buffer a part of the PDO
> Device Extension of each of the devices on the Bus ?
>
> Is it ok to have such large buffers in the PDO Device Extension ?
> If not, what is the alternative ?
>
> I will be accessing these buffers in a driver-created thread mostly.
>
> Thanks for your help in advance.
>
> Thanks
> Puja
>
> ---
> You are currently subscribed to ntdev as: [email protected]
> To unsubscribe send a blank email to $subst('Email.Unsub')
>

OSR_Community_User

I agree with Jamey. Storing a pointer to such big buffers is a better idea.
Another (good) reason for keeping the pointer is that in case your driver
cannot allocate this big chunck, it can flag some sort of flag, and perform
at a degraded scale (and may be allocate less memory). But if the DevExt
itself is 128K, and you run out of resources at IoCreateDevice() time, the
call will fail, and you are stuck. It's better to perform at lower standards
than to fail all together.

Shweta.

So, are you saying that you will pass a size of 128K+ when creating the
device object?

I suspect this is OK. The device extension is allocated from NPP and as long
as the resources exist, it should work.

I would think a better mechanism would be to store a pointer in the DE and
then allocate your memory and store the pointer in the DE. I guess is is a
matter prefernece more than anything else.

Jamey

> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]]On Behalf Of [email protected]
> Sent: Tuesday, May 02, 2000 5:59 PM
> To: NT Developers Interest List
> Subject: [ntdev] Re: Large Buffer in Device Extension ?
>
>
>
> Sorry, the Buffer Size I need is 128 K (NOT 128 M). It was a typo.
> The question still remains, can I allocate a buffer of this size in the
> PDO Device Extension ?
>
> Thanks
> Puja
>
>
> On 05/02/00, "[email protected]" wrote:
> > Hello,
>
> In my W2K Bus Driver, I need a buffer of size 128 M for each of the
> devices
> on the bus. Can I make this 128 M buffer a part of the PDO
> Device Extension of each of the devices on the Bus ?
>
> Is it ok to have such large buffers in the PDO Device Extension ?
> If not, what is the alternative ?
>
> I will be accessing these buffers in a driver-created thread mostly.
>
> Thanks for your help in advance.
>
> Thanks
> Puja
>
> ---
> You are currently subscribed to ntdev as: [email protected]
> To unsubscribe send a blank email to $subst('Email.Unsub')
>


---
You are currently subscribed to ntdev as: [email protected]
To unsubscribe send a blank email to $subst('Email.Unsub')

______________________________________________
FREE Personalized Email at Mail.com
Sign up at http://www.mail.com/?sr=signup

OSR_Community_User

Thanks Jamey and Shweta, for your response.

If I wanted to store only a pointer in the Device Extension, I would
allocate memory with ExAllocatePool(), correct ?

Another question:

I will be allocating this memory in DriverEntry(),
accessing this memory in a Driver-created thread,
freeing the memory in Unload()

Since I will be at IRQL PASSIVE_LEVEL in all 3 situations above, I donot
necessarily need Non-Paged Pool memory. Will it give me better performance
if I used NPP memory (instead of allocating it from Paged Pool), though ?

Thanks
Puja


On 05/02/00, "Shweta Dubey " wrote:
> I agree with Jamey. Storing a pointer to such big buffers is a better idea.
> Another (good) reason for keeping the pointer is that in case your driver
> cannot allocate this big chunck, it can flag some sort of flag, and perform
> at a degraded scale (and may be allocate less memory). But if the DevExt
> itself is 128K, and you run out of resources at IoCreateDevice() time, the
> call will fail, and you are stuck. It's better to perform at lower standards
> than to fail all together.
>
> Shweta.
>
> So, are you saying that you will pass a size of 128K+ when creating the
> device object?
>
> I suspect this is OK. The device extension is allocated from NPP and as long
> as the resources exist, it should work.
>
> I would think a better mechanism would be to store a pointer in the DE and
> then allocate your memory and store the pointer in the DE. I guess is is a
> matter prefernece more than anything else.
>
> Jamey
>
> > -----Original Message-----
> > From: [email protected]
> > [mailto:[email protected]]On Behalf Of [email protected]
> > Sent: Tuesday, May 02, 2000 5:59 PM
> > To: NT Developers Interest List
> > Subject: [ntdev] Re: Large Buffer in Device Extension ?
> >
> >
> >
> > Sorry, the Buffer Size I need is 128 K (NOT 128 M). It was a typo.
> > The question still remains, can I allocate a buffer of this size in the
> > PDO Device Extension ?
> >
> > Thanks
> > Puja
> >
> >
> > On 05/02/00, "[email protected]" wrote:
> > > Hello,
> >
> > In my W2K Bus Driver, I need a buffer of size 128 M for each of the
> > devices
> > on the bus. Can I make this 128 M buffer a part of the PDO
> > Device Extension of each of the devices on the Bus ?
> >
> > Is it ok to have such large buffers in the PDO Device Extension ?
> > If not, what is the alternative ?
> >
> > I will be accessing these buffers in a driver-created thread mostly.
> >
> > Thanks for your help in advance.
> >
> > Thanks
> > Puja
> >
> > ---
> > You are currently subscribed to ntdev as: [email protected]
> > To unsubscribe send a blank email to $subst('Email.Unsub')
> >
>
>
> ---
> You are currently subscribed to ntdev as: [email protected]
> To unsubscribe send a blank email to $subst('Email.Unsub')
>
> ______________________________________________
> FREE Personalized Email at Mail.com
> Sign up at http://www.mail.com/?sr=signup

OSR_Community_User-35

You won't incur any page-faults when you access non-paged pool, so it
could perform better than paged-pool. Will you touch the pages frequently
enough to prevent paged-pool from being swapped out?

-----------------------------------------------------------------------
Dave Cox
Hewlett-Packard Co.
HPSO/SSMO (Santa Barbara)
https://ecardfile.com/id/Dave+Cox


-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Tuesday, May 02, 2000 12:27 PM
To: NT Developers Interest List
Subject: [ntdev] Re: Large Buffer in Device Extension ?



Thanks Jamey and Shweta, for your response.

If I wanted to store only a pointer in the Device Extension, I would
allocate memory with ExAllocatePool(), correct ?

Another question:

I will be allocating this memory in DriverEntry(),
accessing this memory in a Driver-created thread,
freeing the memory in Unload()

Since I will be at IRQL PASSIVE_LEVEL in all 3 situations above, I donot
necessarily need Non-Paged Pool memory. Will it give me better performance
if I used NPP memory (instead of allocating it from Paged Pool), though ?

Thanks
Puja


On 05/02/00, "Shweta Dubey <[email protected]>" wrote:
> I agree with Jamey. Storing a pointer to such big buffers is a better
idea.
> Another (good) reason for keeping the pointer is that in case your driver
> cannot allocate this big chunck, it can flag some sort of flag, and
perform
> at a degraded scale (and may be allocate less memory). But if the DevExt
> itself is 128K, and you run out of resources at IoCreateDevice() time, the
> call will fail, and you are stuck. It's better to perform at lower
standards
> than to fail all together.
>
> Shweta.
>
> So, are you saying that you will pass a size of 128K+ when creating the
> device object?
>
> I suspect this is OK. The device extension is allocated from NPP and as
long
> as the resources exist, it should work.
>
> I would think a better mechanism would be to store a pointer in the DE and
> then allocate your memory and store the pointer in the DE. I guess is is a
> matter prefernece more than anything else.
>
> Jamey
>
> > -----Original Message-----
> > From: [email protected]
> > [mailto:[email protected]]On Behalf Of [email protected]
> > Sent: Tuesday, May 02, 2000 5:59 PM
> > To: NT Developers Interest List
> > Subject: [ntdev] Re: Large Buffer in Device Extension ?
> >
> >
> >
> > Sorry, the Buffer Size I need is 128 K (NOT 128 M). It was a typo.
> > The question still remains, can I allocate a buffer of this size in the
> > PDO Device Extension ?
> >
> > Thanks
> > Puja
> >
> >
> > On 05/02/00, "[email protected]" wrote:
> > > Hello,
> >
> > In my W2K Bus Driver, I need a buffer of size 128 M for each of the
> > devices
> > on the bus. Can I make this 128 M buffer a part of the PDO
> > Device Extension of each of the devices on the Bus ?
> >
> > Is it ok to have such large buffers in the PDO Device Extension ?
> > If not, what is the alternative ?
> >
> > I will be accessing these buffers in a driver-created thread mostly.
> >
> > Thanks for your help in advance.
> >
> > Thanks
> > Puja
> >
> > ---
> > You are currently subscribed to ntdev as: [email protected]
> > To unsubscribe send a blank email to $subst('Email.Unsub')
> >
>
>
> ---
> You are currently subscribed to ntdev as: [email protected]
> To unsubscribe send a blank email to $subst('Email.Unsub')
>
> ______________________________________________
> FREE Personalized Email at Mail.com
> Sign up at http://www.mail.com/?sr=signup

---
You are currently subscribed to ntdev as: [email protected]
To unsubscribe send a blank email to $subst('Email.Unsub')

OSR_Community_User

It depends on how how often you access the buffers. I would start with
making it PP and if there are problems, switch to NPP. NPP is a precious
resource, so unless you absolutly need it to be non-paged, I would use PP.

Jamey

> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]]On Behalf Of [email protected]
> Sent: Tuesday, May 02, 2000 7:27 PM
> To: NT Developers Interest List
> Subject: [ntdev] Re: Large Buffer in Device Extension ?
>
>
>
> Thanks Jamey and Shweta, for your response.
>
> If I wanted to store only a pointer in the Device Extension, I would
> allocate memory with ExAllocatePool(), correct ?
>
> Another question:
>
> I will be allocating this memory in DriverEntry(),
> accessing this memory in a Driver-created thread,
> freeing the memory in Unload()
>
> Since I will be at IRQL PASSIVE_LEVEL in all 3 situations above, I donot
> necessarily need Non-Paged Pool memory. Will it give me better
> performance
> if I used NPP memory (instead of allocating it from Paged Pool), though ?
>
> Thanks
> Puja
>
>
> On 05/02/00, "Shweta Dubey <[email protected]>" wrote:
> > I agree with Jamey. Storing a pointer to such big buffers is a
> better idea.
> > Another (good) reason for keeping the pointer is that in case
> your driver
> > cannot allocate this big chunck, it can flag some sort of flag,
> and perform
> > at a degraded scale (and may be allocate less memory). But if the DevExt
> > itself is 128K, and you run out of resources at
> IoCreateDevice() time, the
> > call will fail, and you are stuck. It's better to perform at
> lower standards
> > than to fail all together.
> >
> > Shweta.
> >
> > So, are you saying that you will pass a size of 128K+ when creating the
> > device object?
> >
> > I suspect this is OK. The device extension is allocated from
> NPP and as long
> > as the resources exist, it should work.
> >
> > I would think a better mechanism would be to store a pointer in
> the DE and
> > then allocate your memory and store the pointer in the DE. I
> guess is is a
> > matter prefernece more than anything else.
> >
> > Jamey
> >
> > > -----Original Message-----
> > > From: [email protected]
> > > [mailto:[email protected]]On Behalf Of [email protected]
> > > Sent: Tuesday, May 02, 2000 5:59 PM
> > > To: NT Developers Interest List
> > > Subject: [ntdev] Re: Large Buffer in Device Extension ?
> > >
> > >
> > >
> > > Sorry, the Buffer Size I need is 128 K (NOT 128 M). It was a typo.
> > > The question still remains, can I allocate a buffer of this
> size in the
> > > PDO Device Extension ?
> > >
> > > Thanks
> > > Puja
> > >
> > >
> > > On 05/02/00, "[email protected]" wrote:
> > > > Hello,
> > >
> > > In my W2K Bus Driver, I need a buffer of size 128 M for each of the
> > > devices
> > > on the bus. Can I make this 128 M buffer a part of the PDO
> > > Device Extension of each of the devices on the Bus ?
> > >
> > > Is it ok to have such large buffers in the PDO Device Extension ?
> > > If not, what is the alternative ?
> > >
> > > I will be accessing these buffers in a driver-created thread mostly.
> > >
> > > Thanks for your help in advance.
> > >
> > > Thanks
> > > Puja
> > >
> > > ---
> > > You are currently subscribed to ntdev as: [email protected]
> > > To unsubscribe send a blank email to $subst('Email.Unsub')
> > >
> >
> >
> > ---
> > You are currently subscribed to ntdev as: [email protected]
> > To unsubscribe send a blank email to $subst('Email.Unsub')
> >
> > ______________________________________________
> > FREE Personalized Email at Mail.com
> > Sign up at http://www.mail.com/?sr=signup
>
> ---
> You are currently subscribed to ntdev as: [email protected]
> To unsubscribe send a blank email to $subst('Email.Unsub')
>

OSR_Community_User

Yes, I would be running in a constant loop doing memcmp() on this buffer,
to see if there's been any changes (from previous state). So I guess the
pages shouldn't be paged-out anyways and I could get away with using Paged
Pool.

Another question: since I'll be using Paged Pool memory, do I still need
to
allocate this during my DriverEntry() (and free in Unload()) or could I do
so as I create/delete the PDOs for the child devices on the Bus ?

Thanks so much for your help!

Puja


On 05/02/00, ""COX,DAVID (HP-Roseville,ex1)" " wrote:
> You won't incur any page-faults when you access non-paged pool, so it
> could perform better than paged-pool. Will you touch the pages frequently
> enough to prevent paged-pool from being swapped out?
>
> -----------------------------------------------------------------------
> Dave Cox
> Hewlett-Packard Co.
> HPSO/SSMO (Santa Barbara)
> https://ecardfile.com/id/Dave+Cox
>
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]]
> Sent: Tuesday, May 02, 2000 12:27 PM
> To: NT Developers Interest List
> Subject: [ntdev] Re: Large Buffer in Device Extension ?
>
>
>
> Thanks Jamey and Shweta, for your response.
>
> If I wanted to store only a pointer in the Device Extension, I would
> allocate memory with ExAllocatePool(), correct ?
>
> Another question:
>
> I will be allocating this memory in DriverEntry(),
> accessing this memory in a Driver-created thread,
> freeing the memory in Unload()
>
> Since I will be at IRQL PASSIVE_LEVEL in all 3 situations above, I donot
> necessarily need Non-Paged Pool memory. Will it give me better performance
> if I used NPP memory (instead of allocating it from Paged Pool), though ?
>
> Thanks
> Puja

OSR_Community_User

> could I do so as I create/delete the PDOs for
> the child devices on the Bus ?

Sure.

> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]]On Behalf Of [email protected]
> Sent: Tuesday, May 02, 2000 8:01 PM
> To: NT Developers Interest List
> Subject: [ntdev] Re: Large Buffer in Device Extension ?
>
>
>
> Yes, I would be running in a constant loop doing memcmp() on this buffer,
> to see if there's been any changes (from previous state). So I guess the
> pages shouldn't be paged-out anyways and I could get away with
> using Paged
> Pool.
>
> Another question: since I'll be using Paged Pool memory, do I still need
> to
> allocate this during my DriverEntry() (and free in Unload()) or
> could I do
> so as I create/delete the PDOs for the child devices on the Bus ?
>
> Thanks so much for your help!
>
> Puja
>
>
> On 05/02/00, ""COX,DAVID (HP-Roseville,ex1)" <[email protected]>" wrote:
> > You won't incur any page-faults when you access non-paged pool, so it
> > could perform better than paged-pool. Will you touch the pages
> frequently
> > enough to prevent paged-pool from being swapped out?
> >
> > -----------------------------------------------------------------------
> > Dave Cox
> > Hewlett-Packard Co.
> > HPSO/SSMO (Santa Barbara)
> > https://ecardfile.com/id/Dave+Cox
> >
> >
> > -----Original Message-----
> > From: [email protected] [mailto:[email protected]]
> > Sent: Tuesday, May 02, 2000 12:27 PM
> > To: NT Developers Interest List
> > Subject: [ntdev] Re: Large Buffer in Device Extension ?
> >
> >
> >
> > Thanks Jamey and Shweta, for your response.
> >
> > If I wanted to store only a pointer in the Device Extension, I would
> > allocate memory with ExAllocatePool(), correct ?
> >
> > Another question:
> >
> > I will be allocating this memory in DriverEntry(),
> > accessing this memory in a Driver-created thread,
> > freeing the memory in Unload()
> >
> > Since I will be at IRQL PASSIVE_LEVEL in all 3 situations above, I donot
> > necessarily need Non-Paged Pool memory. Will it give me better
> performance
> > if I used NPP memory (instead of allocating it from Paged
> Pool), though ?
> >
> > Thanks
> > Puja
>
> ---
> You are currently subscribed to ntdev as: [email protected]
> To unsubscribe send a blank email to $subst('Email.Unsub')
>

OSR_Community_User-35

There is no restriction that you must allocate nonpaged pool only in
DriverEntry() (and free only in Unload()). It may be a rule of thumb
for some people -- if you wait to allocate, there may be less NPP,
and/or it may be more fragmented.

-----------------------------------------------------------------------
Dave Cox
Hewlett-Packard Co.
HPSO/SSMO (Santa Barbara)
https://ecardfile.com/id/Dave+Cox


-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Monday, May 01, 2000 7:01 PM
To: NT Developers Interest List
Subject: [ntdev] Re: Large Buffer in Device Extension ?



Yes, I would be running in a constant loop doing memcmp() on this buffer,
to see if there's been any changes (from previous state). So I guess the
pages shouldn't be paged-out anyways and I could get away with using Paged
Pool.

Another question: since I'll be using Paged Pool memory, do I still need
to
allocate this during my DriverEntry() (and free in Unload()) or could I do
so as I create/delete the PDOs for the child devices on the Bus ?

Thanks so much for your help!

Puja


On 05/02/00, ""COX,DAVID (HP-Roseville,ex1)" <[email protected]>" wrote:
> You won't incur any page-faults when you access non-paged pool, so it
> could perform better than paged-pool. Will you touch the pages frequently
> enough to prevent paged-pool from being swapped out?
>
> -----------------------------------------------------------------------
> Dave Cox
> Hewlett-Packard Co.
> HPSO/SSMO (Santa Barbara)
> https://ecardfile.com/id/Dave+Cox
>
>
> -----Original Message-----
> From: [email protected] [mailto:[email protected]]
> Sent: Tuesday, May 02, 2000 12:27 PM
> To: NT Developers Interest List
> Subject: [ntdev] Re: Large Buffer in Device Extension ?
>
>
>
> Thanks Jamey and Shweta, for your response.
>
> If I wanted to store only a pointer in the Device Extension, I would
> allocate memory with ExAllocatePool(), correct ?
>
> Another question:
>
> I will be allocating this memory in DriverEntry(),
> accessing this memory in a Driver-created thread,
> freeing the memory in Unload()
>
> Since I will be at IRQL PASSIVE_LEVEL in all 3 situations above, I donot
> necessarily need Non-Paged Pool memory. Will it give me better
performance
> if I used NPP memory (instead of allocating it from Paged Pool), though ?
>
> Thanks
> Puja

---
You are currently subscribed to ntdev as: [email protected]
To unsubscribe send a blank email to $subst('Email.Unsub')

OSR_Community_User

> Since I will be at IRQL PASSIVE_LEVEL in all 3 situations above, I donot
> necessarily need Non-Paged Pool memory. Will it give me better
performance
> if I used NPP memory (instead of allocating it from Paged Pool), though ?

Do not do this. Nonpaged memory is intended for access from high IRQL
only. It is a waste of resources to use it if it will always be accessed
from
PASSIVE_LEVEL.

Max