Allocating paged or non-paged memory

r2d2 · August 22, 2006, 4:32pm

Hello,

I am working on a File System filter driver and I am some what confused about what type of memory I should allocate with in my driver. Here’s what I am trying to do:

My filter driver receives a configuration from an external application saying attach to all USB devices (or volumes). This message is received in the MessageNotifyCallback() function. The filter driver then needs to enumerate all volumes and then check if they are of type USB, if yes then it will attach to them.
The question should I allocate page memory or non-paged memory when calling
FltEnumerateVolumes() function?

The question is actually applicable to any memory that I might have to allocate in the driver.

Thanks,
Rajesh

Stay in the know. Pulse on the new Yahoo.com. Check it out.

Don_Burn_1 · August 22, 2006, 4:51pm

For the specific question of FltEnumerateVolumes you can probably use
PagedPool but in general you have to allocate memory based on it usage,
there is no magic rule that acts for all cases. I would strongly recomend
you put aside developing a file system driver until you have studied a
decent book on Windows Drivers in general, since this question is that of an
extreme novice, and a FSF is not where to start.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
http://www.windrvr.com
Remove StopSpam from the email to reply

“Rajesh Upadhyay” wrote in message
news:xxxxx@ntfsd…
> Hello,
>
> I am working on a File System filter driver and I am some what confused
> about what type of memory I should allocate with in my driver. Here’s what
> I am trying to do:
>
> My filter driver receives a configuration from an external application
> saying attach to all USB devices (or volumes). This message is received in
> the MessageNotifyCallback() function. The filter driver then needs to
> enumerate all volumes and then check if they are of type USB, if yes then
> it will attach to them.
> The question should I allocate page memory or non-paged memory when
> calling
> FltEnumerateVolumes() function?
>
> The question is actually applicable to any memory that I might have to
> allocate in the driver.
>
> Thanks,
> Rajesh
>
>
>
>
>
>
> ---------------------------------
> Stay in the know. Pulse on the new Yahoo.com. Check it out.

r2d2 · August 22, 2006, 7:41pm

Thanks for your information. I am actually reading 'Windows NT: File Systems Internal" by Rajeev Nagar , however, there are times when the information in this book is too overwhelming for me. Is there a book specifically on filter driver development?

OSR_Community_User · August 23, 2006, 1:57am

> Thanks for your information. I am actually reading 'Windows NT: File

Systems Internal"
by Rajeev Nagar , however, there are times when the information in this
book is too
overwhelming for me. Is there a book specifically on filter driver
development?

“Windows NT: File Systems Internal” *is* specifically
on filter driver development.

L.

OSR_Community_User · August 23, 2006, 3:05am

If you think that writing filters is simpler than writing file systems,
you are mistaken.

File systems have a complex upper-edge interface but generally a simple
lower-edge interface (e.g., the block interface used by physical file
systems or some protocol interface/network interface used by network
file systems.)

Filter drivers have a complex upper-edge interface (same as the file
system) and a complex lower edge interface (same as the file system.)
If you just sit quietly and do nothing (“passive filter”) then you’ll be
pretty safe because it does not require much understanding. If you
decide to change the flow of operations (“active filter”) you have to
understand this interface profoundly well, lest you break it.

While I’m sure some will disagree with me, I’d suggest writing a file
system first and THEN writing a filter driver - by then you’ll have
sufficient understanding of the interface that you might be able to
write a correct active filter.

Unfortunately, the usual approach that people take to this business is
to undertake a filter because they think it is simpler than the real
thing (and with passive examples easily found, this myth is perpetuated)
when in fact that’s actually not the real case.

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@yahoo.com
Sent: Wednesday, 23 August 2006 9:43 AM
To: ntfsd redirect
Subject: RE:[ntfsd] Allocating paged or non-paged memory

Thanks for your information. I am actually reading 'Windows NT: File
Systems Internal" by Rajeev Nagar , however, there are times when the
information in this book is too overwhelming for me. Is there a book
specifically on filter driver development?

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@osr.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

OSR_Community_User · August 23, 2006, 4:39am

That’s pretty much it (outside of the DDK/WDK). It also, by design,
does not contain everything you need to know to write a filesystem.

FOR WHAT IT IS WORTH:

Based on the level of inexperience that your original question pretty
much guarantees, what you are attempting to do is pretty much a
guaranteed disaster, assuming that this is a production driver that must
be completed with even a remotely reasonable time frame. If you are
doing just for development or personal use, I in no way mean to
discourage you. However, if this is your job, my advice would that, no
matter what the consequences of more or less refusing to do this are, do
so, because this is a death sentence.

MM

>> xxxxx@yahoo.com 2006-08-22 19:42 >>>
Thanks for your information. I am actually reading 'Windows NT: File
Systems Internal" by Rajeev Nagar , however, there are times when the
information in this book is too overwhelming for me. Is there a book
specifically on filter driver development?

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@evitechnology.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

r2d2 · August 23, 2006, 12:21pm

You guys are discouraging me I agree that writing a windows driver is a tough job, but I have to start some where. I wouldn’t be working on driver today if I hadn’t written the “hello world” program at some point in my career. What do you guys say?

Don_Burn_1 · August 23, 2006, 12:26pm

wrote in message news:xxxxx@ntfsd…
> You guys are discouraging me I agree that writing a windows driver is
> a tough job, but I have to start some where. I wouldn’t be working on
> driver today if I hadn’t written the “hello world” program at some point
> in my career. What do you guys say?
>

Agreed, but this is obviously your first driver, and bottom line is instead
of choosing “hello world” you choose to do something extremely large and
complex. I don’t think you would be here if your first attempt at
programming project was a complete SQL database.

If you are going to start kernel programming, get KMDF and create a really
simple driver (OSR had a software only sample that is a good starting
point). Experiment with kernel environment, take a course or two if
possible. Right now you are jumping in the deep end of the pool, without
knowing how to swim.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
http://www.windrvr.com
Remove StopSpam from the email to reply

OSR_Community_User · August 23, 2006, 12:33pm

I agree totally, unless your job depends on this file system driver
materializing in a workable form, in a reasonable time frame. If it
does, my advice earlier was to bail out now, because this would seem to
be a very unlikely achievable goal by anyone with your experience level.
If nothing rides on it, I’m all for experimentation, as it is the only
way to learn.

MM

>> xxxxx@yahoo.com 2006-08-23 12:21 >>>
You guys are discouraging me I agree that writing a windows driver
is a tough job, but I have to start some where. I wouldn’t be working on
driver today if I hadn’t written the “hello world” program at some point
in my career. What do you guys say?

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@evitechnology.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

r2d2 · August 23, 2006, 1:19pm

I have actually written a driver that performs encryption/decryption/hashing etc on a user supplied
buffer and returning it back to the user. The driver worked fine on XP and on 98 till date. However, it
was a very simple driver, it used BUFFERED_IO mechanism so the user buffer was part of the IRP
and any data that the driver used to allocate was in non-paged memory. But this time I am writing a
FSD which is different ball game so I thought may be I should clarify certain fundamentals and was
looking for help on this mailing list.
By mentioning that I have written a driver before, I am not trying to indicate that I am a guru in this field and not a novice, but my question was to look for any general guideline for allocating memory.
I am reading books and I’ll be going for some training sessions but at the same time I have to continue on this project as well, bailing out is not an option, unfortunately :-(.

David_Beaver · August 23, 2006, 2:04pm

I’m one of those who would disagree with you about writing a file system
first, Tony - but only mildly. I think with the mini-filter model that’s
available these days, it’s pretty easy to write a simple filter and then
scale up to a more complex one. That’s difficult to do as easily with a file
system. But my definition of “pretty easy” may be somewhat at variance with
that definition by most developers, especially those that don’t have some
kernel development behind them… And it’s as complex as a filesystem when
you get into a really complicated filter. And by the time you’re finished
writing the complex filter, you’ll have spent as much time learning the way
file systems work as you spend learning how filters work - because they
really can’t be dis-entangled.

Nevertheless, for the OP, there a set of resources that s/he really needs to
look at before asking questions like this:

Get, read, and understand “Microsoft Windows Internals” by Mark
Russinovich and David Solomon. A very, very good book, with lots of
information you need to know if you’re going to be able to actually develop
well in Windows kernel mode. Get the forth edition unless you’re really into
history. ISBN: 0735619174.
There are a number of other good books out there about kernel/driver
development, all of which will address the original question (about paged vs
nonpaged). I like Tony’s book which is now available on OSRonline (and a bit
outdated), but there are others.
Don’t underestimate the amount of learning available through MSDN online.
It can be a bit of a challenge to
Get a course on kernel development with OSR or one of the other folks
doing kernel training out there. I haven’t heard of a bad one yet. This
tends to be a high dollar solution, but it’s the fastest way to getting the
foundations laid.

…dave

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tony Mason
Sent: Wednesday, August 23, 2006 12:05 AM
To: Windows File Systems Devs Interest List
Subject: RE: [ntfsd] Allocating paged or non-paged memory

If you think that writing filters is simpler than writing file systems, you
are mistaken.

File systems have a complex upper-edge interface but generally a simple
lower-edge interface (e.g., the block interface used by physical file
systems or some protocol interface/network interface used by network file
systems.)

Filter drivers have a complex upper-edge interface (same as the file
system) and a complex lower edge interface (same as the file system.) If you
just sit quietly and do nothing (“passive filter”) then you’ll be pretty
safe because it does not require much understanding. If you decide to
change the flow of operations (“active filter”) you have to understand this
interface profoundly well, lest you break it.

While I’m sure some will disagree with me, I’d suggest writing a file system
first and THEN writing a filter driver - by then you’ll have sufficient
understanding of the interface that you might be able to write a correct
active filter.

Unfortunately, the usual approach that people take to this business is to
undertake a filter because they think it is simpler than the real thing (and
with passive examples easily found, this myth is perpetuated) when in fact
that’s actually not the real case.

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@yahoo.com
Sent: Wednesday, 23 August 2006 9:43 AM
To: ntfsd redirect
Subject: RE:[ntfsd] Allocating paged or non-paged memory

Thanks for your information. I am actually reading 'Windows NT: File Systems
Internal" by Rajeev Nagar , however, there are times when the information in
this book is too overwhelming for me. Is there a book specifically on filter
driver development?

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: xxxxx@osr.com To unsubscribe send
a blank email to xxxxx@lists.osr.com

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

OSR_Community_User · August 23, 2006, 2:20pm

>time I am writing a

FSD which is different ball game so I thought may be I should clarify certain

What are the details? Do you need to support memory-mapped files there? Do you
need to support Windows caching of the files there?

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

Calvin_Guan · August 24, 2006, 1:48am

> -----Original Message-----

From: xxxxx@lists.osr.com [mailto:bounce-259876-
xxxxx@lists.osr.com] On Behalf Of Martin O’Brien
Sent: Wednesday, August 23, 2006 1:38 AM

discourage you. However, if this is your job, my advice would that,
no
matter what the consequences of more or less refusing to do this are,
do
so, because this is a death sentence.

Interesting. It wouldn’t be that bad. Nothing is better than being
forced and *paid* to learn new stuff (that is useful in a person’s
future career) in a short period of time. Even the project leads to a
deadend, the OP would have learned enough to move forward to the next
step in his career. Sounds like a good deal to me…

Calvin Guan (DDK MVP)
Sr. Staff Engineer
NetXtreme NTX Miniport
Broadcom Corporation
Connecting Everything(r)

OSR_Community_User · August 24, 2006, 4:36am

>Interesting. It wouldn’t be that bad. Nothing is better than being

forced and *paid* to learn new stuff

If the developer is really smart and is paid on per-month basis - then surely.
The only issue the employer will have is slipped deadline, which is sometimes
tolerable, since the employer will see that nearly nobody will do the same
faster.

The employers usually also know the life and understand that, if they have
really seriously tight deadline - then only people with this particular
experience can help. Searching for such people to hire them is also timely,
and is also bad for deadline - though HR-side, not development-side.

But, if the developer is paid per-project - then this is a major money waste
for the developer. He is paid on underestimated time basis, not on time really
spent.

If the developer is not sure about his own smartness - then this can be a
disaster, regardless of the payment method. He can end with a code full of bugs
introduced not due to typos/attention issues (each developer does this), but
due to platform mis-understanding. Such bugs can lead to 100% rewrite.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

Don_Burn_1 · August 24, 2006, 8:21am

“Calvin (Hao) Guan” wrote:

> Interesting. It wouldn’t be that bad. Nothing is better than being
> forced and paid to learn new stuff (that is useful in a person’s
> future career) in a short period of time. Even the project leads to a
> deadend, the OP would have learned enough to move forward to the next
> step in his career. Sounds like a good deal to me…

Of course I know of a few projects that prove the opposite:

1. To help the developer, they brought in a “expert” who said it could be
done in the time frame that was clearly impossible. The “expert” taught the
dev every bad kernel programming hack known to man. Unfortunately the dev
learned the methods of the expert. The expert moved far enough away that
his reputation did not catch up with him, the dev needed to stay local, and
is now doing any lousy programming job he can get because his reputation is
ruined.

2. At a former company I worked for a number of us were approached to do a
development job on an early PC, with the promise of “you will learn all the
new technology”. Doing a back of the envelope calculation we realized the
VP in charge had totally overloaded the computer and the project would never
work. The poor fool who accepted did learn new technology, and also learned
what happens to an employee who is blamed for a multi-million dollar fiasco
by a well known VP. The last I heard he was up to selling used cars, his
reputation in the computer industry destroyed.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
http://www.windrvr.com
Remove StopSpam from the email to reply

OSR_Community_User · August 25, 2006, 5:58pm

CALVIN:

Under your scenario, which is a totally realistic, common one, I would
call it ideal. However, under one along the lines of one’s boss either
has no idea of what the probability of what he is asking is for is, or,
the considerably worse one of he does, but he either does not care or
has no choice because some above him either does not care or have no
choice, it might not go to well for the dev, if anyone in that chain of
command above him of undefined length, still does not know are care
about the same issues if something should go wrong during what sounds
like a very short time frame. I wasn’t telling him that he shouldn’t do
it, or that I wouldn’t necessarily do it, but rather that the technical
issues here should not be the main concern, but would more safely be
addressed after he or she (I don’t recall) considers the what he or she
considers the worst case scenario to be, and the how he or she would
feel about the ramifications.

>> xxxxx@broadcom.com 2006-08-24 01:47 >>>
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:bounce-259876-
xxxxx@lists.osr.com] On Behalf Of Martin O’Brien
Sent: Wednesday, August 23, 2006 1:38 AM

discourage you. However, if this is your job, my advice would that,
no
matter what the consequences of more or less refusing to do this
are,
do
so, because this is a death sentence.

Interesting. It wouldn’t be that bad. Nothing is better than being
forced and *paid* to learn new stuff (that is useful in a person’s
future career) in a short period of time. Even the project leads to a
deadend, the OP would have learned enough to move forward to the next
step in his career. Sounds like a good deal to me…

Calvin Guan (DDK MVP)
Sr. Staff Engineer
NetXtreme NTX Miniport
Broadcom Corporation
Connecting Everything(r)

Questions? First check the IFS FAQ at
https://www.osronline.com/article.cfm?id=17

You are currently subscribed to ntfsd as: unknown lmsubst tag argument:
‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com