driver writing back to user space

NTDEV
1

Hi from a new member / new to device drivers.

I’m implementing a driver that writes real time data back to my application
buffers as fast as possible. It’s working okay, using necessary probe and
lock to fix the buffers in RAM and to write to them.

I’m worried if it is / can be made safe against the user process terminating
abnormally - if it goes away before it can stop the driver I’m going to get
a bug check DRIVER_LEFT_LOCKED_PAGES_IN_PROCESS. Fine if everything runs
properly and nothing shuts down my user mode app before I can switch off the
driver, but is there any way of guaranteeing this?

There are a couple of reasons including performance issues why I don’t want
to move the buffers wholly into the kernel and rely on signalling events for
the app to read the data - but I suppose it would be a plan B if all else
fails.

If an experienced driver writer can tell me to stop and go to plan B
immediately I’d appreciate the tip!

Thanks in advance

Mike

2

How are you getting the pointers into memory in the first place? If you are
using a bunch of IRP’s to pass in buffers with something like
METHOD_OUT_DIRECT you will recieve a cancel for the IRP before the process
fully cleans up, and you can cleanup your usage of the buffers.

Note: this is one of the reasons for avoiding tricks like events, since the
handling of process termination is much harder.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
http://www.windrvr.com
Remove StopSpam from the email to reply

“Mike Kemp” wrote in message news:xxxxx@ntdev…
> Hi from a new member / new to device drivers.
>
> I’m implementing a driver that writes real time data back to my
> application buffers as fast as possible. It’s working okay, using
> necessary probe and lock to fix the buffers in RAM and to write to them.
>
> I’m worried if it is / can be made safe against the user process
> terminating abnormally - if it goes away before it can stop the driver I’m
> going to get a bug check DRIVER_LEFT_LOCKED_PAGES_IN_PROCESS. Fine if
> everything runs properly and nothing shuts down my user mode app before I
> can switch off the driver, but is there any way of guaranteeing this?
>
> There are a couple of reasons including performance issues why I don’t
> want to move the buffers wholly into the kernel and rely on signalling
> events for the app to read the data - but I suppose it would be a plan B
> if all else fails.
>
> If an experienced driver writer can tell me to stop and go to plan B
> immediately I’d appreciate the tip!
>
> Thanks in advance
>
> Mike
>

3

In your IRP_MJ_CLEANUP dispatch handler, cleanup the locked pages. By
doing the unload in the cleanup handler, you guard against the app going
away w/out properly behaving. You can’t do this in IRP_MJ_CLOSE b/c
there is no guarantee that CLOSE will occur in the context of the
application, IRP_MJ_CLEANUP does have that guarantee. Furthermore, by
doing the unlock in CLEANUP, the app doesn’t have to manually notify the
driver of cleanup since it will always be cleaned up when the handle is
closed.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mike Kemp
Sent: Tuesday, May 30, 2006 8:49 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] driver writing back to user space

Hi from a new member / new to device drivers.

I’m implementing a driver that writes real time data back to my
application
buffers as fast as possible. It’s working okay, using necessary probe
and
lock to fix the buffers in RAM and to write to them.

I’m worried if it is / can be made safe against the user process
terminating
abnormally - if it goes away before it can stop the driver I’m going to
get
a bug check DRIVER_LEFT_LOCKED_PAGES_IN_PROCESS. Fine if everything runs

properly and nothing shuts down my user mode app before I can switch off
the
driver, but is there any way of guaranteeing this?

There are a couple of reasons including performance issues why I don’t
want
to move the buffers wholly into the kernel and rely on signalling events
for
the app to read the data - but I suppose it would be a plan B if all
else
fails.

If an experienced driver writer can tell me to stop and go to plan B
immediately I’d appreciate the tip!

Thanks in advance

Mike

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

4

Mike Kemp wrote:

I’m implementing a driver that writes real time data back to my
application buffers as fast as possible. It’s working okay, using
necessary probe and lock to fix the buffers in RAM and to write to them.

I’m worried if it is / can be made safe against the user process
terminating abnormally - if it goes away before it can stop the driver
I’m going to get a bug check DRIVER_LEFT_LOCKED_PAGES_IN_PROCESS. Fine
if everything runs properly and nothing shuts down my user mode app
before I can switch off the driver, but is there any way of
guaranteeing this?

Your driver will get an IRP_MJ_CLEANUP when the process terminates just
before the driver file handle closes. As long as you unlock the pages
in IRP_MJ_CLEANUP, everybody will be happy.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

5

Hi Don

I simply pass (an array of) user mode pointers in using METHOD_BUFFERED.

Once a (user mode) pointer is in my driver I allocate an MDL to describe it,
do a

MmProbeAndLockPages(pMdl, KernelMode, IoModifyAccess);

to fix it in memory, obtain a kernel mode pointer with

MmGetSystemAddressForMdlSafe(pMdl, HighPagePriority);

and store all these pointers in the device extension.

Then real time interrupt callbacks write data to the buffer(s) pointed to,
ad infinitum (or until I tell the driver to stop).

I’m still getting my head around IRPs but I’d assume that all are complete
at the time this is running, as all my calls are non-overlapped, simply
passing in control values, pointers etc.

I assume that when the user process terminates for any reason, XP notifies
the driver that it’s handle is (being) closed. If I can unlock the memory in
time I’d be okay, but I’m not sure how to do that at the moment…

Thanks for any comments

Mike

----- Original Message -----
From: Don Burn
Newsgroups: ntdev
To: Windows System Software Devs Interest List
Sent: Tuesday, May 30, 2006 4:55 PM
Subject: Re:[ntdev] driver writing back to user space

How are you getting the pointers into memory in the first place? If you are
using a bunch of IRP’s to pass in buffers with something like
METHOD_OUT_DIRECT you will recieve a cancel for the IRP before the process
fully cleans up, and you can cleanup your usage of the buffers.

Note: this is one of the reasons for avoiding tricks like events, since the
handling of process termination is much harder.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
http://www.windrvr.com

“Mike Kemp” wrote in message news:xxxxx@ntdev…
> Hi from a new member / new to device drivers.
>
> I’m implementing a driver that writes real time data back to my
> application buffers as fast as possible. It’s working okay, using
> necessary probe and lock to fix the buffers in RAM and to write to them.
>
> I’m worried if it is / can be made safe against the user process
> terminating abnormally - if it goes away before it can stop the driver I’m
> going to get a bug check DRIVER_LEFT_LOCKED_PAGES_IN_PROCESS. Fine if
> everything runs properly and nothing shuts down my user mode app before I
> can switch off the driver, but is there any way of guaranteeing this?
>
> There are a couple of reasons including performance issues why I don’t
> want to move the buffers wholly into the kernel and rely on signalling
> events for the app to read the data - but I suppose it would be a plan B
> if all else fails.
>
> If an experienced driver writer can tell me to stop and go to plan B
> immediately I’d appreciate the tip!
>
> Thanks in advance
>
> Mike
>

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

6

Thanks - this sounds like it’ll let me do what I want and be a reliable
get-out-of-jail-free when (if) my app crashes…

Best regards - Mike

From: Doron Holan
To: Windows System Software Devs Interest List
Sent: Tuesday, May 30, 2006 4:58 PM
Subject: RE: [ntdev] driver writing back to user space

In your IRP_MJ_CLEANUP dispatch handler, cleanup the locked pages. By
doing the unload in the cleanup handler, you guard against the app going
away w/out properly behaving. You can’t do this in IRP_MJ_CLOSE b/c
there is no guarantee that CLOSE will occur in the context of the
application, IRP_MJ_CLEANUP does have that guarantee. Furthermore, by
doing the unlock in CLEANUP, the app doesn’t have to manually notify the
driver of cleanup since it will always be cleaned up when the handle is
closed.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mike Kemp
Sent: Tuesday, May 30, 2006 8:49 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] driver writing back to user space

Hi from a new member / new to device drivers.

I’m implementing a driver that writes real time data back to my
application
buffers as fast as possible. It’s working okay, using necessary probe
and
lock to fix the buffers in RAM and to write to them.

I’m worried if it is / can be made safe against the user process
terminating
abnormally - if it goes away before it can stop the driver I’m going to
get
a bug check DRIVER_LEFT_LOCKED_PAGES_IN_PROCESS. Fine if everything runs

properly and nothing shuts down my user mode app before I can switch off
the
driver, but is there any way of guaranteeing this?

There are a couple of reasons including performance issues why I don’t
want
to move the buffers wholly into the kernel and rely on signalling events
for
the app to read the data - but I suppose it would be a plan B if all
else
fails.

If an experienced driver writer can tell me to stop and go to plan B
immediately I’d appreciate the tip!

Thanks in advance

Mike

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

7

Mike,

As Doron and Tim pointed out you will recieve an IRP_MJ_CLEANUP when
the process is closing the handle (even a abnormal terminate) to the device.
If you do not mind using OVERLAPPED I/O seriously consider replacing your
model with a bunch of IOCTL’s, one for each buffer that are
METHOD_OUT_DIRECT. This will actually simplify your driver, since:

  1. The I/O subsystem will give you a MDL for the buffer with the
    data locked down
  2. The completion of the IRP provides an easy way to notify the
    user the buffer is ready to be accessed.
  3. The cancel routine of the IRP will handle the cleanup of the
    buffer

Also, KMDF which will simplify the heck out of your driver, is based
around handling IRP’s so much of the pain is taken care of for you.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
http://www.windrvr.com
Remove StopSpam from the email to reply

“Mike Kemp” wrote in message news:xxxxx@ntdev…
> Hi Don
>
> I simply pass (an array of) user mode pointers in using METHOD_BUFFERED.
>
> Once a (user mode) pointer is in my driver I allocate an MDL to describe
> it, do a
>
> MmProbeAndLockPages(pMdl, KernelMode, IoModifyAccess);
>
> to fix it in memory, obtain a kernel mode pointer with
>
> MmGetSystemAddressForMdlSafe(pMdl, HighPagePriority);
>
> and store all these pointers in the device extension.
>
> Then real time interrupt callbacks write data to the buffer(s) pointed to,
> ad infinitum (or until I tell the driver to stop).
>
> I’m still getting my head around IRPs but I’d assume that all are complete
> at the time this is running, as all my calls are non-overlapped, simply
> passing in control values, pointers etc.
>
> I assume that when the user process terminates for any reason, XP notifies
> the driver that it’s handle is (being) closed. If I can unlock the memory
> in time I’d be okay, but I’m not sure how to do that at the moment…
>
> Thanks for any comments
>
> Mike
>
>
> ----- Original Message -----
> From: Don Burn
> Newsgroups: ntdev
> To: Windows System Software Devs Interest List
> Sent: Tuesday, May 30, 2006 4:55 PM
> Subject: Re:[ntdev] driver writing back to user space
>
>
> How are you getting the pointers into memory in the first place? If you
> are
> using a bunch of IRP’s to pass in buffers with something like
> METHOD_OUT_DIRECT you will recieve a cancel for the IRP before the process
> fully cleans up, and you can cleanup your usage of the buffers.
>
> Note: this is one of the reasons for avoiding tricks like events, since
> the
> handling of process termination is much harder.
>
>
> –
> Don Burn (MVP, Windows DDK)
> Windows 2k/XP/2k3 Filesystem and Driver Consulting
> http://www.windrvr.com
>
> “Mike Kemp” wrote in message news:xxxxx@ntdev…
>> Hi from a new member / new to device drivers.
>>
>> I’m implementing a driver that writes real time data back to my
>> application buffers as fast as possible. It’s working okay, using
>> necessary probe and lock to fix the buffers in RAM and to write to them.
>>
>> I’m worried if it is / can be made safe against the user process
>> terminating abnormally - if it goes away before it can stop the driver
>> I’m going to get a bug check DRIVER_LEFT_LOCKED_PAGES_IN_PROCESS. Fine if
>> everything runs properly and nothing shuts down my user mode app before I
>> can switch off the driver, but is there any way of guaranteeing this?
>>
>> There are a couple of reasons including performance issues why I don’t
>> want to move the buffers wholly into the kernel and rely on signalling
>> events for the app to read the data - but I suppose it would be a plan B
>> if all else fails.
>>
>> If an experienced driver writer can tell me to stop and go to plan B
>> immediately I’d appreciate the tip!
>>
>> Thanks in advance
>>
>> Mike
>>
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

8

Hi Don

Thanks for the tip. I’ve got it (almost) working using the current scheme so
will simply add the cleanup for now so I can get the rest of our user mode
system running. Once everything is running I can review the i/o method on
the basis of more experience along the lines you suggest.

I suspect the next change will be to move to KMDF, but I feel that the raw
DDK experience (though painful) is probably a good basis for that move at
the next revision. (I suspect that, like MFC, an understanding of what is
happening underneath makes the more abstracted version understandable).

Thanks also to Tim and Doron…

ATB, Mike

----- Original Message -----
From: Don Burn
Newsgroups: ntdev
To: Windows System Software Devs Interest List
Sent: Tuesday, May 30, 2006 5:41 PM
Subject: Re:[ntdev] Re:driver writing back to user space

Mike,

As Doron and Tim pointed out you will recieve an IRP_MJ_CLEANUP when
the process is closing the handle (even a abnormal terminate) to the device.
If you do not mind using OVERLAPPED I/O seriously consider replacing your
model with a bunch of IOCTL’s, one for each buffer that are
METHOD_OUT_DIRECT. This will actually simplify your driver, since:

  1. The I/O subsystem will give you a MDL for the buffer with the
    data locked down
  2. The completion of the IRP provides an easy way to notify the
    user the buffer is ready to be accessed.
  3. The cancel routine of the IRP will handle the cleanup of the
    buffer

Also, KMDF which will simplify the heck out of your driver, is based
around handling IRP’s so much of the pain is taken care of for you.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
http://www.windrvr.com

“Mike Kemp” wrote in message news:xxxxx@ntdev…
> Hi Don
>
> I simply pass (an array of) user mode pointers in using METHOD_BUFFERED.
>
> Once a (user mode) pointer is in my driver I allocate an MDL to describe
> it, do a
>
> MmProbeAndLockPages(pMdl, KernelMode, IoModifyAccess);
>
> to fix it in memory, obtain a kernel mode pointer with
>
> MmGetSystemAddressForMdlSafe(pMdl, HighPagePriority);
>
> and store all these pointers in the device extension.
>
> Then real time interrupt callbacks write data to the buffer(s) pointed to,
> ad infinitum (or until I tell the driver to stop).
>
> I’m still getting my head around IRPs but I’d assume that all are complete
> at the time this is running, as all my calls are non-overlapped, simply
> passing in control values, pointers etc.
>
> I assume that when the user process terminates for any reason, XP notifies
> the driver that it’s handle is (being) closed. If I can unlock the memory
> in time I’d be okay, but I’m not sure how to do that at the moment…
>
> Thanks for any comments
>
> Mike
>
>
> ----- Original Message -----
> From: Don Burn
> Newsgroups: ntdev
> To: Windows System Software Devs Interest List
> Sent: Tuesday, May 30, 2006 4:55 PM
> Subject: Re:[ntdev] driver writing back to user space
>
>
> How are you getting the pointers into memory in the first place? If you
> are
> using a bunch of IRP’s to pass in buffers with something like
> METHOD_OUT_DIRECT you will recieve a cancel for the IRP before the process
> fully cleans up, and you can cleanup your usage of the buffers.
>
> Note: this is one of the reasons for avoiding tricks like events, since
> the
> handling of process termination is much harder.
>
>
> –
> Don Burn (MVP, Windows DDK)
> Windows 2k/XP/2k3 Filesystem and Driver Consulting
> http://www.windrvr.com
>
> “Mike Kemp” wrote in message news:xxxxx@ntdev…
>> Hi from a new member / new to device drivers.
>>
>> I’m implementing a driver that writes real time data back to my
>> application buffers as fast as possible. It’s working okay, using
>> necessary probe and lock to fix the buffers in RAM and to write to them.
>>
>> I’m worried if it is / can be made safe against the user process
>> terminating abnormally - if it goes away before it can stop the driver
>> I’m going to get a bug check DRIVER_LEFT_LOCKED_PAGES_IN_PROCESS. Fine if
>> everything runs properly and nothing shuts down my user mode app before I
>> can switch off the driver, but is there any way of guaranteeing this?
>>
>> There are a couple of reasons including performance issues why I don’t
>> want to move the buffers wholly into the kernel and rely on signalling
>> events for the app to read the data - but I suppose it would be a plan B
>> if all else fails.
>>
>> If an experienced driver writer can tell me to stop and go to plan B
>> immediately I’d appreciate the tip!
>>
>> Thanks in advance
>>
>> Mike
>>
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

—
Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

9

> I’m worried if it is / can be made safe against the user process terminating

abnormally - if it goes away before it can stop the driver I’m going to get
a bug check DRIVER_LEFT_LOCKED_PAGES_IN_PROCESS.

Yes, clean everything up in MJ_CLEANUP handler.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

10

> I simply pass (an array of) user mode pointers in using METHOD_BUFFERED.

Why not pass lots of overlapped IOCTLs using METHOD_IN/OUT_DIRECT instead,
using the overlapped file?

This loads the burden of calling MmProbeAndLockPages to the IO manager.

I’m still getting my head around IRPs but I’d assume that all are complete
at the time this is running, as all my calls are non-overlapped, simply
passing in control values, pointers etc.

Feeding the single file handle with lots of overlapped IRPs is a good idea.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

11

Hi Maxim

Thanks for the suggestions. I will probably try this the next time I “start
again” (e.g. for a more bullet-proof KMDF version). Right now the MJ_CLEANUP
has solved the outstanding problems and everything is springing into life
using the original scheme!

Am I right in thinking that if I switch to overlapped i/o, then every call
once the handle is opened has to be overlapped? I suppose I simply add a
user mode wait after all the calls that I want to finish straight away, just
leaving my main data i/o’s running in a pending state.

(I’m still in the first phase of making every mistake possible - the best
way to learn)

Mike

----- Original Message -----
From: Maxim S. Shatskih
To: Windows System Software Devs Interest List
Sent: Thursday, June 01, 2006 9:00 PM
Subject: Re: Re:[ntdev] driver writing back to user space

I simply pass (an array of) user mode pointers in using METHOD_BUFFERED.

Why not pass lots of overlapped IOCTLs using METHOD_IN/OUT_DIRECT instead,
using the overlapped file?

This loads the burden of calling MmProbeAndLockPages to the IO manager.

I’m still getting my head around IRPs but I’d assume that all are complete
at the time this is running, as all my calls are non-overlapped, simply
passing in control values, pointers etc.

Feeding the single file handle with lots of overlapped IRPs is a good idea.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

12

> Am I right in thinking that if I switch to overlapped i/o, then every call

once the handle is opened has to be overlapped?

Yes, at least for that specific handle. you can open a second handle for
non-overlapped operations.

I suppose I simply add a user mode wait after all the calls that I want to
finish straight away, just leaving my main data i/o’s running in a pending
state.

No. If you want to wait for the result, simply call GetOverlappedResult for
the overlapped structure that you used for the specific IO operation. That
can block (if you want) until the operation is completed.

–

Kind regards,
Bruno van Dooren
xxxxx@hotmail.com
Remove only “_nos_pam”

13

> No. If you want to wait for the result, simply call GetOverlappedResult for

the overlapped structure that you used for the specific IO operation. That
can block (if you want) until the operation is completed.

This wait requires valid event handle in each OVERLAPPED structure, which is
not always convinient. If you have lots of them - then you can use
Read/WriteFileEx and then SleepEx(0, TRUE) to gather the completion APCs - or
you can use IOCPs (this also works for DeviceIoControl).

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

14

> Am I right in thinking that if I switch to overlapped i/o, then every call

once the handle is opened has to be overlapped?

Yes, on an overlapped file, each call must have an OVERLAPPED structure.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com