Rapid IO

NTDEV
1

I have a WFP driver that queues packets and sends them to user mode for inspection. It mostly works. The problem I am facing is that the method (buffered IO) is giving me a basic problem.

a) If the user mode app is not running, what should the driver do? How would it know the user mode app is not running. It can’t just keep on queuing as this would leak memory.

b) The user mode app cannot register an IRP notification quickly enough so misses packets.

Currently, the packets retry to send to user mode and then give up after so many tries (20 at present), so if the user mode app is too busy to send in the IRP notification in time, packets effectively get dropped as the driver has had not decision from user mode.

Seeing as DeviceIOControl is all I know with regard to IO, my question is this.

Is there a better way of doing this, like registering a user mode function pointer (ie a callback) with the driver?

2

First how big a packets are these, you may want to look at a different
method than METHOD_BUFFERED. Second, you should know if the user mode
app has at least opened the driver. As far as IRP notification have
you looked at IoCompletionPorts? These are much faster than individual
I/O calls with events.

As I stated in an discussion of this a few weeks ago, a colleague and I
got roughly 600,000 messages a second on a slow server out through
hardware that was limiting the performance. I have seen over double
that on a test driver that did nothing with the data.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

“xxxxx@yahoo.co.uk” wrote in message
news:xxxxx@ntdev:

> I have a WFP driver that queues packets and sends them to user mode for inspection. It mostly works. The problem I am facing is that the method (buffered IO) is giving me a basic problem.
>
> a) If the user mode app is not running, what should the driver do? How would it know the user mode app is not running. It can’t just keep on queuing as this would leak memory.
>
> b) The user mode app cannot register an IRP notification quickly enough so misses packets.
>
> Currently, the packets retry to send to user mode and then give up after so many tries (20 at present), so if the user mode app is too busy to send in the IRP notification in time, packets effectively get dropped as the driver has had not decision from user mode.
>
> Seeing as DeviceIOControl is all I know with regard to IO, my question is this.
>
> Is there a better way of doing this, like registering a user mode function pointer (ie a callback) with the driver?

3

The packets aren’t send to user mode, sorry. A message is sent to user mode with the basic particulars of the IP packet to give enough information to make a decision. The decision is then passed back to the driver and it locates the relevant queued packet and operates on it accordingly.

I will look into these completionports you mention.

Thanks

4

Thoughts:

1.) Have application open a handle to WFP driver. Only queue packets as long
as that handle is open. Stop queuing and flush queue when handle is closed.

2.a.) Have multiple read IRPs in-flight concurrently. As read IRPs are
received driver marks them as pending and queues them. Can use a large
number of concurrent reads. Use Cancel Safe Queues to reduce IRP
queuing/cancellation problems. Idea is to have an IRP ready to complete when
packet arrives.

2.b) Instead of reading one packet at a time invent a scheme where you pass
larger buffer to driver and have it fill the larger buffer with multiple
packets. This reduces the number of read operations needed to fetch multiple
packets. May need a timer to insure partially filled buffer eventually
completes if traffic is slow.

2.c) Combination of 2.a) and 2.b)

No matter what - you will probably loose some…

Just ideas. Your mileage may vary.

Good luck,

Thomas F. Divine
http://www.pcausa.com

From:
Sent: Tuesday, August 31, 2010 12:07 PM
To: “Windows System Software Devs Interest List”
Subject: [ntdev] Rapid IO

> I have a WFP driver that queues packets and sends them to user mode for
> inspection. It mostly works. The problem I am facing is that the method
> (buffered IO) is giving me a basic problem.
>
> a) If the user mode app is not running, what should the driver do? How
> would it know the user mode app is not running. It can’t just keep on
> queuing as this would leak memory.
>
> b) The user mode app cannot register an IRP notification quickly enough so
> misses packets.
>
> Currently, the packets retry to send to user mode and then give up after
> so many tries (20 at present), so if the user mode app is too busy to send
> in the IRP notification in time, packets effectively get dropped as the
> driver has had not decision from user mode.
>
> Seeing as DeviceIOControl is all I know with regard to IO, my question is
> this.
>
> Is there a better way of doing this, like registering a user mode function
> pointer (ie a callback) with the driver?
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

5

Actually there is an interesting thing. It probably doesn’t matter some much
how MUCH data you pass up to your driver. Instead, what matters most is HOW
MANY TIMES you pass ANY information from your driver to your application. A
goal is to minimize the frequency that the driver notifies the application,
not so much how large the notifications are.

Of course, the amount of time that the application spends processing a
notification can be a killer.

Thomas F. Divine

From:
Sent: Tuesday, August 31, 2010 12:21 PM
To: “Windows System Software Devs Interest List”
Subject: RE:[ntdev] Rapid IO

> The packets aren’t send to user mode, sorry. A message is sent to user
> mode with the basic particulars of the IP packet to give enough
> information to make a decision. The decision is then passed back to the
> driver and it locates the relevant queued packet and operates on it
> accordingly.
>
> I will look into these completionports you mention.
>
> Thanks
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

6

I would modify Thomas’s ideas by converting the driver to KMDF and then not
have to provide cancel safe queues.

I’ve much the same kind of driver/service/application interface, and since
it is quite possible that the driver and app/service may be out of sync on
start, my driver on in a callout looks for the next IO request packet in a
WDF IO queue. If it does not find one, the callout is marked as blocked and
returned to NDIS. You should be able to establish the same kind of a
failsafe convention for the driver to handle data when your application is
not running.

Gary G. Little
H (952) 223-1349
C (952) 454-4629
xxxxx@comcast.net

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Thomas F. Divine
Sent: Tuesday, August 31, 2010 11:23 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Rapid IO

Thoughts:

1.) Have application open a handle to WFP driver. Only queue packets as long
as that handle is open. Stop queuing and flush queue when handle is closed.

2.a.) Have multiple read IRPs in-flight concurrently. As read IRPs are
received driver marks them as pending and queues them. Can use a large
number of concurrent reads. Use Cancel Safe Queues to reduce IRP
queuing/cancellation problems. Idea is to have an IRP ready to complete when
packet arrives.

2.b) Instead of reading one packet at a time invent a scheme where you pass
larger buffer to driver and have it fill the larger buffer with multiple
packets. This reduces the number of read operations needed to fetch multiple
packets. May need a timer to insure partially filled buffer eventually
completes if traffic is slow.

2.c) Combination of 2.a) and 2.b)

No matter what - you will probably loose some…

Just ideas. Your mileage may vary.

Good luck,

Thomas F. Divine
http://www.pcausa.com

From:
Sent: Tuesday, August 31, 2010 12:07 PM
To: “Windows System Software Devs Interest List”
Subject: [ntdev] Rapid IO

> I have a WFP driver that queues packets and sends them to user mode
> for inspection. It mostly works. The problem I am facing is that the
> method (buffered IO) is giving me a basic problem.
>
> a) If the user mode app is not running, what should the driver do? How
> would it know the user mode app is not running. It can’t just keep on
> queuing as this would leak memory.
>
> b) The user mode app cannot register an IRP notification quickly
> enough so misses packets.
>
> Currently, the packets retry to send to user mode and then give up
> after so many tries (20 at present), so if the user mode app is too
> busy to send in the IRP notification in time, packets effectively get
> dropped as the driver has had not decision from user mode.
>
> Seeing as DeviceIOControl is all I know with regard to IO, my question
> is this.
>
> Is there a better way of doing this, like registering a user mode
> function pointer (ie a callback) with the driver?
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

7

In addition you have a policy decision to make here regarding what is the
appropriate behavior when your application is not up. If you are
guaranteeing that no unfiltered packets are delivered, you have to keep that
commitment by tossing everything until your app is running.

Mark Roddy

On Tue, Aug 31, 2010 at 1:01 PM, Gary G. Little wrote:

> I would modify Thomas’s ideas by converting the driver to KMDF and then not
> have to provide cancel safe queues.
>
> I’ve much the same kind of driver/service/application interface, and since
> it is quite possible that the driver and app/service may be out of sync on
> start, my driver on in a callout looks for the next IO request packet in a
> WDF IO queue. If it does not find one, the callout is marked as blocked and
> returned to NDIS. You should be able to establish the same kind of a
> failsafe convention for the driver to handle data when your application is
> not running.
>
> Gary G. Little
> H (952) 223-1349
> C (952) 454-4629
> xxxxx@comcast.net
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Thomas F. Divine
> Sent: Tuesday, August 31, 2010 11:23 AM
> To: Windows System Software Devs Interest List
> Subject: Re: [ntdev] Rapid IO
>
> Thoughts:
>
> 1.) Have application open a handle to WFP driver. Only queue packets as
> long
> as that handle is open. Stop queuing and flush queue when handle is closed.
>
> 2.a.) Have multiple read IRPs in-flight concurrently. As read IRPs are
> received driver marks them as pending and queues them. Can use a large
> number of concurrent reads. Use Cancel Safe Queues to reduce IRP
> queuing/cancellation problems. Idea is to have an IRP ready to complete
> when
> packet arrives.
>
> 2.b) Instead of reading one packet at a time invent a scheme where you pass
> larger buffer to driver and have it fill the larger buffer with multiple
> packets. This reduces the number of read operations needed to fetch
> multiple
> packets. May need a timer to insure partially filled buffer eventually
> completes if traffic is slow.
>
> 2.c) Combination of 2.a) and 2.b)
>
> No matter what - you will probably loose some…
>
> Just ideas. Your mileage may vary.
>
> Good luck,
>
> Thomas F. Divine
> http://www.pcausa.com
>
>
> --------------------------------------------------
> From:
> Sent: Tuesday, August 31, 2010 12:07 PM
> To: “Windows System Software Devs Interest List”
> Subject: [ntdev] Rapid IO
>
> > I have a WFP driver that queues packets and sends them to user mode
> > for inspection. It mostly works. The problem I am facing is that the
> > method (buffered IO) is giving me a basic problem.
> >
> > a) If the user mode app is not running, what should the driver do? How
> > would it know the user mode app is not running. It can’t just keep on
> > queuing as this would leak memory.
> >
> > b) The user mode app cannot register an IRP notification quickly
> > enough so misses packets.
> >
> > Currently, the packets retry to send to user mode and then give up
> > after so many tries (20 at present), so if the user mode app is too
> > busy to send in the IRP notification in time, packets effectively get
> > dropped as the driver has had not decision from user mode.
> >
> > Seeing as DeviceIOControl is all I know with regard to IO, my question
> > is this.
> >
> > Is there a better way of doing this, like registering a user mode
> > function pointer (ie a callback) with the driver?
> >
> > —
> > NTDEV is sponsored by OSR
> >
> > For our schedule of WDF, WDM, debugging and other seminars visit:
> > http://www.osr.com/seminars
> >
> > To unsubscribe, visit the List Server section of OSR Online at
> > http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

8

Create a custom IOCTL that the application calls upon startup. Once the driver receives that, it knows that the app is there and can modify it’s behavior accordingly. When you receive that IOCTL, take note of the IO_STACK_LOCATION->FileObject. If you see that same FileObject at IRP_MJ_CLOSE, then you know your app has gone away.

9

@Mark: You may have missed my point. While the app is not running, all packets are tossed. The key here is to know when the app is running and queue the packets in good faith. Bearing in mind that the app (or apps) will launch multiple notification IRPs, in this case I would need to decide whether my app (or apps) are running, and if so what to do with the queued packets.

@Steve: Interesting, I will give this some deep thought, thanks for the steer.

10

> The key here is to know when the app is running and queue the
packets in good faith.

Please correct me if I’m wrong, but can’t you use a mutex/event to tell if the application is running?

The strategy I’ve used successfully before (but not in kernel mode!) is something like this:

* Caller (your driver) creates a named mutex
* Callee (your application), upon startup, enters the mutex
* Caller, at some point in its loop, calls WaitOne(0) on the mutex. If the result is WAIT_ABANDONED or WAIT_SUCCESS then the application has quit

or some variation thereof of this method.

Regards,

Mahmoud Al-Qudsi
Director,
NeoSmart Technologies

11

Windows kernel has a well-defined mechanism that allows a driver to when
user-mode handles are close. The kernel mechanisms deal with not only
detecting when application closes the handles, but also with the issue of
cancelling pending IRPs that the driver may have queued.

WDK samples like general/cancel - as well as most other WDK samples -
illustrate how to handle IRP_MJ_CREATE/CLEANUP/CLOSE, etc.

Read the WDK documentation about handling IRPs and their lifecycle.

One could invent other ways, but you are only asking for BIG trouble.

Thomas F. Divine
http://www.pcausa.com

From:
Sent: Tuesday, August 31, 2010 9:40 PM
To: “Windows System Software Devs Interest List”
Subject: RE:[ntdev] Rapid IO

>> The key here is to know when the app is running and queue the
> packets in good faith.
>
> Please correct me if I’m wrong, but can’t you use a mutex/event to tell if
> the application is running?
>
> The strategy I’ve used successfully before (but not in kernel mode!) is
> something like this:
>
> * Caller (your driver) creates a named mutex
> * Callee (your application), upon startup, enters the mutex
> * Caller, at some point in its loop, calls WaitOne(0) on the mutex. If the
> result is WAIT_ABANDONED or WAIT_SUCCESS then the application has quit
>
> or some variation thereof of this method.
>
> Regards,
>
> Mahmoud Al-Qudsi
> Director,
> NeoSmart Technologies
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

12

To the OP:

I would say it’s fairly definite if you have an IO request sitting in a
queue, whether it be a cancel safe queue or a WDF queue, that your
application, or at least someone’s application is running, and has a handle
to the driver.

Gary G. Little
H (952) 223-1349
C (952) 454-4629
xxxxx@comcast.net

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Thomas F. Divine
Sent: Tuesday, August 31, 2010 8:49 PM
To: Windows System Software Devs Interest List
Subject: Re: RE:[ntdev] Rapid IO

Windows kernel has a well-defined mechanism that allows a driver to when
user-mode handles are close. The kernel mechanisms deal with not only
detecting when application closes the handles, but also with the issue of
cancelling pending IRPs that the driver may have queued.

WDK samples like general/cancel - as well as most other WDK samples -
illustrate how to handle IRP_MJ_CREATE/CLEANUP/CLOSE, etc.

Read the WDK documentation about handling IRPs and their lifecycle.

One could invent other ways, but you are only asking for BIG trouble.

Thomas F. Divine
http://www.pcausa.com

From:
Sent: Tuesday, August 31, 2010 9:40 PM
To: “Windows System Software Devs Interest List”
Subject: RE:[ntdev] Rapid IO

>> The key here is to know when the app is running and queue the
> packets in good faith.
>
> Please correct me if I’m wrong, but can’t you use a mutex/event to
> tell if the application is running?
>
> The strategy I’ve used successfully before (but not in kernel mode!)
> is something like this:
>
> * Caller (your driver) creates a named mutex
> * Callee (your application), upon startup, enters the mutex
> * Caller, at some point in its loop, calls WaitOne(0) on the mutex. If
> the result is WAIT_ABANDONED or WAIT_SUCCESS then the application has
> quit
>
> or some variation thereof of this method.
>
> Regards,
>
> Mahmoud Al-Qudsi
> Director,
> NeoSmart Technologies
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

13

OR, skip the IOCTL and just know from getting the CREATE. And as Mr. Prochniak said… getting the close will tell you the app has gone away. This would be the Windows-way to do this…

Peter
OSR

14

> The strategy I’ve used successfully before (but not in kernel mode!) is something like this: *

Caller (your driver) creates a named mutex * Callee (your application), upon startup, enters
the mutex * Caller, at some point in its loop, calls WaitOne(0) on the mutex.
If the result is WAIT_ABANDONED or WAIT_SUCCESS then the application has quit

Is not it easier to wait on process/thread handle with zero timeout??? If application has been terminated the wait will be satisfied, otherwise you will get WAIT_TIMEOUT…

Anton Bassov

15

>Is not it easier to wait on process/thread handle with zero timeout??? If

application has been terminated the wait will be satisfied, otherwise you will
get WAIT_TIMEOUT…

I’m tempted to say yes… But I cannot remember if my reason for not using this method is just that I didn’t have any direct communication between the two and therefore couldn’t know the process ID or if there was also a more compelling reason.

If the OP has no problem sending arbitrary data from the usermode app to the driver, then your method is probably best.

Mahmoud Al-Qudsi
Director,
NeoSmart Technologies

16

>But I cannot remember … if there was also a more compelling reason.

Sorry for the double post, but I just remembered my reason. I had a single process with multiple threads, each thread was an individual caller and threads could vanish from existence at whim.

17

“each thread was an individual caller and threads
could vanish from existence at whim”.

If you want your driver to detect if a client thread dies, you can have the thread to submit an overlapped request, the driver should pend it indefinitely. Then, if a thread goes away without properly cleaning its requests (which SHOULD NOT happen with proper design), the pended request will get IoCancelIrp call down to its cancel routine. Then the driver known the client is gone.

18

Thanks for all the replies. I am learning a lot which is always good!

During the IRP_J_CREATE and IRP_J_CLOSE events I can increment and decrement a client counter, and test this to see if anything is connected.

The results are that the driver chunders away passing packets up to user mode, and if it can’t then it queues them safe in the knowledge a client app is attached.

The problem is that the user mode app just can’t keep up (whoever said .Net was just as fast as native C needs their head examined)

So in an effort to keep the IO request to a minimum I will adapt the code to pass them up in batches and see if this improves things.

19

xxxxx@yahoo.co.uk wrote:

The problem is that the user mode app just can’t keep up (whoever said .Net was just as fast as native C needs their head examined)

A .NET app certainly CAN be just as fast as native C. The problem is
that any handling of raw memory buffers just sucks in .NET. There’s no
way around it. You can’t cast pointers from one type to another, so you
end up calling conversion APIs to convert each individual data item,
over and over and over, and that’s very time-consuming. Try reading in
a block of bytes that consists of a length dword and 2048 floats. You
have to convert one float at a time.

That shortcoming is the one major thing that has really kept me from
switching to C# for most of my diagnostics. Even Python has better
built-in structure/buffer handling then .NET.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.