asynchronous queue

I have a controldevice in my driver with which I communicate to user mode.
Now I want to create a queue in the driver and my devices should put there information asynchronously. An service on the user mode should now look periodically in the driver if information are in the queue.
Can anybody say which example in the wdk example shows me the path to such an solution?

On Sep 9, 2018, at 4:25 AM, xxxxx@x-publisher.com wrote:
>
> I have a controldevice in my driver with which I communicate to user mode.
> Now I want to create a queue in the driver and my devices should put there information asynchronously. An service on the user mode should now look periodically in the driver if information are in the queue.
> Can anybody say which example in the wdk example shows me the path to such an solution?

That’s the wrong model. You should have your user-mode app submit an ioctl that gets pended in the driver. When new information is available, the driver completes that ioctl. This is called “inverted call”. In fact, you can save one step by just returning the data when you complete the ioctl.

Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Even when using the inverted call model, the driver should have a ring buffer of data in case there is no pended IOCTL at the time the data is generated

d

-----Original Message-----
From: xxxxx@lists.osr.com On Behalf Of xxxxx@probo.com
Sent: Sunday, September 9, 2018 4:37 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] asynchronous queue

On Sep 9, 2018, at 4:25 AM, xxxxx@x-publisher.com wrote:
>
> I have a controldevice in my driver with which I communicate to user mode.
> Now I want to create a queue in the driver and my devices should put there information asynchronously. An service on the user mode should now look periodically in the driver if information are in the queue.
> Can anybody say which example in the wdk example shows me the path to such an solution?

That’s the wrong model. You should have your user-mode app submit an ioctl that gets pended in the driver. When new information is available, the driver completes that ioctl. This is called “inverted call”. In fact, you can save one step by just returning the data when you complete the ioctl.

Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.


NTDEV is sponsored by OSR

Visit the list online at: https:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at https:

To unsubscribe, visit the List Server section of OSR Online at https:</https:></https:></https:>

What speaks against the solution that I put the data in a ring buffer or list in the driver and the user app gets the data directly from that buffer without generating an request and this completion things?

The problems with this are you are still calling the kernel, you need a call
to be notified that there is data unless you poll which in today’s systems
is a pretty poor approach. Also, you have the challenges of sharing memory
from kernel to user space, which can be many.

Inverted call is the model that is well proven.

Don Burn
Windows Driver Consulting
Website: http://www.windrvr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@x-publisher.com
Sent: Monday, September 10, 2018 10:58 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] asynchronous queue

What speaks against the solution that I put the data in a ring buffer or
list in the driver and the user app gets the data directly from that buffer
without generating an request and this completion things?


NTDEV is sponsored by OSR

Visit the list online at:
http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software
drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at
http:</http:></http:></http:>

Yes, this polling isn’t a nice solution. In my case I have very less data but I will look how this inverted call must be implemented. Which example from the sdk shows this inverted call approach?

Take a look at
https://www.osr.com/nt-insider/2013-issue1/inverted-call-model-kmdf/

Don Burn
Windows Driver Consulting
Website: http://www.windrvr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@x-publisher.com
Sent: Monday, September 10, 2018 11:41 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] asynchronous queue

Yes, this polling isn’t a nice solution. In my case I have very less data
but I will look how this inverted call must be implemented. Which example
from the sdk shows this inverted call approach?


NTDEV is sponsored by OSR

Visit the list online at:
http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software
drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at
http:</http:></http:></http:>

xxxxx@x-publisher.com wrote:

What speaks against the solution that I put the data in a ring buffer or list in the driver and the user app gets the data directly from that buffer without generating an request and this completion things?

How would your user app know that data was available?  You’d either have
to poll periodically, which is wasteful and not timely, or you’d need an
event, which is exactly what having a pending ioctl does.

Plus, when you’re caching in the driver, you need to figure out how to
deal with buffer overflows.  You can’t cache an infinite amount of data.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Ah, very nice that model. I have one question.
If I understood the concept correctly then it is possible that if data are in my list or ring buffer on the driver I can immediately get that data back. If this is not the case I make a pending request and wait till something is completed.
My requirement says that I should as fast as possible get the data from the driver queue. In this case I let always a pending request in the driver open. Is this right?

If you want the fastest have more than one request queued, then you can
complete the request. Basically have the number of requests that would
handle most flows, and then have a ring buffer or other mechanism for
exceptional circumstances.

Don Burn
Windows Driver Consulting
Website: http://www.windrvr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@x-publisher.com
Sent: Monday, September 10, 2018 2:35 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] asynchronous queue

Ah, very nice that model. I have one question.
If I understood the concept correctly then it is possible that if data are
in my list or ring buffer on the driver I can immediately get that data
back. If this is not the case I make a pending request and wait till
something is completed.
My requirement says that I should as fast as possible get the data from the
driver queue. In this case I let always a pending request in the driver
open. Is this right?


NTDEV is sponsored by OSR

Visit the list online at:
http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software
drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at
http:</http:></http:></http:>

xxxxx@x-publisher.com wrote:

Ah, very nice that model. I have one question.
If I understood the concept correctly then it is possible that if data are in my list or ring buffer on the driver I can immediately get that data back. If this is not the case I make a pending request and wait till something is completed.

Well, the application just submits a read request (or an equivalent
ioctl).  If the driver has data waiting, it copies the data and
completes the request immediately.  Otherwise, it returns, leaving the
request pending.  Later, when data arrives, it pops the next waiting
request and completes it.

My requirement says that I should as fast as possible get the data from the driver queue. In this case I let always a pending request in the driver open. Is this right?

Right.  Or even submit multiple requests, so there’s always one waiting
if data comes in quickly.  That’s how streaming audio/video drivers
handle it.  In addition, if you submit your read requests before
triggering the device, you may be able to eliminate the ring buffer
completely.  That simplifies the processing and eliminates strange
corner cases.

Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

The whole thing can be described with a single word “poll”, which is generally taken as yet another 4-letter word by the kernel-mode developers…

There are some situations when using this approach may be reasonable. For example, consider the scenario when the device produces large amounts of data in large bursts that occur from time to time, while being generally inactive most of the time. In sch case you may, indeed, want to wait a bit after having been notified by a driver just in order to make sure that you read as much data as possible, rather than reading data from the buffer straight away upon receiving notification (or,
in other words, to implement something that is known as select()/poll() semantics on UNIX-like systems).

However, even with this approach you still need asynch notifications from the driver.
If you want to implement polling in its classical sense (i.e. to check the target buffer periodically), I can assure you that this approach is just profoundly dumb…

Anton Bassov

I would also consider using the minifilter’s communication ports for this.
I think your use case is nicely shown in the avscan sample where the driver
simply sends “scan requests” to the user-mode service and the user-mode
service worker threads process these async.
The sample is quite big but looking at the user-mode userscan.c should tell
you enough whether this is a viable option for you.

PS: You do not need to be (mini)filtering anything in your driver in order
to use the communication ports. Just register with the filter manager just
like the nullFilter and then just use the ports in your driver.

Cheers,
Gabriel

On Tue, Sep 11, 2018 at 1:34 AM xxxxx@hotmail.com <
xxxxx@lists.osr.com> wrote:

The whole thing can be described with a single word “poll”, which is
generally taken as yet another 4-letter word by the kernel-mode
developers…

There are some situations when using this approach may be reasonable. For
example, consider the scenario when the device produces large amounts of
data in large bursts that occur from time to time, while being generally
inactive most of the time. In sch case you may, indeed, want to wait a bit
after having been notified by a driver just in order to make sure that you
read as much data as possible, rather than reading data from the buffer
straight away upon receiving notification (or,
in other words, to implement something that is known as select()/poll()
semantics on UNIX-like systems).

However, even with this approach you still need asynch notifications from
the driver.
If you want to implement polling in its classical sense (i.e. to check the
target buffer periodically), I can assure you that this approach is just
profoundly dumb…

Anton Bassov


NTDEV is sponsored by OSR

Visit the list online at: <
http://www.osronline.com/showlists.cfm?list=ntdev\>

MONTHLY seminars on crash dump analysis, WDF, Windows internals and
software drivers!
Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
>


Bercea. G.</http:>

a “large enough” ringbuffer with signalling on transition to
not-empty/not-full generally satisfies the objectives.
Mark Roddy

On Mon, Sep 10, 2018 at 7:33 PM xxxxx@hotmail.com <
xxxxx@lists.osr.com> wrote:

The whole thing can be described with a single word “poll”, which is
generally taken as yet another 4-letter word by the kernel-mode
developers…

There are some situations when using this approach may be reasonable. For
example, consider the scenario when the device produces large amounts of
data in large bursts that occur from time to time, while being generally
inactive most of the time. In sch case you may, indeed, want to wait a bit
after having been notified by a driver just in order to make sure that you
read as much data as possible, rather than reading data from the buffer
straight away upon receiving notification (or,
in other words, to implement something that is known as select()/poll()
semantics on UNIX-like systems).

However, even with this approach you still need asynch notifications from
the driver.
If you want to implement polling in its classical sense (i.e. to check the
target buffer periodically), I can assure you that this approach is just
profoundly dumb…

Anton Bassov


NTDEV is sponsored by OSR

Visit the list online at: <
http://www.osronline.com/showlists.cfm?list=ntdev\>

MONTHLY seminars on crash dump analysis, WDF, Windows internals and
software drivers!
Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

While that WOULD work, Minifilter Communication Ports are really just a specific implementation of Inverted Call. Unless you’re actually writing a Minifilter, I’d suggest you NOT use Minifilter Communication Ports… Nobody who maintains your driver after you, except perhaps Mr. Bercea, is going to expect to see Minifilter Communication Ports used in a device driver. When they DO see them, its going to cause them to stop and wonder what they’re missing… “Is this driver a Minifilter?”

Think more of the people who come AFTER you, who need to maintain your code, than merely what’s expedient for you today. Please.

Peter
OSR
@OSRDrivers

Note that poll ? wait is proven to be more effective when the frequency of ?events? is very high. Most modern NIC / HBA drivers will switch between the two modes depending on traffic

The same efficiency can be applied in both Km and UM code. I know I have in my own private library of code that I have been using in both for about two decades now

Sent from Mailhttps: for Windows 10

________________________________
From: xxxxx@lists.osr.com on behalf of xxxxx@hotmail.com
Sent: Monday, September 10, 2018 7:33:55 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] asynchronous queue



The whole thing can be described with a single word “poll”, which is generally taken as yet another 4-letter word by the kernel-mode developers…

There are some situations when using this approach may be reasonable. For example, consider the scenario when the device produces large amounts of data in large bursts that occur from time to time, while being generally inactive most of the time. In sch case you may, indeed, want to wait a bit after having been notified by a driver just in order to make sure that you read as much data as possible, rather than reading data from the buffer straight away upon receiving notification (or,
in other words, to implement something that is known as select()/poll() semantics on UNIX-like systems).

However, even with this approach you still need asynch notifications from the driver.
If you want to implement polling in its classical sense (i.e. to check the target buffer periodically), I can assure you that this approach is just profoundly dumb…

Anton Bassov


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></https:>

> Note that poll ? wait is proven to be more effective when the frequency of ?events? is

very high. Most modern NIC / HBA drivers will switch between the two modes depending on traffic

Well, this is not about “poll vs wait”, but about “poll vs interrupt” (i.e “poll vs asynch notification)”…

As I had said in my previous post, a combination of these two may be quite useful when the device produces large amounts of data in large bursts that occur from time to time, while being generally inactive most of the time. High-performing NICs and HBAs fall into this class of devices, so that dealing with them may be based upon this approach. Linux NAPI (“new” network API) is the very first example that gets into my head.

However, please note that this approach is not necessarily going to suit everyone.
The fact that it is beneficial for a high-performing device that produces dozens of gigabytes of data per second does not necessarily imply that it may be suitable for, say, a kbd, a mouse,or any other low-datarate device. In fact, most devices would be better off if they apply “process upon interrupt
occurence” principle, rather than combining interrupt mode with polling…

Anton Bassov