Priority???

NTDEV
1

Dear All,

I set up a system thread in my driver. It submits a read irp to poll my
device and I set a complete routine for the irp. If there is any data
received, complete routine will copy the data in a buffer and resend the
irp for polling. Thus if user-model AP send read request to driver, driver
will get data from buffer and complete AP’s request.

My problem is that it seems the polling irp has a higher priority. If
there are many data need get from device, the complete routine will be
continuely called and fill the buffer up soon. At the same time I have a
user-model read request but it can’t be responded in time. How could I
make the ReadFile request and the polling irp synchronized?

Any advice is useful. Thanks!

Best Regards,

Dengwen

2

Hi, DengWen,

Can you stop polling if your buffer reaches some watermark, say, 75% full ? Restart when it goes back to some low mark ? Or, can you poll on a timer, and slow the timer down as the buffer gets closer to the watermark, speeding it up as it becomes emptier ? Maybe using a queue instead of a single buffer ?

Alberto.
----- Original Message -----
From: xxxxx@sunplus.com.cn
To: Windows System Software Devs Interest List
Sent: Thursday, March 31, 2005 10:45 PM
Subject: [ntdev] Priority???

Dear All,

I set up a system thread in my driver. It submits a read irp to poll my device and I set a complete routine for the irp. If there is any data received, complete routine will copy the data in a buffer and resend the irp for polling. Thus if user-model AP send read request to driver, driver will get data from buffer and complete AP’s request.

My problem is that it seems the polling irp has a higher priority. If there are many data need get from device, the complete routine will be continuely called and fill the buffer up soon. At the same time I have a user-model read request but it can’t be responded in time. How could I make the ReadFile request and the polling irp synchronized?

Any advice is useful. Thanks!

Best Regards,

Dengwen — Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256 You are currently subscribed to ntdev as: xxxxx@ieee.org To unsubscribe send a blank email to xxxxx@lists.osr.com

3

Send a lot of overlapped ReadFile requests, provide the cancel-safe queue for them in the driver. Then handle their completion in the user app.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: xxxxx@sunplus.com.cn
To: Windows System Software Devs Interest List
Sent: Friday, April 01, 2005 7:45 AM
Subject: [ntdev] Priority???

Dear All,

I set up a system thread in my driver. It submits a read irp to poll my device and I set a complete routine for the irp. If there is any data received, complete routine will copy the data in a buffer and resend the irp for polling. Thus if user-model AP send read request to driver, driver will get data from buffer and complete AP’s request.

My problem is that it seems the polling irp has a higher priority. If there are many data need get from device, the complete routine will be continuely called and fill the buffer up soon. At the same time I have a user-model read request but it can’t be responded in time. How could I make the ReadFile request and the polling irp synchronized?

Any advice is useful. Thanks!

Best Regards,

Dengwen — Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256 You are currently subscribed to ntdev as: xxxxx@storagecraft.com To unsubscribe send a blank email to xxxxx@lists.osr.com

4

This suggestion will transfer the overflow problem from the driver to the hardware :slight_smile:

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
----- Original Message -----
From: Alberto Moreira
To: Windows System Software Devs Interest List
Sent: Friday, April 01, 2005 8:21 AM
Subject: Re: [ntdev] Priority???

Hi, DengWen,

Can you stop polling if your buffer reaches some watermark, say, 75% full ? Restart when it goes back to some low mark ? Or, can you poll on a timer, and slow the timer down as the buffer gets closer to the watermark, speeding it up as it becomes emptier ? Maybe using a queue instead of a single buffer ?

Alberto.
----- Original Message -----
From: xxxxx@sunplus.com.cn
To: Windows System Software Devs Interest List
Sent: Thursday, March 31, 2005 10:45 PM
Subject: [ntdev] Priority???

Dear All,

I set up a system thread in my driver. It submits a read irp to poll my device and I set a complete routine for the irp. If there is any data received, complete routine will copy the data in a buffer and resend the irp for polling. Thus if user-model AP send read request to driver, driver will get data from buffer and complete AP’s request.

My problem is that it seems the polling irp has a higher priority. If there are many data need get from device, the complete routine will be continuely called and fill the buffer up soon. At the same time I have a user-model read request but it can’t be responded in time. How could I make the ReadFile request and the polling irp synchronized?

Any advice is useful. Thanks!

Best Regards,

Dengwen — Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256 You are currently subscribed to ntdev as: xxxxx@ieee.org To unsubscribe send a blank email to xxxxx@lists.osr.com

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

5

Note that I put question marks in both cases: the feasibility of my proposition depends on his particular environment.

Yet if the software cannot keep up with the hardware, he has a problem anyway, and shuffling priorities is unlikely to solve it. It’s called “flow control”: if the software can’t keep up, throttle the hardware. I suggest looking at the issue as a standard producer-consumer problem, it’s been hacked to death and back by the Computer Science community. If you want to keep tabs on a full buffer condition, you can use a “full” semaphore which you initialize with the number of items your buffer can keep; if you also want to keep tabs on an empty buffer condition, you need a second semaphore. The solution is easily sucked up from any OS 101 textbook. The producer thread waits on the “full” semaphore, the consumer releases it after taking one item out of the buffer.

If you have a better solution, by all means, publish it!

Alberto.

----- Original Message -----
From: Maxim S. Shatskih
To: Windows System Software Devs Interest List
Sent: Friday, April 01, 2005 3:02 AM
Subject: Re: [ntdev] Priority???

This suggestion will transfer the overflow problem from the driver to the hardware :slight_smile:

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
----- Original Message -----
From: Alberto Moreira
To: Windows System Software Devs Interest List
Sent: Friday, April 01, 2005 8:21 AM
Subject: Re: [ntdev] Priority???

Hi, DengWen,

Can you stop polling if your buffer reaches some watermark, say, 75% full ? Restart when it goes back to some low mark ? Or, can you poll on a timer, and slow the timer down as the buffer gets closer to the watermark, speeding it up as it becomes emptier ? Maybe using a queue instead of a single buffer ?

Alberto.
----- Original Message -----
From: xxxxx@sunplus.com.cn
To: Windows System Software Devs Interest List
Sent: Thursday, March 31, 2005 10:45 PM
Subject: [ntdev] Priority???

Dear All,

I set up a system thread in my driver. It submits a read irp to poll my device and I set a complete routine for the irp. If there is any data received, complete routine will copy the data in a buffer and resend the irp for polling. Thus if user-model AP send read request to driver, driver will get data from buffer and complete AP’s request.

My problem is that it seems the polling irp has a higher priority. If there are many data need get from device, the complete routine will be continuely called and fill the buffer up soon. At the same time I have a user-model read request but it can’t be responded in time. How could I make the ReadFile request and the polling irp synchronized?

Any advice is useful. Thanks!

Best Regards,

Dengwen — Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256 You are currently subscribed to ntdev as: xxxxx@ieee.org To unsubscribe send a blank email to xxxxx@lists.osr.com

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

6

You bump into the same problem: what do you do when you don’t have any more space to hold items in your csq ? If the software cannot keep up with the hardware, your csq is going to accumulate items jolly fast and potentially clog the system. Any kind of buffering runs into this eventual limitation, either the buffer is big enough to handle eventual differences in speed between producer (the hardware) and consumer (the user) or the system’s going to back up and you will need some flow control mechanism to avoid a system clog.

Alberto.

----- Original Message -----
From: Maxim S. Shatskih
To: Windows System Software Devs Interest List
Sent: Friday, April 01, 2005 3:01 AM
Subject: Re: [ntdev] Priority???

Send a lot of overlapped ReadFile requests, provide the cancel-safe queue for them in the driver. Then handle their completion in the user app.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: xxxxx@sunplus.com.cn
To: Windows System Software Devs Interest List
Sent: Friday, April 01, 2005 7:45 AM
Subject: [ntdev] Priority???

Dear All,

I set up a system thread in my driver. It submits a read irp to poll my device and I set a complete routine for the irp. If there is any data received, complete routine will copy the data in a buffer and resend the irp for polling. Thus if user-model AP send read request to driver, driver will get data from buffer and complete AP’s request.

My problem is that it seems the polling irp has a higher priority. If there are many data need get from device, the complete routine will be continuely called and fill the buffer up soon. At the same time I have a user-model read request but it can’t be responded in time. How could I make the ReadFile request and the polling irp synchronized?

Any advice is useful. Thanks!

Best Regards,

Dengwen — Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256 You are currently subscribed to ntdev as: xxxxx@storagecraft.com To unsubscribe send a blank email to xxxxx@lists.osr.com

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

7

> Dear All,

I set up a system thread in my driver. It submits a read irp to poll my
device and I set a complete routine for the irp. If there is any data
received, complete routine will copy the data in a buffer and resend the
irp for polling. Thus if user-model AP send read request to driver, driver
will get data from buffer and complete AP’s request.

My problem is that it seems the polling irp has a higher priority. If
there are many data need get from device, the complete routine will be
continuely called and fill the buffer up soon. At the same time I have a
user-model read request but it can’t be responded in time. How could I
make the ReadFile request and the polling irp synchronized?

Any advice is useful. Thanks!

Is your system thread polling repeatedly or does it sleep for some
time between each check of the hardware? When polling (as opposed to
interrupt driven), it is important to not check the hardware more
often than necessary. This can be controlled with a timer. When the
polling thread is sleeping the user (consuming) thread will have an
opportunity to complete a read request.

Robert Newton

8

Flow control is unsuitable for realtime traffic. For it, only the proper app’s behaviour (send lots of overlapped reads) gives the chances of really receiving all data without the dropped frames.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
----- Original Message -----
From: Alberto Moreira
To: Windows System Software Devs Interest List
Sent: Friday, April 01, 2005 6:06 PM
Subject: Re: [ntdev] Priority???

Note that I put question marks in both cases: the feasibility of my proposition depends on his particular environment.

Yet if the software cannot keep up with the hardware, he has a problem anyway, and shuffling priorities is unlikely to solve it. It’s called “flow control”: if the software can’t keep up, throttle the

9

Csq does not allocate anything, it just reuses the IRP’s tail as an anchor.
The main memory hog is the buffer memory, and my suggestion is to offload this policy to the app, thus making drivers and hardware simpler.
For instance, DirectShow works this way.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
----- Original Message -----
From: Alberto Moreira
To: Windows System Software Devs Interest List
Sent: Friday, April 01, 2005 6:09 PM
Subject: Re: [ntdev] Priority???

You bump into the same problem: what do you do when you don’t have any more space to hold items in your csq ? If the software cannot keep up with the hardware, your csq is going to accumulate items jolly fast and potentially clog the system. Any kind of buffering runs into this eventual limitation, either the buffer is big enough to handle eventual differences in speed between producer (the hardware) and consumer (the user) or the system’s going to back up and you will need some flow control mechanism to avoid a system clog.

Alberto.

----- Original Message -----
From: Maxim S. Shatskih
To: Windows System Software Devs Interest List
Sent: Friday, April 01, 2005 3:01 AM
Subject: Re: [ntdev] Priority???

Send a lot of overlapped ReadFile requests, provide the cancel-safe queue for them in the driver. Then handle their completion in the user app.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: xxxxx@sunplus.com.cn
To: Windows System Software Devs Interest List
Sent: Friday, April 01, 2005 7:45 AM
Subject: [ntdev] Priority???

Dear All,

I set up a system thread in my driver. It submits a read irp to poll my device and I set a complete routine for the irp. If there is any data received, complete routine will copy the data in a buffer and resend the irp for polling. Thus if user-model AP send read request to driver, driver will get data from buffer and complete AP’s request.

My problem is that it seems the polling irp has a higher priority. If there are many data need get from device, the complete routine will be continuely called and fill the buffer up soon. At the same time I have a user-model read request but it can’t be responded in time. How could I make the ReadFile request and the polling irp synchronized?

Any advice is useful. Thanks!

Best Regards,

Dengwen — Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256 You are currently subscribed to ntdev as: xxxxx@storagecraft.com To unsubscribe send a blank email to xxxxx@lists.osr.com

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

10

This didn’t come out as I intended, I was in a hurry and I didn’t make myself clear - apologies.

What I did mean is the queue on the other side. The problem here isn’t too many IRPs but too few, and the “queue” I wanted to refer to wasn’t the csq but the buffer itself. The issue is, of course, that if the poller is left unchecked the buffer will fill unless the csq has a steady enough supplies of IRPs to remove the payload as the poller generates it.

I don’t know if letting the app handling the buffer works any better, because all you do is to move the shared item from kernel side to user side. There’s a major difference here from a video driver, a poller is an input device while video is output. Yes, it’s easy enough to buffer output on the app side because the app controls the pace of the data generation; but if you’re polling an input device, neither the driver nor the app have any control over the amount of data provided by the input. It’s more like a camera device, you’d better be able to handle its bandwidth, buffered or not, or you’re going to get a choppy image.

The other point was correctly made by Robert Newton, which I forgot to emphasize: don’t let your threads freewheel, put them on a timer or on a semaphore.

Alberto.

----- Original Message -----
From: Maxim S. Shatskih
To: Windows System Software Devs Interest List
Sent: Saturday, April 02, 2005 1:55 AM
Subject: Re: [ntdev] Priority???

Csq does not allocate anything, it just reuses the IRP’s tail as an anchor.
The main memory hog is the buffer memory, and my suggestion is to offload this policy to the app, thus making drivers and hardware simpler.
For instance, DirectShow works this way.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
----- Original Message -----
From: Alberto Moreira
To: Windows System Software Devs Interest List
Sent: Friday, April 01, 2005 6:09 PM
Subject: Re: [ntdev] Priority???

You bump into the same problem: what do you do when you don’t have any more space to hold items in your csq ? If the software cannot keep up with the hardware, your csq is going to accumulate items jolly fast and potentially clog the system. Any kind of buffering runs into this eventual limitation, either the buffer is big enough to handle eventual differences in speed between producer (the hardware) and consumer (the user) or the system’s going to back up and you will need some flow control mechanism to avoid a system clog.

Alberto.

----- Original Message -----
From: Maxim S. Shatskih
To: Windows System Software Devs Interest List
Sent: Friday, April 01, 2005 3:01 AM
Subject: Re: [ntdev] Priority???

Send a lot of overlapped ReadFile requests, provide the cancel-safe queue for them in the driver. Then handle their completion in the user app.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: xxxxx@sunplus.com.cn
To: Windows System Software Devs Interest List
Sent: Friday, April 01, 2005 7:45 AM
Subject: [ntdev] Priority???

Dear All,

I set up a system thread in my driver. It submits a read irp to poll my device and I set a complete routine for the irp. If there is any data received, complete routine will copy the data in a buffer and resend the irp for polling. Thus if user-model AP send read request to driver, driver will get data from buffer and complete AP’s request.

My problem is that it seems the polling irp has a higher priority. If there are many data need get from device, the complete routine will be continuely called and fill the buffer up soon. At the same time I have a user-model read request but it can’t be responded in time. How could I make the ReadFile request and the polling irp synchronized?

Any advice is useful. Thanks!

Best Regards,

Dengwen — Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256 You are currently subscribed to ntdev as: xxxxx@storagecraft.com To unsubscribe send a blank email to xxxxx@lists.osr.com

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com

11

If you don’t have flow control, you’re going to lose data whenever your system can’t handle the bandwidth. There’s no such a thing as a free lunch! And you know, while multiple parallel activities help response time, throughput is often better handled by doing one thing at a time and doing it well: the least context switching you incurr, the highest your bandwidth will be.

Alberto.

----- Original Message -----
From: Maxim S. Shatskih
To: Windows System Software Devs Interest List
Sent: Saturday, April 02, 2005 1:54 AM
Subject: Re: [ntdev] Priority???

Flow control is unsuitable for realtime traffic. For it, only the proper app’s behaviour (send lots of overlapped reads) gives the chances of really receiving all data without the dropped frames.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com
----- Original Message -----
From: Alberto Moreira
To: Windows System Software Devs Interest List
Sent: Friday, April 01, 2005 6:06 PM
Subject: Re: [ntdev] Priority???

Note that I put question marks in both cases: the feasibility of my proposition depends on his particular environment.

Yet if the software cannot keep up with the hardware, he has a problem anyway, and shuffling priorities is unlikely to solve it. It’s called “flow control”: if the software can’t keep up, throttle the

Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: unknown lmsubst tag argument: ‘’
To unsubscribe send a blank email to xxxxx@lists.osr.com