Clarify what exactly is serialized about driver dispatch function calls.

Reading through Windows Internals 6th Edition, page 12, I see…

The I/O manager serializes IRPs for a driver by ensuring that the driver processes only one IRP at a time.
and…
Drivers can process multiple IRPs concurrently, but serialization is usually required for most devices because they cannot concurrently handle multiple I/O requests.
And on page 38-39…
Drivers must synchronize their access to global driver data and hardware registers for two reasons:

  • The execution of a driver can be preempted by higher-priority threads and time-slice (or quantum) expiration or can be interrupted by higher IRQL interrupts.
  • __On multiprocessor systems, Windows can run driver code simultaneously on more than one processor.

I understand how the I/O manager can have multiple IRPs going on at the same time, even from the same File and the same calling thread if asynchronous I/O is being used. But the meaning of “processes only one IRP at a time” is unclear. When is the driver “processing” an IRP, and does this prevent the driver from processing multiple IRPs at the same time in different threads?
What I believe this means is as follows, and you gurus please confirm or correct. I’d like to have this post be useful to future driver writers with similar concerns.

  • Driver is processing an IRP while one of its dispatch routines is being called, up to the time the routine returns.
  • If the routine returns STATUS_PENDING, the processing is still completed, but the IRP will remain in the system to be processed by a DPC, or further dispatch routines to possibly different drivers.
  • Driver might be processing two IRPs in different threads at the same time. I assume this because
    1. This would enable greater driver throughput.
    2. Otherwise, there would be no need for the driver to synchronize shared data between dispatch routines, and would only need to worry about ISRs or timer DPCs.

Now, if the driver can process multiple IRPs concurrently, then what does “processes only one IRP at a time” mean, and what difference does this make to the driver code (i.e., what synchronizations are unnecessary as a result?

So assuming I have to protect my driver code from being preemted by a higher priority thread, both of which are in a critical section, it seems that at a minimum, I would need a spinlock, because that raises the first thread to DISPATCH_LEVEL and assures that it will complete quickly. Simply using an interlocked exchange guard to protect the critical section wouldn’t work because the second thread would keep waiting for the guard to be released. The spinlock also locks other processors out of the critical section.

then what does “processes only one IRP at a time” mean

I have no idea. On its face this statement is incorrect.

and what difference does this make to the driver code (i.e., what synchronizations are unnecessary as a result?

Absolutely none.

The Windows I/O subsystem is designed to be inherently asynchronous. It will hand the driver as many IRPs, as it receives them, as threads send to devices “owned” by the driver. The I/O Manager does not queue, stream, sequence, or hold IRPs.

The I/O subsystem runs on multiple processors and is inherently reentrant. There is effectively no limit on the number of IRPs that can be active in a driver at a given time.

I have no idea what that sentence means, but I’m confident that it makes some other sense in context. Because, by itself, it’s not accurate.

Peter

On its face this statement is incorrect.

This is because the OP simply took it out of context. Let’s look at what the books actually says

[begin quote]

A start I/O routine. A driver can use a start I/O routine to initiate a data transfer to or from
a device. This routine is defined only in drivers that rely on the I/O manager to queue their
incoming I/O requests. The I/O manager serializes IRPs for a driver by ensuring that the driver
processes only one IRP at a time. Drivers can process multiple IRPs concurrently, but serializa-
tion is usually required for most devices because they cannot concurrently handle multiple I/O
requests.

[end quote]

Anton Bassov

This is because the OP simply took it out of context

Thanks for that.

You were more interested than I was in figuring out what was wrong. :wink:

OP… Seriously…

Peter

You were more interested than I was in figuring out what was wrong. :wink:

Well, a statement about a " driver processing only one IRP at a time" that was supposedly made in a source as authoritative as “Windows Internals” seemed so strange to me that I decided to investigate the OP’s claim a bit further…

Anton Bassov

There is a serialization guarantee… for some IRPs. Doron has the best write-up I’ve seen here: https://blogs.msdn.microsoft.com/doronh/2007/01/30/which-pnp-irps-are-state-changing/ and the follow-up https://blogs.msdn.microsoft.com/doronh/2007/02/07/which-pnp-and-power-irps-are-synchronized-against-each-other/

(Grab those pages quick, before the blogs website is shut down)

There is a serialization guarantee… for some IRPs.

PnP and Power are indeed special cases. Sure. An important point, especially given my full-throated and categorical denial of any serialization above.

Good addition, Mr, Tippet, as usual. Many thanks.

@“Peter_Viscarola_(OSR)” said:

There is a serialization guarantee… for some IRPs.

PnP and Power are indeed special cases. Sure. An important point, especially given my full-throated and categorical denial of any serialization above.

Good addition, Mr, Tippet, as usual. Many thanks.

Oh… re-reading the discussion more closely, I see that the OP was probably more interested in usermode I/O. And yeah, as you guys said, there’s a lot less automatic serialization built into vanilla WDM for READ/WRITE/IOCTL irps. But of course, if the OP wants an easy and flexible serialization story for those, WDF is just begging to solve these problems :wink:

But the different perspectives in this thread probably explain why the book seems to contradict itself. PNP + Power IRPs have a complex serialization story. Other types of IRPs are can be highly parallelizable. (Unless the driver writer does something to prevent that, like adding a queue or letting WDF do it for you.)

I have a low priority work item to figure out where to move the blog pages. The current tech community engagement administrators here does not want it.

d

Bent from my phone


From: Jeffrey_Tippet_[MSFT]
Sent: Thursday, January 31, 2019 4:18:09 PM
To: Doron Holan
Subject: Re: [NTDEV] Clarify what exactly is serialized about driver dispatch function calls.

OSR https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommunity.osr.com%2F&data=02|01|doron.holan%40microsoft.com|eb8e43f8038942c6758d08d687dabf45|72f988bf86f141af91ab2d7cd011db47|1|1|636845771392578360&sdata=1OgOcdqXHUBfIsApaDmebxF8SxOhA5SmBqeGUzNkh2s%3D&reserved=0
Jeffrey_Tippet_[MSFT] commented on Clarify what exactly is serialized about driver dispatch function calls.

> @Peter_Viscarola_(OSR) said:
>
> > There is a serialization guarantee… for some IRPs.
>
> PnP and Power are indeed special cases. Sure. An important point, especially given my full-throated and categorical denial of any serialization above.
>
> Good addition, Mr, Tippet, as usual. Many thanks.

Oh… re-reading the discussion more closely, I see that the OP was probably more interested in usermode I/O. And yeah, as you guys said, there’s a lot less automatic serialization built into vanilla WDM for READ/WRITE/IOCTL irps. But of course, if the OP wants an easy and flexible serialization story for those, WDF is just begging to solve these problems

But the different perspectives in this thread probably explain why the book seems to contradict itself. PNP + Power IRPs have a complex serialization story. Other types of IRPs are can be highly parallelizable. (Unless the driver writer does something to prevent that, like adding a queue or letting WDF do it for you.)

That’s because your company is pretending that they are a cloud only
company. So just intersperse ‘cloud’, ‘kubernetes’ and and ‘rest api’
randomly in your excellent blogs and it will all be fine.

Mark Roddy

Boy, what a storm I started!
Speaking as the OP, I have a simple-minded device that’s not PnP nor Power. I could have several Files open on the single Device. I figure that IRPs for a given File will be delivered in the same sequence they are issued, provided that the user is using synchronous I/O and all requests are issued from the same thread. That way a new IRP could not even be created until the previous one was completed.
I was concerned about IRPs from different Files being given to the driver concurrently, and I think the answer here is that indeed that is possible, and my driver has to contend with that.
In my driver, each File is associated exclusively with a single CPU, and the work for each IRP is mostly performed by a DPC targeted at that CPU. So this naturally serializes the work. If the user uses asynchronous I/O or makes calls from two different threads, that’s their problem, not mine; the device is not intended for that.
It also handles concurrent IRPs from different Files, as long as there’s no interaction between them.
BTW, the link above the comment
<Jeffrey_Tippet_[MSFT] commented on Clarify what exactly is serialized about driver dispatch function calls.
is kinda strange. I clicked on it and got to the OSR home page! Jeffrey, could you post that link yourself, because I’m interested in what it says.

@anton_bassov said:

[begin quote]

A start I/O routine. A driver can use a start I/O routine to initiate a data transfer to or from
a device. This routine is defined only in drivers that rely on the I/O manager to queue their
incoming I/O requests. The I/O manager serializes IRPs for a driver by ensuring that the driver
processes only one IRP at a time. Drivers can process multiple IRPs concurrently, but serializa-
tion is usually required for most devices because they cannot concurrently handle multiple I/O
requests.

[end quote]

I see that the two lines that I quoted only pertained to the use of start I/O routines, and does not apply in any other situations. Thanks for making that clear, Anton.

I clicked on it and got to the OSR home page!

That’s because it’s a link to the home page. Mr. Tippet was posting to the forum via email. If you look closely you’ll see that the link is from the email containing your question… so it is indeed a link to this discussion that appear in the original email that the forum sent to Mr. Tippet.

Peter