multiple spinlocks and KeLowerIrql()... really?!

I have trouble believing that we’re actually having this discussion in a group of what I assume are senior, experienced, kernel-mode developers.

OF COURSE it’s OK to release Spin Locks in an arbitrary order. As I say in class when I explain this (quoting Tony Mason, in fact) “Nobody has ever deadlocked as a result of a RELEASE operation.” The only constraint in Windows is the one that Mr. Roddy cited: You must keep the IRQLs straight. Duh.

OF COURSE it can be useful to do this. As I said previously, and as Mr. Roddy clarified, if you acquire Lock1 and then Lock2, and if Lock1 is HOT, and you need to do significantly more processing that requires just Lock2… you should absolutely release Lock1 before you release Lock2. However…

HOWEVER, as I also said previously and to the point Mr M M noted, this code tends to “look” wrong when you read it casually. So, there’s a maintenance cost involved. You have an obligation to “comment the shit out of this code” so that those who come after you understand what you’re doing.

So, in general, it’s easier to “go with the flow” and release locks in the inverse order in which they were acquired. In fact, the WDF Spin Lock functions require this.

And, let’s state for the record: There is absolutely no requirement to avoid the use of WDM function calls in a general-purpose KMDF driver. C’mon, people! How many of you call ExAllocatePool instead of WdfMemoryCreate? Get a block of device memory, and want to map it into kernel virtual address space? What do you call from your KMDF driver? You call MmMapIoSpace. There isn’t a WDF equivalent. In fact, one of the primary the ADVANTAGES of having access to WDM functions, is that the WDF team doesn’t have to create meaningless abstractions for every single WDM function that could be useful. Are we seriously supposed to live without reader/writer locks, and just use WDFSPINLOCK and WDFWAITLOCK? OF COURSE NOT.

Let’s first acknowledge that “strict alternation” is a specific algorithm in CS. And, frankly, I have no idea how it applies here. Spin Locks, by the very nature, achieve “strict alternation.”

Let’s further state that given the above interpretation of “strict alternation”, we could never have nested spin locks This is, of course, blatantly incorrect.

So, either our interpretation of “strict alternation” is incorrect, or the cited rule is incorrect. To me, it doesn’t really matter.

Finally, let me again state that if there’s anything in the docs, or in the static analysis tools, that requires you ONLY release spin locks in the inverse order in which they were acquired, that documentation or tool is WRONG. Plain and simple. This has never, not ever, been a requirement in Windows (because the idea of *requiring* the locks to be released in any given order is plainly silly).

Peter
OSR
@OSRDrivers

> -----Original Message-----

Sent: Wednesday, September 13, 2017 10:01 AM
To: Windows System Software Devs Interest List
> Subject: RE:[ntdev] multiple spinlocks and KeLowerIrql()… really?!

> Finally, let me again state that if there’s anything in the docs, or in
> the static analysis tools, that requires you ONLY release spin locks in
> the inverse order in which they were acquired, that documentation or tool
> is WRONG. Plain and simple. This has never, not ever, been a requirement
> in Windows (because the idea of requiring the locks to be released in
> any given order is plainly silly).

Since I made a statement that was pretty much to that effect, allow me to withdraw that statement. If I could strike it from the record …

In further pondering, it becomes clear (again, since I knew that long ago) that you can only deadlock if you acquire in the wrong order.

As stated by you, Tony Mason, and many others.

Phil
Not speaking for LogRhythm
Philip D Barila
Senior Software Engineer
720.881.5364 (W) ?
LogRhythm.com
?

?

?

?

Okay, I must have missed that part. I thought the OP was working to repair / merge old code and these locks were artifacts.

But it does not matter. This use case is invalid. If it is possible to do something useful while holding only lock 2, then it is possible to acquire it without holding lock 1 and the two locks can be completely separated.

That is obviously a very broad and sweeping statement for which I have provided no evidence, but think about it. What could possibly be safe to do while holding only lock 2 if it is always a direct subordinate of lock 1? Lock 2 would be a completely useless lock. To be useful therefore, the lock must be acquirable independently. If the lock is acquirable independently, then it will be possible to remove the convolution between the locks. This is true of any two locks in any programming model including Windows KM where the extra complications of IRQL are involved.

The is a very important pattern where lock 1 protects the data structure and while holding it one must ?acquire a reference? to something stored in it and this is a key pattern when removing the convolution between locks.

Again I have stated a thesis and provided only the briefest of justifications.

None of this has anything to do with KM programming specifically and it may not help the OP in the slightest.

Sent from Mailhttps: for Windows 10

From: Mark Roddymailto:xxxxx
Sent: September 12, 2017 10:25 PM
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: Re: RE:[ntdev] multiple spinlocks and KeLowerIrql()… really?!

The case was already discussed: resource1 is “hot” and its lock needs to be held to safely acquire the lock for resource2. Having acquired the lock for resource2 you want to drop the lock for resource1.

Mark Roddy

On Tue, Sep 12, 2017 at 6:55 PM, Marion Bond > > wrote:
Leave ?wrongness? aside for a moment and think about ?usefulness?. Perhaps you can think better than me, but when is it useful to release locks out of order?

I have done this in a few cases where the locking scheme was very complex and there was an interaction between the data structure that must be protected from corruption and the data element that is being accessed, but the general rule is still don?t do it ? if for no other reason that it will leave others very much confused and likely to break your code. Without a clear benefit of some kind, this is a good reason not to

Sent from Mailhttps: for Windows 10

From: Mark Roddymailto:xxxxx
Sent: September 12, 2017 6:04 PM
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: Re: RE:[ntdev] multiple spinlocks and KeLowerIrql()… really?!

There really is nothing wrong with unordered lock release as long as in NT you maintain the correct irql levels. Ordering is required on acquisition, not release. But if you can demonstrate an example of wrongness about unordered release I’d be happy to revise my view on this.

Mark Roddy

On Tue, Sep 12, 2017 at 4:49 PM, Marion Bond > > wrote:
While there are a few legitimate cases for releasing locks in an order that is different than the acquisition order, almost certainly this is the wrong approach

The use of a third lock solution, is equivalent to replacing lock 1 & lock 2 with lock 3 and might help you with a short term problem hos to fix broken code, but is not a solution to the multiple lock problem

If you can tell us more, we may be able to help you better, but otherwise the general rule is don?t do this.

Sent from Mailhttps: for Windows 10

From: Phil Barilamailto:xxxxx
Sent: September 12, 2017 11:46 AM
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: RE: RE:[ntdev] multiple spinlocks and KeLowerIrql()… really?!

> -----Original Message-----
> Sent: Tuesday, September 12, 2017 9:35 AM
> Subject: RE:[ntdev] multiple spinlocks and KeLowerIrql()… really?!
>
> thanks for your thoughts folks - I am playing with the Spinlock Three idea
> which is currently looking good… I figure that anything is better than
> directly manipulating IRQL… even juggling another spinlock!
>
> The reason for the release order is dictated by another part of the code -
> I am trying to avoid deadlock and excessive changes to existing code

If you don’t wrap every reference to locks 1 and/or 2 inside 3, you still run the risk of deadlock.

Either always release in reverse order, or just use one lock. Anything else is a partially loaded revolver.

Phil
Not speaking for LogRhythm
Philip D Barila
Senior Software Engineer
720.881.5364tel: (W)
LogRhythm.com


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:

— NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers! Details at To unsubscribe, visit the List Server section of OSR Online at


NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:

— NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers! Details at To unsubscribe, visit the List Server section of OSR Online at</http:></http:></http:></http:></http:></http:></http:></http:></http:></tel:></mailto:xxxxx></mailto:xxxxx></https:></mailto:xxxxx></mailto:xxxxx></https:></mailto:xxxxx></mailto:xxxxx></https:>

>And, let’s state for the record: There is absolutely no requirement to avoid

the use of WDM function calls in a general-purpose KMDF driver.

I think the important qualifier here is “general-purpose KMDF driver”. SDV does not support three-way hybrid drivers, so a KMDF-WDM hybrid, NDIS-KMDF hybrid or NDIS-WDM hybrid are all ok A NDIS-KMDF-WDM hybrid is not ok. Last year I was working on a complex virtual NDIS NIC miniport, which I was beginning to run WHQL tests on and made this not so well documented discovery.

Initially the driver used some KMDF and WDM functionality, but was mostly an NDIS driver. The KMDF functionality was used to interface to a parent virtual bus as a target device, and the unique WDM only functionality was to queue a DPC on a specific processor. KDMF has no way to queue a directed DPC, so I had to remove the KMDF calls to get SDV to accept the WDM calls.

It seems possible (but no direct experience) this limitation may also apply to other classes of drivers with miniport/frameworks, like a stoport-KMDF driver possibly can’t be a storport-KMDF-WDM hybrid.

So, my direct experience is that if you had a NDIS-KMDF driver, you could not just stir in some WDM calls to fill the gaps in KMDF. A NDIS-KMDF-WDM driver might work just fine, but it will be impossible to pass certification tests, due to limitations in SDV.

Jan

Absolutely. That’s why I wrote it that way :wink:

Absolutely correct, Mr. Bottorff. You’re correct in pointing this out.

Let’s be clear here, though. This isn’t a KMDF issue. It’s an NDIS issue. You can’t call arbitrary kernel functions in ANY NDIS driver (and pass the HLK tests), whether it’s got a KMDF portion or not.

Of course, the vast majority of KMDF drivers are not in this position. They’re *just* KMDF drivers.

Peter
OSR
@OSRDrivers

What about a layered miniport driver that exposes NDIS interface on its upper edge and deals
with WDM driver of the ACTUAL hardware device on its lower one by sending IRPs to it? What about a miniport driver that registers its control device with NdisRegisterDeviceEx() and gets a pointer to a “regular” WDM DEVICE_OBJECT that has to be accessed in “a regular” WDM-style fashion? Are they going to fail HLK test as well just because of making WDM calls in NDIS driver?

Anton Bassov

HLK only fails limited number of imports (in NDISTEST 6.0). A lot of imports, including spinlock and IRQL functions are OK. Because these are what NDIS macros expand to.

And that line of thinking is on its way out. If you’ve looked at the NetAdapter preview, you’ve seen that NDIS miniports will be moving to regular KMDF, with no “forbidden” APIs.

Even for classic NDIS 6 miniport drivers, we are not hard-liners on calling out into WDM APIs. The blocklist of imported APIs is mostly there to prevent you from shooting yourself in the foot (e.g., calling IoCreateDevice, when NDIS has overridden your dispatch table). If you are in a situation where NDISTest is being unreasonable about an imported WDM API, send us a mail.