Using spin locks for serialization

So I’ve been given this code for logging to a file (yes WPP is a better option) and in order to ensure that everything is in order, spin locks are being used for serialization. I know this is wrong, as running elevated at DISPATCH_LEVEL prevents DPCs from running as well as degrades system performance.

I’m dealing with some real stubbornness here, so I’m looking for compelling arguments of why spin locks shouldn’t be used in this way. If I’m wrong, I’d like to learn that as well.

You CANNOT call ZwWriteFile while holding a spinlock.

If you open the file with APPEND access, all writes will be serialized even across separately open handles.

If you open the file for synchronous write, all writes on this handle will be serialized implicitly.

Alex,

Yes, I know that ZwWriteFile must be called at dispatch level. What is happening is that a buffer is prepared and inserted in a queue to be processed by a system thread to minimize performance impact.

The spinlock is being held when processing this buffer. My point is, spinlocks are to be used to protect shared memory and a thread shouldn’t elevate itself with a spinlock for such a long period of time. But this is falling on deaf ears.

Turn on driver verifier on the driver, when the verifier blue screens the
system because you ran at dispatch for too long, you will have a valid
argument. Note: when reviewing 3rd party drivers for clients the first
thing I tell the client is run with the driver verifier, if the driver fails
that forget about the supplier.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@gmail.com
Sent: Tuesday, October 28, 2014 8:36 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Using spin locks for serialization

Alex,

Yes, I know that ZwWriteFile must be called at dispatch level. What is
happening is that a buffer is prepared and inserted in a queue to be
processed by a system thread to minimize performance impact.

The spinlock is being held when processing this buffer. My point is,
spinlocks are to be used to protect shared memory and a thread shouldn’t
elevate itself with a spinlock for such a long period of time. But this is
falling on deaf ears.


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Verifier is being run. Thanks Don.

Wait. A spin lock might be a perfect solution.

It depends on how long you’re holding the lock, and how much contention there is on the lock.

Remember… String moves are fast. Don’t over think this. Beware of premature optimization.

Hold the lock only whe you need it, and be done with it.

Peter
OSR
@OSRDrivers

Peter,

When you say “Hold the lock only when you need it, and be done with it”, do you mean to synchronize access to shared memory? Or as a means to serialize logging?

The problem is the spinlock is released to call some Rtl function that has to be called at PASSIVE_LEVEL which theoretically would allow thread 2 on proc 2 to come in and grab the spin lock and proceed itself, until it has to release it to call that Rtl function. Let’s say that thread 1 is running on proc 1, and the processor gets interrupted and starts running at DIRQL, then when all things are said and done, log statement 2 winds up before long statement 1. Now maybe I am being pedantic, but I’ve worked in sustaining engineering and race conditions are tough to deal with.

The interesting thing, the previous version of this code had a means for catching if any out of order situations occurred. But that was cut.

You should give more details. What is the lock actually protecting ? A shared buffer ? A list ?

A spin lock is a good choice if you use ExInterlockedXxx functions (atomic insertion/removal of entries) to manage a list of buffers. Each logger should allocate a BUFFER_ENTRY and the buffer itself, fill that buffer and then use ExInterlockedPushEntryList to insert the buffer entry in the list.

You should have a worker thread that would periodically remove (pop) entries, writes buffers to disk, free buffers and entries and then sleep a certain amount of time. Entries can be popped in a loop.

==================
while(BufferEntry = ExInterlockedPopEntryList(&ListHead,&SpinLock)){
// Proceed with this buffer entry
}
// No more entries so relax here
KeDelayExecutionThread(…);

A system thread runs at passive level with normal kernel APCs disabled but special kernel APCs enabled. So you can use ZwWriteFile within such a thread’s routine. Look at the documentation.

If you are dropping the lock to go back to passive that meant you started at passive. If all log call sites are at passive use a KEVENT as your lock and then you don’t have the irwl problem

d

Bent from my phone


From: xxxxx@hotmail.commailto:xxxxx
Sent: ?10/?28/?2014 10:13 PM
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: RE:[ntdev] Using spin locks for serialization

You should give more details. What is the lock actually protecting ? A shared buffer ? A list ?

A spin lock is a good choice if you use ExInterlockedXxx functions (atomic insertion/removal of entries) to manage a list of buffers. Each logger should allocate a BUFFER_ENTRY and the buffer itself, fill that buffer and then use ExInterlockedPushEntryList to insert the buffer entry in the list.

You should have a worker thread that would periodically remove (pop) entries, writes buffers to disk, free buffers and entries and then sleep a certain amount of time. Entries can be popped in a loop.

==================
while(BufferEntry = ExInterlockedPopEntryList(&ListHead,&SpinLock)){
// Proceed with this buffer entry
}
// No more entries so relax here
KeDelayExecutionThread(…);
==================

A system thread runs at passive level with normal kernel APCs disabled but special kernel APCs enabled. So you can use ZwWriteFile within such a thread’s routine. Look at the documentation.


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx></mailto:xxxxx>

> The spinlock is being held when processing this buffer.

No need.

Spinlocks are only required to protect the queue, not the _buffer data. So, after detaching the buffer from the queue, release the spinlock and go on processing it without spinlocks held.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

You’re missing the OP’s point… which I grant you is pretty darn difficult to determine.

OP: You’re attempting to serialize the ordering of log entries across multiple processors, right? So that the log entry for event X appears in the log prior to the log entry for event X+1 ? Is that correct?

And the functions that call your log package all run at IRQL PASSIVE_LEVEL?

I suspect you’re out-thinking yourself on this one.

IF that’s what you’re saying, there is no way to guarantee that the logging will be strictly in order. You can surely see this yourself: Suppose the code running on Thread1 calls your log function, but before the function can run, that thread is pre-empted… now code running on Thread2 runs, calls your log function, finishes, and only THEN does Thread1 resume running. Result, the event Thread2’s info is logged before Thead1’s… out of order.

There is *really* no way to prevent this.

The best you’re going to be able to do is “pretty much in order” no matter what you do… except…

If you want to bump *all* the processing up to IRQL DISPATCH_LEVEL, including the functions that call the logger, THEN you can be sure that items will be logged in-order… and THEN you have to serialize the queue using a Spin Lock as Mr. Shaktskih suggests.

If you’re willing to accept “pretty much in order”, what you do is tag the log queue entry IMMEDIATELY on entry with something that implies order. You this either by calling InterlockedIncrement immediately to get a strictly increasing “queue entry number” value, or reserving a slot in a queue for your log information. In this way, your log entries will be guaranteed to be in the correct order… as defined by the order in which the logging function allocates their tags.

THEN you can insert them in some sort of list, using whatever locking primitive you want (like a spin lock).

I dunno… maybe I have your use case entirely wrong. In which case I need to read more closely or drink more coffee…

BTW, you should know that WPP messages are *not* guaranteed to be in order (or, at least they weren’t back when I was working with the guy who invented it). So, it’s not like this is a solvable problem.

Peter
OSR
@OSRDrivers

Interesting. My response didn’t make it through.

Doron. Thanks. The answer was right in front of my nose as you pointed out.

Peter,

I agree with you. With regards to the tag, that was in there, and then removed.

Anyways. Thanks to all, you’ve been helpful.