Are callbacks guaranteed to run consecutively?

Hello,

callbacks registered with ExRegisterCallback and called via ExNotifyCallback are currently processed one after another.
I’d like to know if it’s safe to assume that this behaviour won’t change in a future version (i.e., parallel execution), because otherwise I’d have to guard against race conditions (multiple callback routines access the same resource).

The documentation isn’t very specific, but on the other hand, this hasn’t changed in almost a decade.

Thanks.

Hmm, I’m quite sure it won’t change, because callback routines can run at DISPATCH_LEVEL, so threading seems unlikely.

Also note that ExNotifyCallback can be called on the same notification callback chain on 2 different threads at the same time, so if you do not control who is making the call to ExNotifyCallback, it does not really matter if the calls are serialized per thread b/c 2 thread could be making a call into your driver’s callback at the same time.

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@hushmail.com
Sent: Friday, December 21, 2007 5:19 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Are callbacks guaranteed to run consecutively?

Hello,

callbacks registered with ExRegisterCallback and called via ExNotifyCallback are currently processed one after another.
I’d like to know if it’s safe to assume that this behaviour won’t change in a future version (i.e., parallel execution), because otherwise I’d have to guard against race conditions (multiple callback routines access the same resource).

The documentation isn’t very specific, but on the other hand, this hasn’t changed in almost a decade.

Thanks.


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

> Hmm, I’m quite sure it won’t change, because callback routines can run

at DISPATCH_LEVEL, so threading seems unlikely.

The only thing DISPATCH_LEVEL implies is that context switch cannot occur * on a given CPU*.
However, where is the guarantee that the same piece of code is not going on run on another CPU at the same time??? Therefore, the above assumption is really naive, unless you find a MSDN statement saying that callback objects are protected by some kind of “callback spinlock” the way ISRs are protected by interrupt spinlock…

Anton Bassov

They are not protected by a lock, but there is an api contract that all registered callbacks have been invoked and returned before the call to ExNotifyCallback returns to the caller. So while you could spread the callbackss out across processors, doing so at dispatch level and then waiting for all them to return so that the API can return is somewhat hard to do at dispatch level given that you are not allowed to synchronously wait for any period of time.

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@hotmail.com
Sent: Friday, December 21, 2007 12:02 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Are callbacks guaranteed to run consecutively?

Hmm, I’m quite sure it won’t change, because callback routines can run
at DISPATCH_LEVEL, so threading seems unlikely.

The only thing DISPATCH_LEVEL implies is that context switch cannot occur * on a given CPU*.
However, where is the guarantee that the same piece of code is not going on run on another CPU at the same time??? Therefore, the above assumption is really naive, unless you find a MSDN statement saying that callback objects are protected by some kind of “callback spinlock” the way ISRs are protected by interrupt spinlock…

Anton Bassov


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Doron Holan wrote:

Also note that ExNotifyCallback can be called on the same notification callback
chain on 2 different threads at the same time, so if you do not control who is
making the call to ExNotifyCallback, it does not really matter if the calls are
serialized per thread b/c 2 thread could be making a call into your driver’s
callback at the same time.

Thanks for the reply, but this doesn’t apply in my case.
While the callback chain could get notified multiple times concurrently, each one would pass different arguments, so no race condition in my case.

I actually want to implement some kind of first-come, first-served mechanism (right term?), where an arbitrary number of callback routines can get registered, but only one of them (and I don’t care which one) would service the notify.

Unless the callback routines negotiate amongst themselves or one of the parameters indicates who to process the notify I don’t see how ExNotifyCallback will work to process only one callback if more than one is registered.

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@hushmail.com
Sent: Friday, December 21, 2007 1:16 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Are callbacks guaranteed to run consecutively?

Doron Holan wrote:

Also note that ExNotifyCallback can be called on the same notification callback
chain on 2 different threads at the same time, so if you do not control who is
making the call to ExNotifyCallback, it does not really matter if the calls are
serialized per thread b/c 2 thread could be making a call into your driver’s
callback at the same time.

Thanks for the reply, but this doesn’t apply in my case.
While the callback chain could get notified multiple times concurrently, each one would pass different arguments, so no race condition in my case.

I actually want to implement some kind of first-come, first-served mechanism (right term?), where an arbitrary number of callback routines can get registered, but only one of them (and I don’t care which one) would service the notify.


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Doron,

They are not protected by a lock, but there is an api contract that all registered
callbacks have been invoked and returned before the call to ExNotifyCallback
returns to the caller.

Consider the scenario when threads X and Y that run at DPC level on CPUs respectively A and B call ExNotifyCallback() independently from one another, and specify the same callback object and context parameters on their calls. You cannot really resolve this situation without a spinlock, can you??? Therefore, if individual callback objects are not guarded by spinlocks, then there must be some global spinlock that guards all callback objects in the system (which is, apparently, far from being the best approach one can possibly imagine)…

Anton Bassov

xxxxx@hotmail.com wrote:

Consider the scenario when threads X and Y that run at DPC level on CPUs respectively A and B call ExNotifyCallback() independently from one another, and specify the same callback object and context parameters on their calls. You cannot really resolve this situation without a spinlock, can you??? Therefore, if individual callback objects are not guarded by spinlocks, then there must be some global spinlock that guards all callback objects in the system (which is, apparently, far from being the best approach one can possibly imagine)…

I’m not sure I understand your objection. What you describe is a usage
problem, not an API design flaw. The callback objects don’t need to be
protected by spinlock, although it’s quite possible that the routine
being called might have to use one to protect shared data.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Yes, there is an internal lock local to the callback object but the callbacks are not necessarily called with lock held. The lock is there to maintain state and list integrity

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@hotmail.com
Sent: Friday, December 21, 2007 3:17 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Are callbacks guaranteed to run consecutively?

Doron,

They are not protected by a lock, but there is an api contract that all registered
callbacks have been invoked and returned before the call to ExNotifyCallback
returns to the caller.

Consider the scenario when threads X and Y that run at DPC level on CPUs respectively A and B call ExNotifyCallback() independently from one another, and specify the same callback object and context parameters on their calls. You cannot really resolve this situation without a spinlock, can you??? Therefore, if individual callback objects are not guarded by spinlocks, then there must be some global spinlock that guards all callback objects in the system (which is, apparently, far from being the best approach one can possibly imagine)…

Anton Bassov


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Tim,

I’m not sure I understand your objection. What you describe is a usage problem,
not an API design flaw. The callback objects don’t need to be protected by spinlock,
although it’s quite possible that the routine being called might have to use one to
protect shared data.

Please read the whole thread again carefully - the question that we discuss here is whether it is
routine’s responsibility to provide synchronization mechanism or whether the OS takes care of all synchronization issues itself . According to Doron, the latter is the case. At the same time, he says that there is no spinlock associated with any given callback object. Therefore, what I am saying here is that, if both Doron’s statements are correct, then there must be some global spinlock that guards all callback objects in the system ( because ExNotifyCallback() may be called at IRAL0==DPC level), which hardly seems to be the most efficient solution…

Anton Bassov

Doron,

there is an internal lock local to the callback object but the callbacks are not
necessarily called with lock held. The lock is there to maintain state and list integrity

Please thing about the above statement carefully…

Conditional acquisition of a spinlock that guards unconditional operation just defeats the very purpose of using spinlocks, don’t you think??? Therefore, either the above statement or the one about guaranteed synchronization that API provides is wrong - these 2 statements are just mutually exclusive, so that they just cannot be both right, no matter how you look at it…

Anton Bassov

You are making a mountain out of a mole hill here

o each ex callback object has its own lock to maintain the list of callbacks registered on it
o the callbacks may or not may not be called with the list lock being held (this is so calls made to ExNotifyCallback at passive invoke the callback at passive
o the list lock is there to maintain the state of each registered callback as well as the overall callback object
o the lock is not used to serialize multiple notifications on the same callback object from different contexts or IRQLs. This means that a thread which calls ExNotifyCallback at passive could be interrupted at dispatch level on the same thread, make a call to ExNotifyCallback with the same object and the registered callback chain will be called at dispatch in the middle of making the calls at passive.

My earlier point was that if there were such a design in the callback object to spread the callbacks across different threads (processors more likely), it would be very difficult to do this at dispatch level b/c there is no way at dispatch to wait for the farmed out requests to other processors to come back and still follow the waiting rules that dispatch level dictates.

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@hotmail.com
Sent: Friday, December 21, 2007 6:45 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Are callbacks guaranteed to run consecutively?

Doron,

there is an internal lock local to the callback object but the callbacks are not
necessarily called with lock held. The lock is there to maintain state and list integrity

Please thing about the above statement carefully…

Conditional acquisition of a spinlock that guards unconditional operation just defeats the very purpose of using spinlocks, don’t you think??? Therefore, either the above statement or the one about guaranteed synchronization that API provides is wrong - these 2 statements are just mutually exclusive, so that they just cannot be both right, no matter how you look at it…

Anton Bassov


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

If callbacks would be called with the lock held - then this would be
documented to not call ExNotifyCallback from another callback, since this would
be a deadlock.


Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

wrote in message news:xxxxx@ntdev…
> Hello,
>
> callbacks registered with ExRegisterCallback and called via ExNotifyCallback
are currently processed one after another.
> I’d like to know if it’s safe to assume that this behaviour won’t change in a
future version (i.e., parallel execution), because otherwise I’d have to guard
against race conditions (multiple callback routines access the same resource).
>
> The documentation isn’t very specific, but on the other hand, this hasn’t
changed in almost a decade.
>
> Thanks.
>

Doron,

You are making a mountain out of a mole hill here

Not really - the only thing I am doing here is trying to figure out how it all works internally. In my opinion, there is some inconsistency in MSDN documentation - on one hand, it says that
ExNotifyCallback() provides synchronization, but, one another hand, it says that callback routine is invoked at caller’s IRQL. Taking into consideration that ExNotifyCallback() is callable at IRQL<= DISPATCH_LEVEL, these 2 statements seem to be mutually exclusive for SMP system
(although perfectly consistent with one another if we speak about UP system)

each ex callback object has its own lock to maintain the list of callbacks registered on it

This is what we should have started with. However, originally you said that there is no spinlock to guard every *given* callback object. Furthermore, another question is still open. More on it below

the callbacks may or not may not be called with the list lock being held
(this is so calls made to ExNotifyCallback at passive invoke the callback at passive

Let’s say ExNotifyCallback() is called by thread A on CPU X at PASSIVE_LEVEL. If thread A does not obtain a spinlock before invoking callbacks, what is going to happen if thread B that runs on CPU Y at DISPATCH_LEVEL calls ExNotifyCallback() with the same parameters thread A did??? Therefore, it looks like MSDN statement about synchronization that ExNotifyCallback() provides is not really true…

Anton Bassov

> If callbacks would be called with the lock held - then this would be documented

to not call ExNotifyCallback from another callback, since this would be a deadlock.

Sure - this is exactly what would happen if there was a single spinlock for all callbacks in the system. However, as it follows from Doron latest post, there is a separate spinlock for every given callback object, although originally he said there was none…

Anton Bassov

Anton,

The basics of the callback are that thread calling ExNotifyCallback
walks the list calling each callback. Yes that is synchronization since you
can create a contract based on this, is it a general mutex like sync, NO.
By the way the PsXXXNotify callbacks are just ExNotifyCallbacks, the Ps
subsystem adds code to limit them to 8.


Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr
Remove StopSpam to reply

wrote in message news:xxxxx@ntdev…
>
> Doron,
>
>> You are making a mountain out of a mole hill here
>
> Not really - the only thing I am doing here is trying to figure out how it
> all works internally. In my opinion, there is some inconsistency in MSDN
> documentation - on one hand, it says that
> ExNotifyCallback() provides synchronization, but, one another hand, it
> says that callback routine is invoked at caller’s IRQL. Taking into
> consideration that ExNotifyCallback() is callable at IRQL<=
> DISPATCH_LEVEL, these 2 statements seem to be mutually exclusive for SMP
> system
> (although perfectly consistent with one another if we speak about UP
> system)
>
>> each ex callback object has its own lock to maintain the list of
>> callbacks registered on it
>
> This is what we should have started with. However, originally you said
> that there is no spinlock to guard every given callback object.
> Furthermore, another question is still open. More on it below
>
> > the callbacks may or not may not be called with the list lock being held
>> (this is so calls made to ExNotifyCallback at passive invoke the callback
>> at passive
>
> Let’s say ExNotifyCallback() is called by thread A on CPU X at
> PASSIVE_LEVEL. If thread A does not obtain a spinlock before invoking
> callbacks, what is going to happen if thread B that runs on CPU Y at
> DISPATCH_LEVEL calls ExNotifyCallback() with the same parameters thread A
> did??? Therefore, it looks like MSDN statement about synchronization that
> ExNotifyCallback() provides is not really true…
>
>
> Anton Bassov
>
>
>

I never said there was no lock in the callback object, I said the lock was not necessarily held across registered callbacks on a the callback object. The synchronization that the callback object is against the unregistration of a callback while a notification is in progress and calling the callback. The callback object does not serialize multiple notifications made on it at the same time.

Let’s say ExNotifyCallback() is called by thread A on CPU X at PASSIVE_LEVEL. If thread A does not obtain a
spinlock before invoking callbacks, what is going to happen if thread B that runs on CPU Y at DISPATCH_LEVEL
calls ExNotifyCallback() with the same parameters thread A did??? Therefore, it looks like MSDN statement
about synchronization that ExNotifyCallback() provides is not really true…

Given what I said above, they are not serialized against each other. Let’s say that thread A on CPU1 calls ExNotifyCallback and has called (or is about to call) callback1. Then thread B on CPU2 unregisters callback1. The unregister call will not return until callback1 returns and the callback object has processed that return.

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@hotmail.com
Sent: Saturday, December 22, 2007 9:16 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Are callbacks guaranteed to run consecutively?

Doron,

You are making a mountain out of a mole hill here

Not really - the only thing I am doing here is trying to figure out how it all works internally. In my opinion, there is some inconsistency in MSDN documentation - on one hand, it says that
ExNotifyCallback() provides synchronization, but, one another hand, it says that callback routine is invoked at caller’s IRQL. Taking into consideration that ExNotifyCallback() is callable at IRQL<= DISPATCH_LEVEL, these 2 statements seem to be mutually exclusive for SMP system
(although perfectly consistent with one another if we speak about UP system)

each ex callback object has its own lock to maintain the list of callbacks registered on it

This is what we should have started with. However, originally you said that there is no spinlock to guard every *given* callback object. Furthermore, another question is still open. More on it below

the callbacks may or not may not be called with the list lock being held
(this is so calls made to ExNotifyCallback at passive invoke the callback at passive

Let’s say ExNotifyCallback() is called by thread A on CPU X at PASSIVE_LEVEL. If thread A does not obtain a spinlock before invoking callbacks, what is going to happen if thread B that runs on CPU Y at DISPATCH_LEVEL calls ExNotifyCallback() with the same parameters thread A did??? Therefore, it looks like MSDN statement about synchronization that ExNotifyCallback() provides is not really true…

Anton Bassov


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

True, and it should be called out in the docs. But wouldn’t calling the callback chain from within the callback chain be bad design to begin with? You would need a way to back off and not make another call in ExNotifyCallback when your callback is invoked a 2nd (or 3rd or 4th) time. You basically described recursion in another form and that is not a good design choice in the kernel.

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S. Shatskih
Sent: Saturday, December 22, 2007 7:32 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Are callbacks guaranteed to run consecutively?

If callbacks would be called with the lock held - then this would be
documented to not call ExNotifyCallback from another callback, since this would
be a deadlock.


Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

wrote in message news:xxxxx@ntdev…
> Hello,
>
> callbacks registered with ExRegisterCallback and called via ExNotifyCallback
are currently processed one after another.
> I’d like to know if it’s safe to assume that this behaviour won’t change in a
future version (i.e., parallel execution), because otherwise I’d have to guard
against race conditions (multiple callback routines access the same resource).
>
> The documentation isn’t very specific, but on the other hand, this hasn’t
changed in almost a decade.
>
> Thanks.
>


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Correct, but, if some functions in the 3rd party code are called by the OS
with the lock acquired, it is a good idea to document this (at least the
IRQL).


Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

“Doron Holan” wrote in message news:xxxxx@ntdev…
True, and it should be called out in the docs. But wouldn’t calling the
callback chain from within the callback chain be bad design to begin with? You
would need a way to back off and not make another call in ExNotifyCallback when
your callback is invoked a 2nd (or 3rd or 4th) time. You basically described
recursion in another form and that is not a good design choice in the kernel.

d

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Maxim S. Shatskih
Sent: Saturday, December 22, 2007 7:32 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Are callbacks guaranteed to run consecutively?

If callbacks would be called with the lock held - then this would be
documented to not call ExNotifyCallback from another callback, since this would
be a deadlock.


Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

wrote in message news:xxxxx@ntdev…
> Hello,
>
> callbacks registered with ExRegisterCallback and called via ExNotifyCallback
are currently processed one after another.
> I’d like to know if it’s safe to assume that this behaviour won’t change in a
future version (i.e., parallel execution), because otherwise I’d have to guard
against race conditions (multiple callback routines access the same resource).
>
> The documentation isn’t very specific, but on the other hand, this hasn’t
changed in almost a decade.
>
> Thanks.
>


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer