Why its bad?

Hi All,
I just want to ask (probably) a basic question. If we look at differences in WDM and Linux driver architectures, WDM is more strict as what we can do at different IRQL levels enforcing a driver writer to design the driver properly or else get in trouble later. In Linux the driver writing is more flexible.

Let’s take for example a simple case where:-

Case: Application sends an IOCTL to do some data transfer to a hardware.

In Windows:

  1. The command is put in some queue and fired to the hardware.
  2. When the transfer completes, the ISR notifies the command completion.
  3. A DPC is queued with correct information about what to process.
  4. From the DPC the command is completed to the application.

In Linux [It is possible and OK to do the following] :-

  1. An Event object is created and put to non-signaled state, the command is fired and we wait on the event.
  2. In ISR we set the event, to unblock the user more IOCTL thread. We complete the command from 1 itself.

Thus in Linux the code becomes very simple, but I myself like the WDM approach.

So my question is that why are we not allowed to signal from ISR context in WDM. We are just setting an event after all. I am sure that there is a perfect explanation for this and that is what I am looking for here.

Thanks,
Aj

Good question. The first thing that would come to mind is because setting an
event required taking the dispatcher lock. However considering that the
dispatcher lock raises to SYNCH_LEVEL which is higher than DIRQLs on MP
systems, they would only need to redefine SYNCH_LEVEL on UP kernels (which
is equal to DISPATCH_LEVEL so that would deadlock). I read that SYNCH_LEVEL
was never intended to be always above all DIRQLs but the question is also
why is that.

Perhaps they do have a good reason, though it would be pretty convenient to
be able just to set an event right from an ISR, in particular if the DPC
serves no other purpose than just setting the event.

//Daniel

“Ajitabh Saxena” wrote in message
>news:xxxxx@ntdev…
>So my question is that why are we not allowed to signal from ISR context in
>WDM.
>We are just setting an event after all. I am sure that there is a perfect
>explanation for this and that is what I am looking for here.

> I am sure that there is a perfect explanation for this and that is what I am looking for here.

The “perfect explanations” is that, once LInux does not have the concept of IRQL, there is no such thing as interrupt spinlock under Linux. Instead, there are “regular” spinlocks and “bottom-half ones” that run with respectively both hardware/software and software-only interrupts disabled. It is (hopefully) understandable that ISRs may use only “regular” spinlocks. Once dispatcher objects like tasks,queues,events, etc are protected by “regular” spinlocks that require the owner to run with interrupts disabled, they are available for use by ISRs.

Simple explanation,don’t you think…

Anton Bassov

> However considering that the dispatcher lock raises to SYNCH_LEVEL which

Well, apparently, setting an event involves some pre/post steps that don’t require a dispatcher lock,and, hence,
it is more reasonable to do them at DPC level…

Anton Bassov

>Well, apparently, setting an event involves some pre/post steps that don’t

require
a dispatcher lock,and, hence, it is more reasonable to do them at DPC
level…

Well, not in the WRK (which is on display everywhere). First it checks if
the event is not already set and then immediately takes the dispatcher lock
for the entire function body.

//Daniel

A dispatcher is declared to run on DISPATCH_LEVEL. An ISR runs on higher IRQL. Thus, an ISR can interrupt the dispatcher. Since KeSetEvent calls the dispatcher, it would be VERY BAD IDEA to do that from an ISR.

>A dispatcher is declared to run on DISPATCH_LEVEL. An ISR runs on higher

IRQL.

The dispatcher lock actually raises to SYNCH_LEVEL which runs higher than
any ISR.

SYNCH_LEVEL matches DISPATCH_LEVEL only on uniprocessor kernels and only
since they deliberately redefined that, as you can read in Hector’s memo:
http://www.osronline.com/article.cfm?id=146

//Daniel

>The dispatcher lock actually raises to SYNCH_LEVEL which runs higher than any ISR.

…and this probably doesn’t hold true anymore since Windows 7…

>>The dispatcher lock actually raises to SYNCH_LEVEL which runs higher than

>any ISR.

…and this probably doesn’t hold true anymore since Windows 7…

Probably. Though it makes one wonder what were the original architectural
reasons for this restriction. Perhaps to keep the contract more simple, or
something else we wouldn’t think of.

//Daniel

>>>> A dispatcher is declared to run on DISPATCH_LEVEL.
Understand.

>> An ISR runs on higher IRQL
Understand.

>>> Thus, an ISR can interrupt the dispatcher.
Understand.

>>> Since KeSetEvent calls the dispatcher,

Don’t Understand… why would KeSetEvent call the dispatcher. The documentation says :-

“Calling KeSetEvent causes the event to attain a signaled state. If the event is a notification event, the system attempts to satisfy as many waits as possible on the event object. The event remains signaled until a call to KeClearEvent or KeResetEvent clears it. If the event is a synchronization event, one wait is satisfied before the system automatically clears the event.”

What I want to understand is that “Satisfy as many wait as possible” mean here. Does that mean that it actually tries to run threads waiting on this signaled object??

>>>>it would be VERY BAD IDEA to do that from an ISR.
Absolutely agree.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@broadcom.com
Sent: Wednesday, March 26, 2014 8:08 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Why its bad?

A dispatcher is declared to run on DISPATCH_LEVEL. An ISR runs on higher IRQL. Thus, an ISR can interrupt the dispatcher. Since KeSetEvent calls the dispatcher, it would be VERY BAD IDEA to do that from an ISR.


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Yes.

It runs the list of wait blocks and updates the state of threads that are waiting on the dispatcher object that is now signaled.

Peter
OSR
@OSRDrivers

>>>>> A dispatcher is declared to run on DISPATCH_LEVEL.

Understand.

>>> An ISR runs on higher IRQL
Understand.

>>>> Thus, an ISR can interrupt the dispatcher.
Understand.

>>>> Since KeSetEvent calls the dispatcher,

Don’t Understand… why would KeSetEvent call the dispatcher. The
documentation says :-

“Calling KeSetEvent causes the event to attain a signaled state. If the
event is a notification event, the system attempts to satisfy as many
waits as possible on the event object. The event remains signaled until a
call to KeClearEvent or KeResetEvent clears it. If the event is a
synchronization event, one wait is satisfied before the system
automatically clears the event.”

What I want to understand is that “Satisfy as many wait as possible” mean
here. Does that mean that it actually tries to run threads waiting on this
signaled object??

It means that it iterates through the list of waiting threads, and calls
the dispatcher to dispatch each of the threads. Note that a thread
receives a priority boost when being released from an Event wait. The
dispatcher then uses its judgment to decide whether or not the thread
should actually run. It will run as many as it can, but the remainder
will remain suspended, but runnable, and as soon as sufficient resources
come available, each of those will be dispatched as well. Simple case: I
have 20 threads waiting on an Event in an 8-core machine. At most, 8 of
those threads will be able to run. However, even with the priority boost,
some of them may still be lower priority than, say, a kernel thread
running on that core, and therefore be unable to preempt that thread.
Furthermore, because there were 20 threads released, but only 8 cores,
even if every one of them can run immediately (all cores idle or
preemptible), there are still 12 threads left. These are runnable, and
only need CPU/Memory/Priority to run. [Of course, no thread runs until
the CPU on which the thread is scheduled drops to PASSIVE_LEVEL] It is the
“calls the dispatcher” part that makes this uncallable at >
DISPATCH_LEVEL.

So it doesn’t “try to run” anything; it merely marks the threads as
runnable, with a certain priority, and leaves their fate to this
dispatcher. It chooses, in an unspecified manner, which threads will run
as soon as the appropriate core goes to PASSIVE_LEVEL, and holds the rest
in the runnable queue.

The real truth is far more complex. If you have Windows Internals, 6th
Edition,Part 1 you might start around page 184,
joe

>>>>>it would be VERY BAD IDEA to do that from an ISR.
Absolutely agree.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@broadcom.com
Sent: Wednesday, March 26, 2014 8:08 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Why its bad?

A dispatcher is declared to run on DISPATCH_LEVEL. An ISR runs on higher
IRQL. Thus, an ISR can interrupt the dispatcher. Since KeSetEvent calls
the dispatcher, it would be VERY BAD IDEA to do that from an ISR.


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

> should actually run. It will run as many as it can, but the remainder

will remain suspended

Not “suspended”, but “runnable on the queue”, incrementing the Processor Queue perf counter.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

> Though it makes one wonder what were the original architectural reasons for this restriction.

Perhaps to keep the contract more simple, or something else we wouldn’t think of.

AFAIK, “the original architectural reasons” for this pre - date NT by almost two decades, and are specific to PDP-11 platform. This is where the concepts of IRQL and splx() are rooted. These concepts lost their practical meaning when OSes were ported to other platforms This is why I put your statement in the quotation marks - there are no objective reasons for these restrictions since the very first day of NT, and there were no objective reasons for them at the time NT was conceived either. They made their way into NT simply for the historic reasons …

Anton Bassov

Weeeeelllll… SORT of. That’s probably where Dave’s thinking came from, for sure, though by the time he was working on NT it was more than 20 years after he had stopped working on RSX-11M (his PDP-11 operating system).

RSX-11M had a sort of spin lock, but I’ll be damned if I can remember what it was used for. I can’t recall ever acquiring it, or even teaching people about it… but that was a long, long, time ago. It is possible there wasn’t even such a concept prior to the PDP-11/74, which was the Symmetric Multiprocessor PDP-11 that never (really commercially) shipped.

The concept of IRQL? Weeeeeellll… RSX-11M had user-state, system-state, and a small number of interrupt states. These are SORT of equivalent to IRQL.

Tangential to the topic is the fact that the DPC List is a direct descendant of the RSX-11M Fork List… exact same concept.

Peter
OSR
@OSRDrivers

> [quote]

AFAIK, “the original architectural reasons” for this pre - date NT by
almost
two decades, and are specific to PDP-11 platform. This is where the
concepts of
IRQL and splx() are rooted.
[/quote]

Weeeeelllll… SORT of. That’s probably where Dave’s thinking came from,
for sure, though by the time he was working on NT it was more than 20
years after he had stopped working on RSX-11M (his PDP-11 operating
system).

RSX-11M had a sort of spin lock, but I’ll be damned if I can remember what
it was used for. I can’t recall ever acquiring it, or even teaching people
about it… but that was a long, long, time ago. It is possible there
wasn’t even such a concept prior to the PDP-11/74, which was the Symmetric
Multiprocessor PDP-11 that never (really commercially) shipped.

Without DPCs, we had the problem of running noninterruptible for “too
long”. This meant that a device which interrupted at level 5 (the serial
ports) could consume, between interrupt latency (considerable) and
returning from the interrupt, the entire CPU. Unfortunately, the tape
drive was less critical, level 3, but if the tape was spinning and hit the
BOT/EOT marks while serial buffers were being manipulated, the tape driver
wouldn’t get the interrupt, and wouldn’t stop the spindle, so the tape
ended up flapping off one or another of the reels. So we had to “open
priority windows” (enable interrupts and lower the CPU IRQL level), but of
course that meant that our device could interrupt again, and enter its ISR
recursively, which was a profoundly unhappy situation to be in. So we had
to make sure OUR device was disabled before opening a window, which led to
additional complexities…the Windows model is so much cleaner than what
we did on PDP-11s (I never used RSX-11M, we were always rolling our own
software, whether for graphics displays, line multiplexors, Ethernet
concentrators, or Hydra).

The DPC model is SO much cleaner!
joe

The concept of IRQL? Weeeeeellll… RSX-11M had user-state, system-state,
and a small number of interrupt states. These are SORT of equivalent to
IRQL.

Tangential to the topic is the fact that the DPC List is a direct
descendant of the RSX-11M Fork List… exact same concept.

Peter
OSR
@OSRDrivers


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

>> should actually run. It will run as many as it can, but the remainder

> will remain suspended

Not “suspended”, but “runnable on the queue”, incrementing the Processor
Queue perf counter.

Well, I was using “suspended” in a more generic sense of “not running”,
but you are absolutely right. I should have said “waiting”.
joe


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

> Well, I was using “suspended” in a more generic sense of “not running”, but you are absolutely right.

I should have said “waiting”.

…and you would still be wrong if you said so…

A thread would be in a READY state, which is different from the WAITING one. BTW, IIRC, there is no such state as SUSPENDED one anyway…

Anton Bassov

> RSX-11M had a sort of spin lock, but I’ll be damned if I can remember what it was used for.

I can’t recall ever acquiring it, or even teaching people about it… but that was a long, long, time ago.



Probably, it was the a spinlock that Mr.Kyler was referring to in the quote below



Anton Bassov

>> Well, I was using “suspended” in a more generic sense of “not running”,

> but you are absolutely right.
> I should have said “waiting”.

…and you would still be wrong if you said so…

A thread would be in a READY state, which is different from the WAITING
one. BTW, IIRC, there is no such state as SUSPENDED one anyway…

It all depends who you took your first OS course from. I had Dave Parnas.

Anton Bassov


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer