+1
“We know the future will be just like the past because in te past, the
future was just lie the past”
When applied to the formation geological features of the Earth, it became
known as “uniformitarianism”. The counterargument, that features on the
Earth firned as the result of major upheavals, was known as
“catastrophism”.
They were both right.
I believe it was the late Stephen J. Gould who introduced the idea of
“puctuated equilibrium”, which can be summed up as “Things are pretty
stable until there is a major disruption, then they settle down to a new
stable state”. In the case of computers, the integrated-over-time
utilization does not tell you the performance over the next five seconds,
and the programmer has no synoptic view of what is going on. In addition,
CPU utilization does not indicate how much time is spent in the kernel
doing kernel things, how much time is spent in the kernel acting as the
(privileged) agent of a user thread, and how much utilization is in user
space. While I believe the kernel/user split can now be determined, the
kernel-as-kernel and kernel-as-agent numbers are much harder to determine.
Basing any thread assignments on an improperly-interpreted set of numbers
means your results are going to be more than a bit flaky, especially if
you have large, multithreaded apps with “bursty” behavior, e.g. SQL Server
getting a complex query, or IIS serving up a dynamic Web page, or dealing
with server-side scripting in response to some external user action.
Anyone who thinks building a scheduler is easy has never built a
scheduler. I worked with two teams on two different operating system in
my career, and I helped design some of data collection experiments. In
our multiprocessor OS, we went through a period of several months where we
were putting up a new scheduler every week, each one tweaked to deliver
something better than the previous version. Mostly, such tweaks did make
an improvement, but in sone cases the performance was substantially worse,
and in other cases we solved problem A only to uncover problem B, which in
turn had to be solved.
IBM avoided this issue in their TSS/360 OS: their scheduler was a state
machine, and by loading a new state transition table, you could change its
behavior. And when we profiled the OS, we found that 37% of the CPU
cycles were going to the scheduler, on a giant mainframe with about the
same computing power of a 286, but with less memory (512K for the 360,
640K for the 286), and we were supporting 60 concurrent users!
In addition, it was not just the scheduler. The scheduler and the virtual
memory manager were goombahs, and held complex dialogs trying to optimize
both system responsiveness and overall system performance For example, we
disovered certain aspects of the Working Set model required subtle changes
in our VM manager and in our scheduler, and how they interacted.
If you think LRU and Working Set solved all the problems of VM management,
have I got a deal for you! My late uncle had been a high-ranking officer
in the Nigerian army, and has this… (no, I can’t give you any details
until you buy into this project).
So I have the same reaction when somebody not only makes the claim that
they can do “optimal scheduling” for an app, but also starts asking “how
do I do this?”. My first response should simply be “ARE YOU OUT OF YOUR
FREAKIN’ MIND!!!”
My second reaction is: “do you have any data that suggests that the
current scheduler is doing a poor job?” Each “scheduler meeting” we held
had, across the width of the room, a 20’ or so section of the system
performance graph (for grins, he printed out the graph for an hour; it was
something close to 100 feet). Between six and ten people who lived in the
kernel source code watched while he pointed out where improvements could
be made, or where the new scheduler introduced problems, and we would then
propose various alternate implementations that might fix that problem with
introducing new ones. Sometimes we were even right.
So we have a newbie, who has ZERO data to back up the idea that the
Windows scheduler is inadequate, proposing to write a scheduler, and
furthermore has no idea of how to demonstrate that this frankenscheduler
is an improvement over the existing one. Yeah, right.
joe
xxxxx@gmail.com wrote:
> I want to find out what APIs are avaliable in DDK for determining the
> CPU Usage at the current stage? For example if CPU usage is less then a
> heavy process can be started.
You need to think about what you’re asking. Remember, at any given
time, a particular CPU’s utilization is either 0% or 100%. When you see
another number, that value has been integrated over time. So, it’s
possible to compute a CPU’s average utilization over the last second, or
over the last 5 seconds, but any information you get is history, and
doesn’t reflect future performance. You can’t base a scheduling
decision on that.
If you have a CPU-intensive process and you don’t want it to interfere,
just start it up and reduce its priority. The system will schedule it
when no other threads are ready.
> Secondly if in my driver if I am creating four threads and on my system
> if there are two virtual processors then is it possible to assign three
> light threads to one processor and one heavy thread to the second
> processor.
The Windows thread scheduler is the result of 25 years of development,
research, testing, experimentation, and improvement. There is
ABSOLUTELY no way you can do a better job.
> Thirdly if we manually do not assign threads to specific processors then
> how the OS assigns the threads to the processors?
When a thread reaches the end of its time slice, or gives up its time
slice by waiting for some resource, the scheduler runs. The scheduler
maintains a list of all of the threads in the system that are
“ready-to-run”, sorted by priority. It just picks the top of the list
and starts it. The thread that had been running goes back to the bottom
of the stack (at its priority level).
> I had taken a lecture about the multithreading capabilities in Java
> where new frameworks give more control to the programmers instead of
> strict control from the operating system.
This is an almost uniformly bad idea, because of the assertion I made
above: there is no way that even a very good Java programmer can do a
better job of thread scheduling than any of the mainstream operating
systems.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
NTDEV is sponsored by OSR
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer