Is there a way I can detect (in my disk class upper filter driver) the idle time in vista? Also I need to predict the duration of the idle time. Idle time is when my filter driver want to kick-off some process/activity of its own (which may involve disk IO) without affecting the response time of user applications. Thanks.
Define “idle time”: CPU? Disk? No I/O to any peripheral? User no longer at the keyboard? How long must one of these be “idle” (and within what tolerance) before you decide it’s the start of “idle time”?
Given that we have imperfect knowledge of the future, and that a user could walk up to a keyboard and start a program at any point in time, how could you possibly predict how long an “idle time” period would last (regardless of the duration of that idle time)?
If you merely want a vague approximation of “idle”, you can use “no interactive activity” which is what Windows uses to display (for example) the screen saver. You can catch this (and some similar cases) from user-mode.
Otherwise, you need to make your own definition and go with it,
Peter
OSR
Crystal balls tend to work well in predicting the future. All you can do is
hypothesize that, because of low-level activity in the past, there’s a
certain probability that there will be no action in the future.
Note that all you can detect at your level is “disk idle time”. For
example, on laptops, if there has been no disk activity in several minutes,
the disks are powered down to save battery.
Note also that if you choose the wrong interval to measure, you will always
start your action just before some regularly-scheduled disk event. So if
you want to “predict” with any reliability, you need to track the mean idle
interval, whatever that means (for example, intervals of < n milliseconds
might be ignored in computing mean idle time). So if your current idle time
exceeds the mean + 1 sigma, then you have a higher probability that you have
a “real” idle situation. Note that you can compute mean and standard
deviation with only three variables (any good statistics text will give you
this formula) but it involves a sqrt, not easy in kernel space if you use
floating point, but any good text on numerical methods will give you a way
to compute the square root approximation, and you can do this in integers
using 64-bit integers and get a reasonable approximation, good enough for
estimation purposes.
joe
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@yahoo.com
Sent: Tuesday, December 30, 2008 6:34 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] idle time detection/prediction in vista?
Is there a way I can detect (in my disk class upper filter driver) the idle
time in vista? Also I need to predict the duration of the idle time. Idle
time is when my filter driver want to kick-off some process/activity of its
own (which may involve disk IO) without affecting the response time of user
applications. Thanks.
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
–
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.
Am I still in holiday mood!!!
By the look at the request, it was clear that the solution domain is
“Adaptive Algorithm”.
To do it in kernel mode would be a big concern!. Perhaps in worker thread at
the most.
For whatever its worth, the first reference I would look at is —
http://www.soe.ucsc.edu/~sbrandt/290S/miao.pdf . And it should lead to some
direction to approach the problem I guess!
Well, as if I’ve to sign something !
Prokash Sinha
http://prokash.squarespace.com
Success has many fathers, but failure is an orphan.
----- Original Message -----
From: “Joseph M. Newcomer”
To: “Windows System Software Devs Interest List”
Sent: Wednesday, December 31, 2008 7:54 AM
Subject: RE: [ntdev] idle time detection/prediction in vista?
> Crystal balls tend to work well in predicting the future. All you can do
> is
> hypothesize that, because of low-level activity in the past, there’s a
> certain probability that there will be no action in the future.
>
> Note that all you can detect at your level is “disk idle time”. For
> example, on laptops, if there has been no disk activity in several
> minutes,
> the disks are powered down to save battery.
>
> Note also that if you choose the wrong interval to measure, you will
> always
> start your action just before some regularly-scheduled disk event. So if
> you want to “predict” with any reliability, you need to track the mean
> idle
> interval, whatever that means (for example, intervals of < n milliseconds
> might be ignored in computing mean idle time). So if your current idle
> time
> exceeds the mean + 1 sigma, then you have a higher probability that you
> have
> a “real” idle situation. Note that you can compute mean and standard
> deviation with only three variables (any good statistics text will give
> you
> this formula) but it involves a sqrt, not easy in kernel space if you use
> floating point, but any good text on numerical methods will give you a way
> to compute the square root approximation, and you can do this in integers
> using 64-bit integers and get a reasonable approximation, good enough for
> estimation purposes.
> joe
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of
> xxxxx@yahoo.com
> Sent: Tuesday, December 30, 2008 6:34 PM
> To: Windows System Software Devs Interest List
> Subject: [ntdev] idle time detection/prediction in vista?
>
> Is there a way I can detect (in my disk class upper filter driver) the
> idle
> time in vista? Also I need to predict the duration of the idle time. Idle
> time is when my filter driver want to kick-off some process/activity of
> its
> own (which may involve disk IO) without affecting the response time of
> user
> applications. Thanks.
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> –
> This message has been scanned for viruses and dangerous content by
> MailScanner, and is believed to be clean.
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
Surprisingly, the cost of keeping the mean and variance in three variables
is incredibly efficient. The first usage I saw of this was by Jim Mitchell
(now at Sun, and the Godfather of Java [he signed off on the first budget
for the project]); he kept storage allocator statistics in floating point on
a machine with slow floating point, but pointed out that the cost of
gratuitous memory splitting on performance was an order of magnitude or two
larger than the cost of floating point on a machine with the power of a 286
(a multimillion-dollar mainframe).
Variance = (sum(x[i]**2) - (sum(x[i]))**2) / (n * (n - 1))
Where **2 means “squared”. You need only three variables to maintain this,
SumOfSquareOfSamples, SumOfSamples and n. The only expensive computation is
doing the integer-square-root approximation to convert variance to standard
deviations.
joe
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Prokash Sinha
Sent: Wednesday, December 31, 2008 12:26 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] idle time detection/prediction in vista?
Am I still in holiday mood!!!
By the look at the request, it was clear that the solution domain is
“Adaptive Algorithm”.
To do it in kernel mode would be a big concern!. Perhaps in worker thread at
the most.
For whatever its worth, the first reference I would look at is —
http://www.soe.ucsc.edu/~sbrandt/290S/miao.pdf . And it should lead to some
direction to approach the problem I guess!
Well, as if I’ve to sign something !
Prokash Sinha
http://prokash.squarespace.com
Success has many fathers, but failure is an orphan.
----- Original Message -----
From: “Joseph M. Newcomer”
To: “Windows System Software Devs Interest List”
Sent: Wednesday, December 31, 2008 7:54 AM
Subject: RE: [ntdev] idle time detection/prediction in vista?
> Crystal balls tend to work well in predicting the future. All you can do
> is
> hypothesize that, because of low-level activity in the past, there’s a
> certain probability that there will be no action in the future.
>
> Note that all you can detect at your level is “disk idle time”. For
> example, on laptops, if there has been no disk activity in several
> minutes,
> the disks are powered down to save battery.
>
> Note also that if you choose the wrong interval to measure, you will
> always
> start your action just before some regularly-scheduled disk event. So if
> you want to “predict” with any reliability, you need to track the mean
> idle
> interval, whatever that means (for example, intervals of < n milliseconds
> might be ignored in computing mean idle time). So if your current idle
> time
> exceeds the mean + 1 sigma, then you have a higher probability that you
> have
> a “real” idle situation. Note that you can compute mean and standard
> deviation with only three variables (any good statistics text will give
> you
> this formula) but it involves a sqrt, not easy in kernel space if you use
> floating point, but any good text on numerical methods will give you a way
> to compute the square root approximation, and you can do this in integers
> using 64-bit integers and get a reasonable approximation, good enough for
> estimation purposes.
> joe
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of
> xxxxx@yahoo.com
> Sent: Tuesday, December 30, 2008 6:34 PM
> To: Windows System Software Devs Interest List
> Subject: [ntdev] idle time detection/prediction in vista?
>
> Is there a way I can detect (in my disk class upper filter driver) the
> idle
> time in vista? Also I need to predict the duration of the idle time. Idle
> time is when my filter driver want to kick-off some process/activity of
> its
> own (which may involve disk IO) without affecting the response time of
> user
> applications. Thanks.
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> –
> This message has been scanned for viruses and dangerous content by
> MailScanner, and is believed to be clean.
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
—
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
–
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
> doing the integer-square-root approximation to convert variance to standard
deviations.
…or just leave the variance as variance, or just square up all values to which the variance must be compared 
if sqrt is slow and multiplication is not, this makes sense.
–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com
I’m much worried about prediction than floating point computations.
Calculating and optimized computation of those statistical params are well
known, so it would not be hard to even find optimized versions.
Finding the time point ( or small epoch) to do the disk r/w and exponential
backoff when in traffic jam would be the key to solve this efficiently, IMO.
Well, as if I’ve to sign something !
Prokash Sinha
http://prokash.squarespace.com
Success has many fathers, but failure is an orphan.
----- Original Message -----
From: “Joseph M. Newcomer”
To: “Windows System Software Devs Interest List”
Sent: Wednesday, December 31, 2008 4:55 PM
Subject: RE: [ntdev] idle time detection/prediction in vista?
> Surprisingly, the cost of keeping the mean and variance in three variables
> is incredibly efficient. The first usage I saw of this was by Jim
> Mitchell
> (now at Sun, and the Godfather of Java [he signed off on the first budget
> for the project]); he kept storage allocator statistics in floating point
> on
> a machine with slow floating point, but pointed out that the cost of
> gratuitous memory splitting on performance was an order of magnitude or
> two
> larger than the cost of floating point on a machine with the power of a
> 286
> (a multimillion-dollar mainframe).
>
> Variance = (sum(x[i]**2) - (sum(x[i]))**2) / (n * (n - 1))
>
> Where **2 means “squared”. You need only three variables to maintain
> this,
> SumOfSquareOfSamples, SumOfSamples and n. The only expensive computation
> is
> doing the integer-square-root approximation to convert variance to
> standard
> deviations.
> joe
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Prokash Sinha
> Sent: Wednesday, December 31, 2008 12:26 PM
> To: Windows System Software Devs Interest List
> Subject: Re: [ntdev] idle time detection/prediction in vista?
>
> Am I still in holiday mood!!!
>
> By the look at the request, it was clear that the solution domain is
> “Adaptive Algorithm”.
>
> To do it in kernel mode would be a big concern!. Perhaps in worker thread
> at
> the most.
>
> For whatever its worth, the first reference I would look at is —
>
> http://www.soe.ucsc.edu/~sbrandt/290S/miao.pdf . And it should lead to
> some
> direction to approach the problem I guess!
>
> Well, as if I’ve to sign something !
> Prokash Sinha
> http://prokash.squarespace.com
> Success has many fathers, but failure is an orphan.
>
> ----- Original Message -----
> From: “Joseph M. Newcomer”
> To: “Windows System Software Devs Interest List”
> Sent: Wednesday, December 31, 2008 7:54 AM
> Subject: RE: [ntdev] idle time detection/prediction in vista?
>
>
>> Crystal balls tend to work well in predicting the future. All you can do
>> is
>> hypothesize that, because of low-level activity in the past, there’s a
>> certain probability that there will be no action in the future.
>>
>> Note that all you can detect at your level is “disk idle time”. For
>> example, on laptops, if there has been no disk activity in several
>> minutes,
>> the disks are powered down to save battery.
>>
>> Note also that if you choose the wrong interval to measure, you will
>> always
>> start your action just before some regularly-scheduled disk event. So if
>> you want to “predict” with any reliability, you need to track the mean
>> idle
>> interval, whatever that means (for example, intervals of < n milliseconds
>> might be ignored in computing mean idle time). So if your current idle
>> time
>> exceeds the mean + 1 sigma, then you have a higher probability that you
>> have
>> a “real” idle situation. Note that you can compute mean and standard
>> deviation with only three variables (any good statistics text will give
>> you
>> this formula) but it involves a sqrt, not easy in kernel space if you use
>> floating point, but any good text on numerical methods will give you a
>> way
>> to compute the square root approximation, and you can do this in integers
>> using 64-bit integers and get a reasonable approximation, good enough for
>> estimation purposes.
>> joe
>>
>> -----Original Message-----
>> From: xxxxx@lists.osr.com
>> [mailto:xxxxx@lists.osr.com] On Behalf Of
>> xxxxx@yahoo.com
>> Sent: Tuesday, December 30, 2008 6:34 PM
>> To: Windows System Software Devs Interest List
>> Subject: [ntdev] idle time detection/prediction in vista?
>>
>> Is there a way I can detect (in my disk class upper filter driver) the
>> idle
>> time in vista? Also I need to predict the duration of the idle time. Idle
>> time is when my filter driver want to kick-off some process/activity of
>> its
>> own (which may involve disk IO) without affecting the response time of
>> user
>> applications. Thanks.
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>> –
>> This message has been scanned for viruses and dangerous content by
>> MailScanner, and is believed to be clean.
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> –
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
Maxim S. Shatskih wrote:
> doing the integer-square-root approximation to convert variance to standard
> deviations.
>…or just leave the variance as variance, or just square up all values to which the variance must be compared
This is good advice, but a good Newton-Raphson integer square root only
takes one addition and one shift per iteration, and it only takes about
3 iterations to get a value that is “good enough” for most purposes.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
I knew it converged quickly, but it has been about 40 years since I last
wrote one, so I couldn’t do it without a reference book. The last one I
looked at (in floating point) didn’t even have a loop to converge; it did
the “iterations” inline (I remember looking at this code in 1970 or so…)
joe
-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tim Roberts
Sent: Friday, January 02, 2009 1:05 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] idle time detection/prediction in vista?
Maxim S. Shatskih wrote:
> doing the integer-square-root approximation to convert variance to
> standard deviations.
>…or just leave the variance as variance, or just square up all values to
which the variance must be compared
is not, this makes sense.
This is good advice, but a good Newton-Raphson integer square root only
takes one addition and one shift per iteration, and it only takes about
3 iterations to get a value that is “good enough” for most purposes.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
–
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.