Monitoring disk access

NTDEV
1

Hello,

My aim is to log all volume read writes.
So either I write a volume level filter driver to monitor this data and log
it OR
I can enable diskperf to log all volume read writes and then write
a WMI client to access this data. My requirement is that the log should
contain all the blocks that were read or written without skipping *any* of
the
info.
As I understand, kernel event logging is for reporting purposes - so can
I use it for this purpose ? Is it reliable enough to give me all the info ?
Or is the volume level filter driver approach better ? I am new to drivers
world
so would like to evaluate all user mode methods before I get into the kernel
world :).
Also for simplicity I am assuming that I have a dedicated volume where I’ll
be
storing this log. So I wont be monitoring this volume thereby avoiding the
circular
dependancy issue.

Any help would be appreciated.

Thanks,

  • Gary.
2

Gary, just to clarify:

The diskperf driver does not log the individual reads and writes. it simply
maintains performance statistics, including:

– Number of bytes read and written
– Number of reads and writes that have occurred
– Count of the number of currently outstanding requests
– The number of split requests

In other words, it’s not designed for this purpose.

Thanks,
June Blender
MSFT, DDK Tools

“Gary” wrote in message news:xxxxx@ntdev…
> Hello,
>
> My aim is to log all volume read writes.
> So either I write a volume level filter driver to monitor this data and
> log
> it OR
> I can enable diskperf to log all volume read writes and then write
> a WMI client to access this data. My requirement is that the log should
> contain all the blocks that were read or written without skipping any of
> the
> info.
> As I understand, kernel event logging is for reporting purposes - so can
> I use it for this purpose ? Is it reliable enough to give me all the info
> ?
> Or is the volume level filter driver approach better ? I am new to drivers
> world
> so would like to evaluate all user mode methods before I get into the
> kernel
> world :).
> Also for simplicity I am assuming that I have a dedicated volume where
> I’ll
> be
> storing this log. So I wont be monitoring this volume thereby avoiding the
> circular
> dependancy issue.
>
> Any help would be appreciated.
>
> Thanks,
> - Gary.
>
>
>
>
>

3

Gary wrote:

My aim is to log all volume read writes.

(sorry I didn’t see this earlier)

I guess it depends on what “log” means. You mean you need the disk, the
offset, and the length of the operation? Or you mean you want the data,
too.

a WMI client to access this data. My requirement is that the log should
contain all the blocks that were read or written without skipping *any* of
the
info.

Without skipping *any*, *anytime*, *ever*?? Hmmmm (read on)…

As I understand, kernel event logging is for reporting purposes - so can
I use it for this purpose ? Is it reliable enough to give me all the info ?
Or is the volume level filter driver approach better ? I am new to drivers
world
so would like to evaluate all user mode methods before I get into the kernel
world :).

Good choice.

Kernel Event Logging works extrodinarily well in this circumstance.
I’ve personally used it quite extensively. There are a couple of things
about which you need to be aware, however, which impact the data you
receive:

a) Event logging collects the data before the disk class driver. Thus,
the data is as sent by the file system, before it’s been divided into
chunks that are small enough for the adapter to handle. So, let’s say
the disk adapter on the system is capable of reading or writing only 64K
chunks at a time… if the user sends down a 1MB read, you’ll see the
1MB read, not a series of 64K reads. That either matters or it doesn’t,
depending on what you need to do.

b) On some systems, items in the event log can be passed to you out of
order. If order is critically important, you’ll need to consider this.

c) By default, the timestamp used for event logging is systemtime. If
you want better granularity, you’ll need to be sure to specify that you
want the high precision timer to be used.

d) Without *ever* skipping *any* write *ever*?? Hmmmm… You’ll need to
size your event collection buffers carefully.

To be clear: I am not trying to dissuade you from using kernel tracing.
It’s a great feature, and I recommend it highly. When we at OSR
considered writing a filter driver versus using kernel event logging for
this purpose, we decided to use tracing… even though for us writing a
disk filter was merely a matter of get some existing code out of source
control.

But like anything else in life, if you want ultimate control, you’ll
need to do it yourself.

Peter
OSR

4

Hi Peter,

Thanks a lot for the info.

I guess it depends on what “log” means. You mean you need the disk, the
offset, and the length of the operation? Or you mean you want the data,
too.

I don’t need the data - just the disk, offset and length of operation.

d) Without *ever* skipping *any* write *ever*?? Hmmmm… You’ll need to
size your event collection buffers carefully.

A WMI client won’t solve the issue of maintaining this info across reboots,
right ?
When the system is going down for a reboot, my app will be stopped way
before
disk activity has been stopped. So I’ll be missing the changes that occured
from time
my app went down to actual system shutdown as well as those that happen at
startup
before my app starts.
Please correct me if this sounds incorrect.

Thanks,
Gary.

“PeterGV” wrote in message news:xxxxx@ntdev…
>
> Gary wrote:
>
> > My aim is to log all volume read writes.
>
> (sorry I didn’t see this earlier)
>
> I guess it depends on what “log” means. You mean you need the disk, the
> offset, and the length of the operation? Or you mean you want the data,
> too.
>
> > a WMI client to access this data. My requirement is that the log should
> > contain all the blocks that were read or written without skipping any
of
> > the
> > info.
>
> Without skipping any, anytime, ever?? Hmmmm (read on)…
>
> > As I understand, kernel event logging is for reporting purposes - so can
> > I use it for this purpose ? Is it reliable enough to give me all the
info ?
> > Or is the volume level filter driver approach better ? I am new to
drivers
> > world
> > so would like to evaluate all user mode methods before I get into the
kernel
> > world :).
>
> Good choice.
>
> Kernel Event Logging works extrodinarily well in this circumstance.
> I’ve personally used it quite extensively. There are a couple of things
> about which you need to be aware, however, which impact the data you
> receive:
>
> a) Event logging collects the data before the disk class driver. Thus,
> the data is as sent by the file system, before it’s been divided into
> chunks that are small enough for the adapter to handle. So, let’s say
> the disk adapter on the system is capable of reading or writing only 64K
> chunks at a time… if the user sends down a 1MB read, you’ll see the
> 1MB read, not a series of 64K reads. That either matters or it doesn’t,
> depending on what you need to do.
>
> b) On some systems, items in the event log can be passed to you out of
> order. If order is critically important, you’ll need to consider this.
>
> c) By default, the timestamp used for event logging is systemtime. If
> you want better granularity, you’ll need to be sure to specify that you
> want the high precision timer to be used.
>
> d) Without ever skipping any write ever?? Hmmmm… You’ll need to
> size your event collection buffers carefully.
>
> To be clear: I am not trying to dissuade you from using kernel tracing.
> It’s a great feature, and I recommend it highly. When we at OSR
> considered writing a filter driver versus using kernel event logging for
> this purpose, we decided to use tracing… even though for us writing a
> disk filter was merely a matter of get some existing code out of source
> control.
>
> But like anything else in life, if you want ultimate control, you’ll
> need to do it yourself.
>
> Peter
> OSR
>

5

Now when I read my last reply, the question sounds pretty dumb to me.
No user mode app can do what I want to.
This has to be done thru a driver.

Thanks for ur help so far.
Gary.

“Gary” wrote in message news:xxxxx@ntdev…
> Hi Peter,
>
> Thanks a lot for the info.
>
> > I guess it depends on what “log” means. You mean you need the disk, the
> > offset, and the length of the operation? Or you mean you want the data,
> > too.
>
> I don’t need the data - just the disk, offset and length of operation.
>
> > d) Without ever skipping any write ever?? Hmmmm… You’ll need to
> > size your event collection buffers carefully.
>
> A WMI client won’t solve the issue of maintaining this info across
reboots,
> right ?
> When the system is going down for a reboot, my app will be stopped way
> before
> disk activity has been stopped. So I’ll be missing the changes that
occured
> from time
> my app went down to actual system shutdown as well as those that happen at
> startup
> before my app starts.
> Please correct me if this sounds incorrect.
>
> Thanks,
> Gary.
>
>
>
> “PeterGV” wrote in message news:xxxxx@ntdev…
> >
> > Gary wrote:
> >
> > > My aim is to log all volume read writes.
> >
> > (sorry I didn’t see this earlier)
> >
> > I guess it depends on what “log” means. You mean you need the disk, the
> > offset, and the length of the operation? Or you mean you want the data,
> > too.
> >
> > > a WMI client to access this data. My requirement is that the log
should
> > > contain all the blocks that were read or written without skipping
any
> of
> > > the
> > > info.
> >
> > Without skipping any, anytime, ever?? Hmmmm (read on)…
> >
> > > As I understand, kernel event logging is for reporting purposes - so
can
> > > I use it for this purpose ? Is it reliable enough to give me all the
> info ?
> > > Or is the volume level filter driver approach better ? I am new to
> drivers
> > > world
> > > so would like to evaluate all user mode methods before I get into the
> kernel
> > > world :).
> >
> > Good choice.
> >
> > Kernel Event Logging works extrodinarily well in this circumstance.
> > I’ve personally used it quite extensively. There are a couple of things
> > about which you need to be aware, however, which impact the data you
> > receive:
> >
> > a) Event logging collects the data before the disk class driver. Thus,
> > the data is as sent by the file system, before it’s been divided into
> > chunks that are small enough for the adapter to handle. So, let’s say
> > the disk adapter on the system is capable of reading or writing only 64K
> > chunks at a time… if the user sends down a 1MB read, you’ll see the
> > 1MB read, not a series of 64K reads. That either matters or it doesn’t,
> > depending on what you need to do.
> >
> > b) On some systems, items in the event log can be passed to you out of
> > order. If order is critically important, you’ll need to consider this.
> >
> > c) By default, the timestamp used for event logging is systemtime. If
> > you want better granularity, you’ll need to be sure to specify that you
> > want the high precision timer to be used.
> >
> > d) Without ever skipping any write ever?? Hmmmm… You’ll need to
> > size your event collection buffers carefully.
> >
> > To be clear: I am not trying to dissuade you from using kernel tracing.
> > It’s a great feature, and I recommend it highly. When we at OSR
> > considered writing a filter driver versus using kernel event logging for
> > this purpose, we decided to use tracing… even though for us writing a
> > disk filter was merely a matter of get some existing code out of source
> > control.
> >
> > But like anything else in life, if you want ultimate control, you’ll
> > need to do it yourself.
> >
> > Peter
> > OSR
> >
>
>
>