Tracing - WPP, ETW, TraceView, WinDbg

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Jose Sua[SMTP:xxxxx@windows.microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Wednesday, February 07, 2007 10:25 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Tracing - WPP, ETW, TraceView, WinDbg

For WPP when tracing in not enabled all you have is a simple compare, if
tracing enabled for the level and flags, which is very fast.

No difference against traditional traces.

When tracing is enabled we do take the arguments and try to log the
arguments.

That’s my concern. When you enable traces, accessing invalid argument can cause crash. So driver has to be tested with both traces enabled and disabled. So there is no real difference against two builds from the same sources. Except, and I understand it can be important in some cases, the need to install different driver binary.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

Testing is part of the business, you need to test your code…

For the basic types there is no problem, and for strings WPP checks that
they are not null or length = 0. That covers most of the cases.

You should give it a try Michal.

Also take a look at the new ETW API on Vista.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Michal Vodicka
Sent: Wednesday, February 07, 2007 2:26 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Tracing - WPP, ETW, TraceView, WinDbg


From:
xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com
] on behalf of Jose Sua[SMTP:xxxxx@windows.microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Wednesday, February 07, 2007 10:25 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Tracing - WPP, ETW, TraceView, WinDbg

For WPP when tracing in not enabled all you have is a simple compare,
if
tracing enabled for the level and flags, which is very fast.

No difference against traditional traces.

When tracing is enabled we do take the arguments and try to log the
arguments.

That’s my concern. When you enable traces, accessing invalid argument
can cause crash. So driver has to be tested with both traces enabled and
disabled. So there is no real difference against two builds from the
same sources. Except, and I understand it can be important in some
cases, the need to install different driver binary.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Jose Sua[SMTP:xxxxx@windows.microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Thursday, February 08, 2007 12:40 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Tracing - WPP, ETW, TraceView, WinDbg

Testing is part of the business, you need to test your code…

Sure but my point is different. Debug prints are usually always enabled during development and they’re very thoroughly tested. If I understand correctly, for the same conditions you’d have to have WPP/ETW loggers running all the time.

For the basic types there is no problem, and for strings WPP checks that
they are not null or length = 0. That covers most of the cases.

What about *p, rogue pointer, an attempt to dump memory longer than available etc.? Something can be catched by lint or prefast but not everything.

You should give it a try Michal.

Maybe I should. But the initial time investment seems to big and I don’t see a gain which’d justify it when we already have good and working solution. Recently, new WDK compiler allowed me to improve it using variable argument macros so I can add prefix, format string and add final “\n” using one DbgPrint() call within a macro :slight_smile:

Also take a look at the new ETW API on Vista.

I just read about AutoLoggers. Could you explain what exactly following paragraph means? Thanks.

The AutoLogger sessions increase the system boot time and should be used sparingly. Services that want to capture information during the boot process should consider adding controller logic to itself instead of using the AutoLogger session.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

Well said Michal. How many endless threads have there been concerning
Vista being unresponsive when run with a debugger attached, generally
due to trace output.

mm

>> xxxxx@upek.com 2007-02-07 19:20 >>>

From:
xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com]
on behalf of Jose Sua[SMTP:xxxxx@windows.microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Thursday, February 08, 2007 12:40 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Tracing - WPP, ETW, TraceView, WinDbg

Testing is part of the business, you need to test your code…

Sure but my point is different. Debug prints are usually always enabled
during development and they’re very thoroughly tested. If I understand
correctly, for the same conditions you’d have to have WPP/ETW loggers
running all the time.

For the basic types there is no problem, and for strings WPP checks
that
they are not null or length = 0. That covers most of the cases.

What about *p, rogue pointer, an attempt to dump memory longer than
available etc.? Something can be catched by lint or prefast but not
everything.

You should give it a try Michal.

Maybe I should. But the initial time investment seems to big and I
don’t see a gain which’d justify it when we already have good and
working solution. Recently, new WDK compiler allowed me to improve it
using variable argument macros so I can add prefix, format string and
add final “\n” using one DbgPrint() call within a macro :slight_smile:

Also take a look at the new ETW API on Vista.

I just read about AutoLoggers. Could you explain what exactly following
paragraph means? Thanks.

The AutoLogger sessions increase the system boot time and should be
used sparingly. Services that want to capture information during the
boot process should consider adding controller logic to itself instead
of using the AutoLogger session.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

On MSDN you will find the extended format strings that WPP supports,
some cool ones like printing your enum names not criptic hex values.

So WPP scans your code and finds the trace statements, and generates the
required code for logging the event. At this point there is type
checking, so the tracing code is not def out like its normally done for
debug code. What WPP passes to ETW are then pointer and size of the
arguments. You can see all this stuff if you look at the generated code
in the TMH file.

Sample, assuming my trace function is printf this is user mode but
similar in kermel mode.

printf(“Pointer %%p:%p\n”, ptr); //logging the pointer value

TraceMessage(Logger, WPP_TRACE_OPTIONS, (LPGUID)TraceGuid, id,
(&(_a1)),((SIZE_T)sizeof(const void*)), 0);

You can see here that we are not logging the format string that is
stripped of and placed in the PDB, which later are extracted as TMF
files.

An Autologger is a trace session that is started at boot time and is
configured by the user, it requires resources. This is OK if you want
to trace a problem with your driver at boot time. But for performance
reason we do not encourage having loggers running all the time, the
comment was referring to services. There is the cost of the buffers that
need to be allocated for the session this value is configurable. Then
there is the frequency and length of the data been logged to the session
by providers. So if frequency is high you will get a performance hit
depending on those valued.

More on this if you look at ETW on MSD.

Thanks,
Jose Sua
Microsoft Corporation
This posting is provided “AS IS” with no warranties and confers no
rights.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Michal Vodicka
Sent: Wednesday, February 07, 2007 4:20 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Tracing - WPP, ETW, TraceView, WinDbg


From:
xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com
] on behalf of Jose Sua[SMTP:xxxxx@windows.microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Thursday, February 08, 2007 12:40 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Tracing - WPP, ETW, TraceView, WinDbg

Testing is part of the business, you need to test your code…

Sure but my point is different. Debug prints are usually always enabled
during development and they’re very thoroughly tested. If I understand
correctly, for the same conditions you’d have to have WPP/ETW loggers
running all the time.

For the basic types there is no problem, and for strings WPP checks
that
they are not null or length = 0. That covers most of the cases.

What about *p, rogue pointer, an attempt to dump memory longer than
available etc.? Something can be catched by lint or prefast but not
everything.

You should give it a try Michal.

Maybe I should. But the initial time investment seems to big and I don’t
see a gain which’d justify it when we already have good and working
solution. Recently, new WDK compiler allowed me to improve it using
variable argument macros so I can add prefix, format string and add
final “\n” using one DbgPrint() call within a macro :slight_smile:

Also take a look at the new ETW API on Vista.

I just read about AutoLoggers. Could you explain what exactly following
paragraph means? Thanks.

The AutoLogger sessions increase the system boot time and should be used
sparingly. Services that want to capture information during the boot
process should consider adding controller logic to itself instead of
using the AutoLogger session.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Jose Sua[SMTP:xxxxx@windows.microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Thursday, February 08, 2007 6:10 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Tracing - WPP, ETW, TraceView, WinDbg

On MSDN you will find the extended format strings that WPP supports,
some cool ones like printing your enum names not criptic hex values.

Nice but one of things which dicourages me if the need of separate preprocessor which allows such cool things. The same effect could be achieved with standard C preprocessor. I’m sure because I did it for our firmware development. We have limited ROM size and code with format strings doesn’t fit. Preprocessor tricks allowed me to store them in separate data segment which is finally extracted from ROM image. The rest works similarly; trace data are buffered in memory, transferred to PC and displayed there using extracted format messages. Without a need for undocumented compiler features.

So WPP scans your code and finds the trace statements, and generates the
required code for logging the event. At this point there is type
checking, so the tracing code is not def out like its normally done for
debug code. What WPP passes to ETW are then pointer and size of the
arguments. You can see all this stuff if you look at the generated code
in the TMH file.

I quite understand it but even type checking doesn’t save code from possible crash. No problem with the approach; the only doubt is if all the code is really tested during development as standard traces are. I guess a coverage tool could solve it.

An Autologger is a trace session that is started at boot time and is
configured by the user, it requires resources. This is OK if you want
to trace a problem with your driver at boot time. But for performance
reason we do not encourage having loggers running all the time, the
comment was referring to services. There is the cost of the buffers that
need to be allocated for the session this value is configurable. Then
there is the frequency and length of the data been logged to the session
by providers. So if frequency is high you will get a performance hit
depending on those valued.

Thanks for clarification. Sometimes it is hard to guess what docs wording exactly mean (sometimes drivers and services are distinguished, sometimes not).

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

>>I quite understand it but even type checking doesn’t save code from possible crash. No problem with the approach; the only doubt is if all the code is really tested during development as standard traces are. I guess a coverage tool could solve it.
<<

I can tell you that for KMDF we most certainly did and still do exactly that [boost coverage to greater levels and having tracing on during a test pass is a lot easier way to get there than some others we’ve tried]. I can also say that given the barrage of messages amd mission statements and other feedback from management and coworkers I get as a test developer at Microsoft, I’d be dumbfounded to discover that we were the only team at Microsoft that actually did this. It is hard to get past the quality gates if you don’t. I can’t and won’t say everyone does it, as I have no way of possibly ever knowing that. If I did have the time it took to find out, I’d much rather investigate something else…

But the same concern can apply to traces. I’ve worked on more than one project [talking about the dozen or more places I’ve worked in the last 30 some years] where traces were #ifdef’d and rarely compiled into a piece of code, and were thus one of the least tested pieces of code in the product. It was often common to have to work on even getting them to compile in more than once case I can still recall.

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Bob Kjelgaard[SMTP:xxxxx@microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Thursday, February 08, 2007 11:04 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Tracing - WPP, ETW, TraceView, WinDbg

I can tell you that for KMDF we most certainly did and still do exactly that [boost coverage to greater levels and having tracing on during a test pass is a lot easier way to get there than some others we’ve tried].

What coverage tool are you using?

I can also say that given the barrage of messages amd mission statements and other feedback from management and coworkers I get as a test developer at Microsoft, I’d be dumbfounded to discover that we were the only team at Microsoft that actually did this. It is hard to get past the quality gates if you don’t.

Exactly. Good to know, it improves my feeling about KMDF.

I can’t and won’t say everyone does it, as I have no way of possibly ever knowing that. If I did have the time it took to find out, I’d much rather investigate something else…

It should be task for project management…

But the same concern can apply to traces. I’ve worked on more than one project [talking about the dozen or more places I’ve worked in the last 30 some years] where traces were #ifdef’d and rarely compiled into a piece of code, and were thus one of the least tested pieces of code in the product. It was often common to have to work on even getting them to compile in more than once case I can still recall.

To have useful traces they need to become standard and mandatory debugging tool. No #ifdefs in sources; just macros which expand to something or nothing according to build configuration. Currently we use 3 build types built at once. Debug (no optimization and traces enabled), release with traces and pure release. This way traces are always compilable or break build. Debug version is usually used during development. Release with traces during internal testing and for problems solution. Relase for final testing and for customers. It can be replaced with rlt build if a problem is investigated.

As mandatory I mean they’re integral part of code and developers should write them at least for function enter (name + parameters) and exit (name + return value(s)). How to actually debug code is on every developer but people usually learn quickly what is more efficient.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

“Michal Vodicka” wrote in message
news:xxxxx@ntdev…

> What coverage tool are you using?

I’ve been using BullseyeCoverage http://www.bullseye.com/ for 9 years now.
It is rock solid and works fine in the kernel.


Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
http://www.windrvr.com
Remove StopSpam from the email to reply

>>What coverage tool are you using?

I know people hate this answer, but I have to give it. I can’t tell you. I’ve read articles that I believe indicate we use such technologies, or I wouldn’t have even said as much as I did…

>Exactly. Good to know, it improves my feeling about KMDF.

My overall point is I’m certain we’re not the only team in Vista who did that specific thing. I just can’t list specific cases other than my own.

Finally, I don’t mean I know every single trace message in KMDF gets hit on a regular basis. We certainly try to find our flaws before they go anywhere else. We seem to have done pretty well, so far- but that’s not yet “perfect”. The rest of it we’re still working on.

>To have useful traces they need

No argument here about the tracing concepts or mechanism you’re describing here. I’ve seen this kind of thing and it is great when it is done this way. It’s an excellent practice.

My point was perhaps redundant by now- it is not the tracing method that governs whether the trace code is as robust as everything else in a product- it is the test strategies and practices used by the producer of the product.

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Bob Kjelgaard[SMTP:xxxxx@microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Friday, February 09, 2007 2:50 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Tracing - WPP, ETW, TraceView, WinDbg

>>What coverage tool are you using?

I know people hate this answer, but I have to give it. I can’t tell you. I’ve read articles that I believe indicate we use such technologies, or I wouldn’t have even said as much as I did…

I’m a bit surprised but don’t need to understand everything :slight_smile:

My point was perhaps redundant by now- it is not the tracing method that governs whether the trace code is as robust as everything else in a product- it is the test strategies and practices used by the producer of the product.

I completely agree. In my experience the best test practices help developers with their work and are adopted voluntarily by them. I’m not quite sure if WPP has this property but as Jose said, I should try. Sometimes :slight_smile:

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

Michal Vodicka writes:

Nice but one of things which dicourages me if the need
of separate preprocessor which allows such cool things.
The same effect could be achieved with standard C
preprocessor. I’m sure because I did it for our firmware
development. We have limited ROM size and code with format
strings doesn’t fit. Preprocessor tricks allowed me to
store them in separate data segment which is finally
extracted from ROM image. The rest works similarly; trace
data are buffered in memory, transferred to PC and
displayed there using extracted format messages. Without
a need for undocumented compiler features.

Isn’t this exactly what WPP does? The output strings are not part of the driver binary. They’re stored in the PDB and/or in TMF files, which the ETW tools use to reconstruct the output. This is also beneficial when you are trying to write many thousands of messages per second. You don’t have the overhead of formatting and storing all those text strings. Only the message arguments, which are usually quite compact, are written to the log.

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Arlie Davis[SMTP:xxxxx@microsoft.com]
Reply To: Windows System Software Devs Interest List
Sent: Friday, February 09, 2007 8:19 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Tracing - WPP, ETW, TraceView, WinDbg

Michal Vodicka writes:
> Nice but one of things which dicourages me if the need
> of separate preprocessor which allows such cool things.
> The same effect could be achieved with standard C
> preprocessor. I’m sure because I did it for our firmware
> development. We have limited ROM size and code with format
> strings doesn’t fit. Preprocessor tricks allowed me to
> store them in separate data segment which is finally
> extracted from ROM image. The rest works similarly; trace
> data are buffered in memory, transferred to PC and
> displayed there using extracted format messages. Without
> a need for undocumented compiler features.

Isn’t this exactly what WPP does? The output strings are not part of the driver binary. They’re stored in the PDB and/or in TMF files, which the ETW tools use to reconstruct the output. This is also beneficial when you are trying to write many thousands of messages per second. You don’t have the overhead of formatting and storing all those text strings. Only the message arguments, which are usually quite compact, are written to the log.

Yes, it is exactly the same :slight_smile: My point was the same result can be achieved with standard tools and with no need for special preprocessor and compiler features.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Don Burn[SMTP:xxxxx@acm.org]
Reply To: Windows System Software Devs Interest List
Sent: Friday, February 09, 2007 12:10 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Tracing - WPP, ETW, TraceView, WinDbg

I’ve been using BullseyeCoverage http://www.bullseye.com/ for 9 years now.
It is rock solid and works fine in the kernel.

Looks very interesting. Could you share you experience with it? I’m mainly interested:

  • how much time is necessary to prepare existing kernel driver for coverage?
  • when and how it saves coverage data – is it possible to test PnP device removals and driver unloads?
  • is possible for example use instrumented driver for complete DTM testing and then analyze complete results?
  • is possible to give instrumented driver to QA lab where it’d be tested by several people on many machines and finally get complete results?
  • what about reboot and power transition testing?

Thanks.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

“Michal Vodicka” wrote in message
news:xxxxx@ntdev…
> ----------
> From:
> xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com]
> on behalf of Don Burn[SMTP:xxxxx@acm.org]
> Reply To: Windows System Software Devs Interest List
> Sent: Friday, February 09, 2007 12:10 AM
> To: Windows System Software Devs Interest List
> Subject: Re:[ntdev] Tracing - WPP, ETW, TraceView, WinDbg
>
> I’ve been using BullseyeCoverage http://www.bullseye.com/ for 9 years
> now.
> It is rock solid and works fine in the kernel.
>
Looks very interesting. Could you share you experience with it? I’m mainly
interested:
- how much time is necessary to prepare existing kernel driver for
coverage?
- when and how it saves coverage data – is it possible to test PnP device
removals and driver unloads?
- is possible for example use instrumented driver for complete DTM testing
and then analyze complete results?
- is possible to give instrumented driver to QA lab where it’d be tested by
several people on many machines and finally get complete results?
- what about reboot and power transition testing?

Michal,

It is a recompile to build a version for coverage. This produces a
file you need to place on the testing machines, plus a driver that has code
inserted to for the coverage. I find that it has around a 10% performance
degradation (though there are times it can be worse, depends the number of
control flow points in the code). I don’t see why it could not be used
with DTM (but I have not gotten DTM working so … It works fine over
reboots, and stops and starts of the driver.

You can definitely give an instrumented driver to a QA lab, I’ve even
done it to a customer. One other thing I like is they provide the runtime
sources, a number of years ago I found a problem with the tool on the
checked build, I was able to fix it since the source was there.

They will give you a copy for a 30 day eval, so it is well worth
checking out.


Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
http://www.windrvr.com
Remove StopSpam from the email to reply

Thanks, Don. Looks very promising. I’ll ask them for evaluation version.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]


From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Don Burn[SMTP:xxxxx@acm.org]
Reply To: Windows System Software Devs Interest List
Sent: Friday, February 09, 2007 10:59 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Tracing - WPP, ETW, TraceView, WinDbg

“Michal Vodicka” wrote in message
> news:xxxxx@ntdev…
> > ----------
> > From:
> > xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com]
> > on behalf of Don Burn[SMTP:xxxxx@acm.org]
> > Reply To: Windows System Software Devs Interest List
> > Sent: Friday, February 09, 2007 12:10 AM
> > To: Windows System Software Devs Interest List
> > Subject: Re:[ntdev] Tracing - WPP, ETW, TraceView, WinDbg
> >
> > I’ve been using BullseyeCoverage http://www.bullseye.com/ for 9 years
> > now.
> > It is rock solid and works fine in the kernel.
> >
> Looks very interesting. Could you share you experience with it? I’m mainly
> interested:
> - how much time is necessary to prepare existing kernel driver for
> coverage?
> - when and how it saves coverage data – is it possible to test PnP device
> removals and driver unloads?
> - is possible for example use instrumented driver for complete DTM testing
> and then analyze complete results?
> - is possible to give instrumented driver to QA lab where it’d be tested by
> several people on many machines and finally get complete results?
> - what about reboot and power transition testing?
>
> Michal,
>
> It is a recompile to build a version for coverage. This produces a
> file you need to place on the testing machines, plus a driver that has code
> inserted to for the coverage. I find that it has around a 10% performance
> degradation (though there are times it can be worse, depends the number of
> control flow points in the code). I don’t see why it could not be used
> with DTM (but I have not gotten DTM working so … It works fine over
> reboots, and stops and starts of the driver.
>
> You can definitely give an instrumented driver to a QA lab, I’ve even
> done it to a customer. One other thing I like is they provide the runtime
> sources, a number of years ago I found a problem with the tool on the
> checked build, I was able to fix it since the source was there.
>
> They will give you a copy for a 30 day eval, so it is well worth
> checking out.
>
>
> –
> Don Burn (MVP, Windows DDK)
> Windows 2k/XP/2k3 Filesystem and Driver Consulting
> http://www.windrvr.com
> Remove StopSpam from the email to reply
>
>
>
>
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at http://www.osronline.com/article.cfm?id=256
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>