Unnecessary obfuscation, "Hey I can't see my spew because of your spew!!!"

Bill McKenzie wrote:

What? You mean you found ETW difficult to deal with…I’m shocked!!

We have observed that every company that does any kind of framework
eventually feels that they have to invent an incredibly fancy and
complicated debug logging system. And, inevitably, the only people in
the entire world who understand it are the two people who wrote the
thing to begin with. Everyone else who tries to use it runs away
screaming, until they end up bypassing it, or finding the ONE
combination that always works, and then they do this:

#define DBGLOG(msg)
OurFancyTraceSystem((DbgLevelAlways,DbgModuleEverywhere,msg))

Have you ever looked at the WinCE debugging mechanism? Run away, run away!

I have long appreciated that Microsoft has resisted the temptation to do
this up, and I’m dismayed that they finally gave in to temptation. I
think there is no better method than giving us a simple DbgPrint and
letting us manage the spew ourselves. I predict that the fancy new WPP
tracing and DbgPrintEx will end up causing more heartburn than hero worship.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

WPP has its good and it’s bad aspects. Unfortunately the “good” ones
tend to be around the collection of logs on an otherwise stable system
(something that works better for app log than driver logs) and the “bad”
ones are all around the things developers need to do like implement the
tracing, or read the traces after they’re collected.

That said, we’re using WPP tracing in UMDF for both our kernel and user
components and once one our very, very smart developers got it setup in
our project it’s been terrific. Of course being the user-mode driver
system we have two things that make this really helpful:

1 - most of our internal partners are user-modey people and
haven’t ever used KD before.
2 - in most cases we don’t crash the system when something goes
wrong, so we don’t have problems with half-written log files.

I don’t want to be an appologist for it because I agree that getting it
working on all platforms is between nearly and completely impossible
(and I’ve pushed back on the team responsible for it, as have other
driver folks here). But when it is working it works quite nicely.

-p

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Tim Roberts
Sent: Thursday, May 11, 2006 2:31 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Unnecessary obfuscation, “Hey I can’t see my spew
because of your spew!!!”

Bill McKenzie wrote:

What? You mean you found ETW difficult to deal with…I’m shocked!!

We have observed that every company that does any kind of framework
eventually feels that they have to invent an incredibly fancy and
complicated debug logging system. And, inevitably, the only people in
the entire world who understand it are the two people who wrote the
thing to begin with. Everyone else who tries to use it runs away
screaming, until they end up bypassing it, or finding the ONE
combination that always works, and then they do this:

#define DBGLOG(msg)
OurFancyTraceSystem((DbgLevelAlways,DbgModuleEverywhere,msg))

Have you ever looked at the WinCE debugging mechanism? Run away, run
away!

I have long appreciated that Microsoft has resisted the temptation to do
this up, and I’m dismayed that they finally gave in to temptation. I
think there is no better method than giving us a simple DbgPrint and
letting us manage the spew ourselves. I predict that the fancy new WPP
tracing and DbgPrintEx will end up causing more heartburn than hero
worship.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

> #define WPP_DEBUG(m)

I know about WPP_DEBUG and didn’t find I liked it. The m parameter is the
format string followed by all the values. I wanted to still be able to pass
through the level+mask, so I could control WHICH DbgPrints came out. I
believe I could modify the WPP template file, but didn’t offhand see a way
to make a local copy of the template file in my projects directory. It’s
more effort to change a file that’s part of the DDK and is installed on all
the other team members systems. The DDK files are not checked into our
version control system, so I have no way to automagically update them on
other team members system. I’d probably get some frown on modifying a
standard DDK file too.

At the moment, I’m using statements like:

DebugPrint((TRACE_LEVEL_ERROR, IRP_MASK, “the value is %x\n”,someValue));

I can use a -func:DebugPrint((LEVEL,MASK,MSG,…)) option in WPP to turn
this into a WPP message or I can

#define DebugPrint(args) DebugPrintVa args

and have it route to a global function DebugPrintVa(level, mask, fmt,…)
that uses va_start/va_list/va_end and such to build a buffer that gets
DbgPrinted.

I’d really like to be able to say:

#define DebugPrint(args) DebugPrintWPP args; \
DebugPrintVa args

But unfortunately the WPP processor runs before the C preprocessor, so this
does not work. I could possible put a WPP print inside my DebugPrintVa
function, using a formatting string of “MyDriver!%s”, but this does all the
formatting work at message print time, counter to the WPP ideal.

I supposed the other option is I write ugly code like:

#ifdef WITH_WPP
DebugPrintWPP(TRACE_LEVEL_ERROR, IRP_MASK, “the value is %x\n”,someValue);
#else
DebugPrintWPP(TRACE_LEVEL_ERROR, IRP_MASK, “the value is %x\n”,someValue);
DebugPrintVa(TRACE_LEVEL_ERROR, IRP_MASK, “the value is %x\n”,someValue);
#endif

  • Jan

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Roddy, Mark[SMTP:xxxxx@stratus.com]
Reply To: Windows System Software Devs Interest List
Sent: Thursday, May 11, 2006 3:28 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Unnecessary obfuscation, “Hey I can’t see my spew because of your spew!!!”

I’ve haven’t found a compelling reason to use wpp for development.

Have you found any compelling reason to use WPP for anything? :wink: Reading messages about it here I’m glad I decided to not try it. Our trace macros work well, are more flexible and I only once encountered the situation where run-time formatting caused that the problem wasn’t reproducible any way. So I spent 30 minutes to change macros implementation to not use formatting and instead, store file, line and arguments to circular buffer. User reproduced problem and invoked crashdump and then I spent some time with manual extracting necessary info from the buffer. Still probably much faster than trying to port our trace macros for WPP.

Both wpp and debugprint suffer from not having a retrievable kernel
ringbuffer that captures run time trace information for post mortem
analysis.

If you run DebugView in the time of crash, it can extract its buffer with enough debug prints from crashdump later.

Runtime configurable debug prints are fine in production drivers, as
long as the driver ships in terse mode.

Not only terse but completely disabled. Usually it is hard to decide what it terse mode. Very rare error message can start appear thousands times per second under some unusual circumstances.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Tim Roberts[SMTP:xxxxx@probo.com]
Reply To: Windows System Software Devs Interest List
Sent: Thursday, May 11, 2006 11:31 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Unnecessary obfuscation, “Hey I can’t see my spew because of your spew!!!”

Have you ever looked at the WinCE debugging mechanism? Run away, run away!

Do you have anything against OutputDebugString()? Porting our Win32 trace library version was matter of making it UNICODE ready. Oh, you probably mean those debug zones :slight_smile:

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

ETW has an extension distributed with the debugger, WMITRACE.dll. The
extension can show the contents of the ETW buffers in the debugger.

!wmitrace.help to get all the options available.
!wmitrace.strdump Dump the Wmi Trace Event Structures
!wmitrace.searchpath sets the search path for TMF

The extension is very useful when you enable tracing at boot time or at
any time and can see the events on your trace session in the debugger.

One way to use this is to start the session in real time mode, tracelog
-rt -kd and then view the events, you must also set the search path
for the TMF files for all this to work. To extract the TMF files use
tracepdb -f .

The extension works very well in Vista because the public symbols
contain the required info for the extension. The problem pre-Vista is
that the public symbols do not export those required ETW symbols for the
extension. The fix that exported the symbols is expected to reach the
down level. I will investigate what the status of that is.

How to Enable Debugging without starting an ETW trace session

First you need to define the WPP_DEBUG macro and build your code.

A sample for a the DDK’s Tracedrv.sys sample driver below:

#define WPP_DEBUG(b) DbgPrint b, DbgPrint(“\n”);

You can use most formats and arguments with WPP_DEBUG. However, you
cannot use extended format specifications, such as %!HEXDUMP!%.

On the Kernel Debugger:

You will need to set the levels and flags values for the WPP control
structure.

1- Locate the address of the WPP control structure in Vista is
WPP_MAIN_CB, else use WPP_GLOBAL_Control

kd> x tracedrv!WPP_MAIN_CB // tracedrv is the WPP instrumented
driver
9fbf3040 tracedrv!WPP_MAIN_CB = union WPP_PROJECT_CONTROL_BLOCK [1]
kd>dt WPP_TRACE_CONTROL_BLOCK 9fbf3040

+0x000 Callback : 0x9fbf127c tracedrv!WppTraceCallback+0
+0x004 ControlGuid : 0x9fbf206c _GUID
{d58c126f-b309-11d1-969e-0000f875a5bc}
+0x008 Next : (null)
+0x010 Logger : 0
+0x018 RegistryPath : (null)
+0x01c FlagsLen : 0x1 ‘’
+0x01d Level : 0x0 ‘’ <— Set the Level
+0x01e Reserved : 0
+0x020 Flags : [1] 0x0 <— Set the Flag

2 - Set the value for the Level and Flags, in our case enable tracing at
level=5 and flags = 0xf

kd>eb 9fbf305d 5 // setting the level value to 5

kd>ed 9fbf3060 0xf // setting the flag value to 0xf

Thanks,

Jose Sua

Microsoft Corporation

This posting is provided “AS IS” with no warranties and confers no
rights.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jan Bottorff
Sent: Thursday, May 11, 2006 3:19 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] Unnecessary obfuscation, “Hey I can’t see my spew
because of your spew!!!”

#define WPP_DEBUG(m)

I know about WPP_DEBUG and didn’t find I liked it. The m parameter is
the format string followed by all the values. I wanted to still be able
to pass through the level+mask, so I could control WHICH DbgPrints came
out. I believe I could modify the WPP template file, but didn’t offhand
see a way to make a local copy of the template file in my projects
directory. It’s more effort to change a file that’s part of the DDK and
is installed on all the other team members systems. The DDK files are
not checked into our version control system, so I have no way to
automagically update them on other team members system. I’d probably get
some frown on modifying a standard DDK file too.

At the moment, I’m using statements like:

DebugPrint((TRACE_LEVEL_ERROR, IRP_MASK, “the value is
%x\n”,someValue));

I can use a -func:DebugPrint((LEVEL,MASK,MSG,…)) option in WPP to turn
this into a WPP message or I can

#define DebugPrint(args) DebugPrintVa args

and have it route to a global function DebugPrintVa(level, mask,
fmt,…) that uses va_start/va_list/va_end and such to build a buffer
that gets DbgPrinted.

I’d really like to be able to say:

#define DebugPrint(args) DebugPrintWPP args; \
DebugPrintVa args

But unfortunately the WPP processor runs before the C preprocessor, so
this does not work. I could possible put a WPP print inside my
DebugPrintVa function, using a formatting string of “MyDriver!%s”, but
this does all the formatting work at message print time, counter to the
WPP ideal.

I supposed the other option is I write ugly code like:

#ifdef WITH_WPP
DebugPrintWPP(TRACE_LEVEL_ERROR, IRP_MASK, “the value is
%x\n”,someValue); #else DebugPrintWPP(TRACE_LEVEL_ERROR, IRP_MASK, “the
value is %x\n”,someValue); DebugPrintVa(TRACE_LEVEL_ERROR, IRP_MASK,
“the value is %x\n”,someValue); #endif

  • Jan

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

>>Have you found any compelling reason to use WPP for anything? :wink:

I used custom logging for a long time. Now I more than happy I switched to
WPP. Yes, it took some time to fix buggy WPP macros, but now it all works
just fine from Win2000 up. It’s very easy to use once the backbone is set up
properly, and the latter is only a one-off task.

The only major problem for me is TraceView shite that gets worse with every
new version. I wish Microsoft published its source code so that I could fix
it myself.

“Michal Vodicka” wrote in message
news:xxxxx@ntdev…
> ----------
> From:
> xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com]
> on behalf of Roddy, Mark[SMTP:xxxxx@stratus.com]
> Reply To: Windows System Software Devs Interest List
> Sent: Thursday, May 11, 2006 3:28 PM
> To: Windows System Software Devs Interest List
> Subject: RE: [ntdev] Unnecessary obfuscation, “Hey I can’t see my spew
> because of your spew!!!”
>
> I’ve haven’t found a compelling reason to use wpp for development.
>
Have you found any compelling reason to use WPP for anything? :wink: Reading
messages about it here I’m glad I decided to not try it. Our trace macros
work well, are more flexible and I only once encountered the situation where
run-time formatting caused that the problem wasn’t reproducible any way. So
I spent 30 minutes to change macros implementation to not use formatting and
instead, store file, line and arguments to circular buffer. User reproduced
problem and invoked crashdump and then I spent some time with manual
extracting necessary info from the buffer. Still probably much faster than
trying to port our trace macros for WPP.

> Both wpp and debugprint suffer from not having a retrievable kernel
> ringbuffer that captures run time trace information for post mortem
> analysis.
>
If you run DebugView in the time of crash, it can extract its buffer with
enough debug prints from crashdump later.

> Runtime configurable debug prints are fine in production drivers, as
> long as the driver ships in terse mode.
>
Not only terse but completely disabled. Usually it is hard to decide what it
terse mode. Very rare error message can start appear thousands times per
second under some unusual circumstances.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

The vista beta2 Traceview is much improved and stable.

Also the WDK templates are much cleaner now and work W2K and Avobe, we
have been wanting to place them in a download somewhere with all the new
updated tools, but have not been able to get to that.

ETW has an extension distributed with the debugger, WMITRACE.dll. The
extension can show the contents of the ETW buffers in the debugger.

!wmitrace.help to get all the options available.
!wmitrace.strdump Dump the Wmi Trace Event Structures
!wmitrace.searchpath sets the search path for TMF

The extension is very useful when you enable tracing at boot time or at
any time and can see the events on your trace session in the debugger.

One way to use this is to start the session in real time mode, tracelog
-rt -kd and then view the events, you must also set the search path
for the TMF files for all this to work. To extract the TMF files use
tracepdb -f .

The extension works very well in Vista because the public symbols
contain the required info for the extension. The problem pre-Vista is
that the public symbols do not export those required ETW symbols for the
extension. The fix that exported the symbols is expected to reach the
down level. I will investigate what the status of that is.

How to Enable Debugging without starting an ETW trace session

First you need to define the WPP_DEBUG macro and build your code.

A sample for a the DDK’s Tracedrv.sys sample driver below:

#define WPP_DEBUG(b) DbgPrint b, DbgPrint(“\n”);

You can use most formats and arguments with WPP_DEBUG. However, you
cannot use extended format specifications, such as %!HEXDUMP!%.

On the Kernel Debugger:

You will need to set the levels and flags values for the WPP control
structure.

1- Locate the address of the WPP control structure in Vista is
WPP_MAIN_CB, else use WPP_GLOBAL_Control

kd> x tracedrv!WPP_MAIN_CB // tracedrv is the WPP instrumented
driver
9fbf3040 tracedrv!WPP_MAIN_CB = union WPP_PROJECT_CONTROL_BLOCK [1]
kd>dt WPP_TRACE_CONTROL_BLOCK 9fbf3040

+0x000 Callback : 0x9fbf127c tracedrv!WppTraceCallback+0
+0x004 ControlGuid : 0x9fbf206c _GUID
{d58c126f-b309-11d1-969e-0000f875a5bc}
+0x008 Next : (null)
+0x010 Logger : 0
+0x018 RegistryPath : (null)
+0x01c FlagsLen : 0x1 ‘’
+0x01d Level : 0x0 ‘’ <— Set the Level
+0x01e Reserved : 0
+0x020 Flags : [1] 0x0 <— Set the Flag

2 - Set the value for the Level and Flags, in our case enable tracing at
level=5 and flags = 0xf

kd>eb 9fbf305d 5 // setting the level value to 5

kd>ed 9fbf3060 0xf // setting the flag value to 0xf

Thanks,

Jose Sua

Microsoft Corporation

This posting is provided “AS IS” with no warranties and confers no
rights.

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of cristalink
Sent: Thursday, May 11, 2006 5:18 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Unnecessary obfuscation, “Hey I can’t see my spew
because of your spew!!!”

>Have you found any compelling reason to use WPP for anything? :wink:

I used custom logging for a long time. Now I more than happy I switched
to WPP. Yes, it took some time to fix buggy WPP macros, but now it all
works just fine from Win2000 up. It’s very easy to use once the backbone
is set up properly, and the latter is only a one-off task.

The only major problem for me is TraceView shite that gets worse with
every new version. I wish Microsoft published its source code so that I
could fix it myself.

“Michal Vodicka” wrote in message
news:xxxxx@ntdev…
> ----------
> From:
> xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.c
> om] on behalf of Roddy, Mark[SMTP:xxxxx@stratus.com] Reply To:
> Windows System Software Devs Interest List
> Sent: Thursday, May 11, 2006 3:28 PM
> To: Windows System Software Devs Interest List
> Subject: RE: [ntdev] Unnecessary obfuscation, “Hey I can’t see my spew

> because of your spew!!!”
>
+740/e haven’t found a compelling reason to use wpp for development.
>
Have you found any compelling reason to use WPP for anything? :wink:
Reading messages about it here I’m glad I decided to not try /t. Our
trace macros work well, are more flexible and I only once encountered
the situation where run-time formatting cau+ed that the problem wasn’t
reproducible any way. So I spent 30 minutes to change macros
implementation to not use formatting and instead, store file, line and
arguments to circular buffer. User reproduced problem and invoked
crashdump and then I spent some time with manual extracting necessary
info from the buffer. Still probably much faster than trying to port our
trace macros for WPP.

> Both wpp and debugprint suffer from not having a retrievable kernel
> ringbuffer that captures run time trace information for post mortem
> analysis.
>
If you run DebugView in the time of crash, it can extract its buffer
with enough debug prints from crashdump later.

> Runtime configurable debug prints are fine in production drivers, as
> long as the driver ships in terse mode.
>
Not only terse but completely disabled. Usually it is hard to decide
what it terse mode. Very rare error message can start appear thousands
times per second under some unusual circumstances.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]


Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer