DDC - updated info for next week

[Lyndon_J_Clarke-2]

Peter
I thank you so much for telling these simple truths. Let;s all say this at
DDC and hope the message gets thru.
Kindest regards, Lyndon

wrote in message news:xxxxx@ntdev…
>I agree with Mr. O’Brien. The focus of the DDC is clearly static analysis,
>because if we’re not using it, that’s obviously because we’re dumb out here
>and slow on the uptake.
>
> The distortion field is clearly alive and well. Sigh!
>
> As usual, Microsoft would be well served by REALLY understanding what we
> DO out here. They THINK they understand, but they don’t.
>
> As I’ve said before, I like and respect the folks who do SDV and PreFast.
> I really do. They’re very smart… most of them smarter than me. But
> they’d just be soooo surprised if they could spend some time, like maybe a
> solid 3 months, out in the real world trying to USE these tools to do
> useful work for pay.
>
> PreFast: Amusing, sometimes useful, but annoying. For maximum
> effectiveness and minimum annoyance have to “buy in” to using it
> throughout the life-cycle of your driver. You have to believe that SAL
> notations are a good thing. I am not convinced.
>
> SDV: Complicated, difficult, and VERY annoying. I have never – not
> once – been able to get SDV to disclose a meaningful error to me. This
> is true even when I try to create specific errors in my code that the
> documentation CLEARLY says SDV will detect and report.
>
> I intellectually understand who/why/where these tools could be useful.
> I suspect that it’ll take about another ten years before I find them
> useful in practical situations, however.
>
> If I were king of the group responsible for these tools, I would redirect
> their efforts to building clever and comprehensive DYNAMIC analysis and
> debugging tools that we can use out here in the real world, without
> significant up-front “adopt it for life or it’s not useful”, requirements.
>
> These dynamic tools are really what the community needs at this point.
> WAY MORE checks in KMDF Verifier. A LOCKING HIERARCHY VALIDATION scheme
> (OSR has a patent pending on one, we’ll be happy to license it) for ALL
> sorts of locks. MORE speculative checking options in Driver Verifier.
> Resurrect Call Usage Verifier – now discontinued. And… just to show
> you how easy it is to miss the simple stuff: WHY are things like
> IoGetNextIrpStackLocation still MACROs?? Why are ANY of the DDK functions
> macros? When they’re macros, they’re impossible to “hook” with Verifier.
> At LEAST make them functions, and mark them as “inline” in the free build.
> There is NO additional overhead to this, and much to be gained. This
> seems obvious to me (and, in fact, was an actual work item for the DDK
> back, oh, about a thousand years ago).
>
> If any of the static analysis folks would like to come to OSR to spend a
> couple of months writing code on real customer projects, and see “how the
> other half lives”, I’ll be happy to provide the office and project
> management. I’ll even give them free lunch for the duration of their stay
> (like every other OSR employee). Maybe they’ll gain some insight. Then
> again, maybe they’ll convince us that PreFast or SDV will make our drivers
> or file systems better and I’ll have to write a big long article in The NT
> Insider about how SDV changed my life.
>
> Peter
> OSR
>

[Bill_McKenzie-3]

You HAVE to have kernel devs on it if you want it to be anywhere in the
ballpark of useful for kernel devs, which it isn’t today. I didn’t say ONLY
kernel devs, but you have to have some folks that actually know the driver
development environment. I am sure they have some bright people on SDV just
like I am sure Netscape and Lotus 1 2 3 had bright people on them. That
doesn’t mean anything. They need to do some serious work on making the tool
useful, I don’t care what kind of backend it has.

AND that tool has been around a LONG LONG time. They have had plenty of
time to make it better, and from what I can tell it has gotten worse. I
can’t even get it to work at all on most of the drivers I have tried it on.

One last thing, I know a lot of driver developers with serious compiler
backgrounds…seems the fields are very closely related as to skill sets to
me?? I personally have messed with parsers a good bit over the last few
years, for wizards and such, but never have created a compiler back end.

Bill M.

“Tim Roberts” wrote in message news:xxxxx@ntdev…
> Bill McKenzie wrote:
>> I actually LOVE static analysis tools. I just don’t understand why they
>> don’t get some real kernel devs on them and tailor them appropriately.
>
> I can answer that question: because kernel devs don’t know squat about
> making a static analysis tool. It’s the wrong skill set. Remember, SDV
> is essentially pass 1 of the Microsoft C++ compiler, with a custom back
> end. You need compiler gurus to write SDV, not driver gurus.
>
> I have a compiler background from my days at Control Data. I was
> really, really impressed when I heard what SDV was doing. There are
> some real eggheads in Microsoft Research. I have no doubt that it will
> get better.
>
> –
> Tim Roberts, xxxxx@probo.com
> Providenza & Boekelheide, Inc.
>
>

[OSR_Community_User]

I agree with you that input from real driver developers would be a good thing, but I’m sure how much of the problem that is at the
moment - it just doesn’t work well enough to be worth using currently.

mm

Bill McKenzie wrote:

You HAVE to have kernel devs on it if you want it to be anywhere in the
ballpark of useful for kernel devs, which it isn’t today. I didn’t say ONLY
kernel devs, but you have to have some folks that actually know the driver
development environment. I am sure they have some bright people on SDV just
like I am sure Netscape and Lotus 1 2 3 had bright people on them. That
doesn’t mean anything. They need to do some serious work on making the tool
useful, I don’t care what kind of backend it has.

AND that tool has been around a LONG LONG time. They have had plenty of
time to make it better, and from what I can tell it has gotten worse. I
can’t even get it to work at all on most of the drivers I have tried it on.

One last thing, I know a lot of driver developers with serious compiler
backgrounds…seems the fields are very closely related as to skill sets to
me?? I personally have messed with parsers a good bit over the last few
years, for wizards and such, but never have created a compiler back end.

Bill M.

“Tim Roberts” wrote in message news:xxxxx@ntdev…
>> Bill McKenzie wrote:
>>> I actually LOVE static analysis tools. I just don’t understand why they
>>> don’t get some real kernel devs on them and tailor them appropriately.
>> I can answer that question: because kernel devs don’t know squat about
>> making a static analysis tool. It’s the wrong skill set. Remember, SDV
>> is essentially pass 1 of the Microsoft C++ compiler, with a custom back
>> end. You need compiler gurus to write SDV, not driver gurus.
>>
>> I have a compiler background from my days at Control Data. I was
>> really, really impressed when I heard what SDV was doing. There are
>> some real eggheads in Microsoft Research. I have no doubt that it will
>> get better.
>>
>> –
>> Tim Roberts, xxxxx@probo.com
>> Providenza & Boekelheide, Inc.
>>
>>
>
>
>

[Bill_McKenzie-3]

Speaking of Microsoft research and drivers this is interesting:

http://channel8.msdn.com/Posts/401/

I would love to know more about how the 4 color theorem was applied in the
proving of driver code.

I wonder if we have to have a massively parallel machine to run these tests

Bill M.

“Bill McKenzie” wrote in message
news:xxxxx@ntdev…
> You HAVE to have kernel devs on it if you want it to be anywhere in the
> ballpark of useful for kernel devs, which it isn’t today. I didn’t say
> ONLY kernel devs, but you have to have some folks that actually know the
> driver development environment. I am sure they have some bright people on
> SDV just like I am sure Netscape and Lotus 1 2 3 had bright people on
> them. That doesn’t mean anything. They need to do some serious work on
> making the tool useful, I don’t care what kind of backend it has.
>
> AND that tool has been around a LONG LONG time. They have had plenty of
> time to make it better, and from what I can tell it has gotten worse. I
> can’t even get it to work at all on most of the drivers I have tried it
> on.
>
> One last thing, I know a lot of driver developers with serious compiler
> backgrounds…seems the fields are very closely related as to skill sets
> to me?? I personally have messed with parsers a good bit over the last
> few years, for wizards and such, but never have created a compiler back
> end.
>
> Bill M.
>
>
> “Tim Roberts” wrote in message news:xxxxx@ntdev…
>> Bill McKenzie wrote:
>>> I actually LOVE static analysis tools. I just don’t understand why they
>>> don’t get some real kernel devs on them and tailor them appropriately.
>>
>> I can answer that question: because kernel devs don’t know squat about
>> making a static analysis tool. It’s the wrong skill set. Remember, SDV
>> is essentially pass 1 of the Microsoft C++ compiler, with a custom back
>> end. You need compiler gurus to write SDV, not driver gurus.
>>
>> I have a compiler background from my days at Control Data. I was
>> really, really impressed when I heard what SDV was doing. There are
>> some real eggheads in Microsoft Research. I have no doubt that it will
>> get better.
>>
>> –
>> Tim Roberts, xxxxx@probo.com
>> Providenza & Boekelheide, Inc.
>>
>>
>
>
>

[Michal_Vodicka-2]

> -----Original Message-----

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Skywing
Sent: Thursday, September 25, 2008 4:22 PM
To: Windows System Software Devs Interest List
Subject: RE: Re:[ntdev] DDC - updated info for next week

Every WinDbg release for the past few releases has
successively broken more and more highly useful debugger
extensons because they are changed to rely on private symbols.

I’d add NT_ASSERT abomination or how it is named which also depends on
private symbols. The macro which generates int 2ch and stores expression
as an annotation in the PDB. It really decreases usefulness of checked
build. One can see there was an assert but can’t see the expression
which caused failure. It was useful in most cases in the past when
RtlAssert was used and now it is gone. For no good reason IMO.

It would be nice if, and my head’s probably in the clouds a
bit here, if for a day, a week, whatever, Microsoft internal
had to only use publicly available debugger extensions, the
public symbol server, etc.

The test could be automated. It just need a set of test data (memory
dumps) and one person who sits and writes a script to test every command
and extension. Tiresome work but needs to be done only once. Then, the
script can be run output saved and verified. When everything is ready,
the script can be used after every WinDbg release and output compared
with the reference one. By human who can understand if a difference is a
bug or not. With a good compare tool it should be quick and easy task.

BTW, I can’t imagine WinDbg development without such test data scripts.
It seems they’re used with private symbols only.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

[Bill_McKenzie-3]

Don,

I really have to disagree on DTM. When was the last time you used it? I
used to have a lot of problems with it, but I find the latest DTM to be a
VERY useful tool. I used it recently as the base of an automated testing
framework for my company’s Windows client software. Using custom scripts I
can test everything from the UI to the drivers in a completely automated
fashion. It is great for controlling multiple platforms and collating
results. Now, I don’t know what its limitations are as far as number of
test platforms, we never used more than say 10. But, all in all for what I
needed the tool was great. The only real problem I had was programmatically
controlling DTM from its external COM interface. THAT is a disaster!

Bill M.

“Don Burn” wrote in message news:xxxxx@ntdev…
> How about scrapping DTM and putting together a real test framework? DTM
> is horribly over engineered for anyone other than WHQL, it just gets in
> the way. For those of us who are small environments, setting it iup is
> horribly complicated.
>
> For firms that have large test environments, it still gets in the way. I
> know a major firm that has pools of hundreds of test machines and their
> own tools to image and run tests, for years when they had a new driver for
> Windows they would use the whole pool to stress test the driver. Now
> thanks to DTM being in the way, they have a pool of 30 systems for Windows
> testing, and reserve the hundreds for other OS’es that do have the DTM
> problems.
>
> DTM IMHO has done more to set back the quality of Windows drivers than
> anything Microsoft has done.
>
>
> –
> Don Burn (MVP, Windows DDK)
> Windows 2k/XP/2k3 Filesystem and Driver Consulting
> Website: http://www.windrvr.com
> Blog: http://msmvps.com/blogs/WinDrvr
> Remove StopSpam to reply
>
>
>
>
> “Bill McKenzie” wrote in message
> news:xxxxx@ntdev…
>>I actually LOVE static analysis tools. I just don’t understand why they
>>don’t get some real kernel devs on them and tailor them appropriately. I
>>think PreFast is a beautiful tool! I don’t even bother with the SAL
>>annotation much (except for the buffer size stuff that is pretty cool) and
>>I still find it terribly useful. I mean, it just catches stupid errors
>>mostly, but that is SO much better to catch at build time and you can
>>easily hook it into the build process to automatically run. The false
>>positives are a good bit annoying, they could trim that a LOT.
>>
>> SDV seems like a great idea, but horribly horribly broken. I, like
>> Peter, have never gotten anything useful out of it. But the idea seems
>> to have a lot of promise. Just needs some serious and directed effort
>> behind it.
>>
>> Run time tools are awesome! I will use all I can get my hands on and I
>> do. But static analysis tools that can be automated into a build are
>> simply beautiful! I wish I had a better handle on just how broad a
>> spectrum of errors those tools could find in a reasonable timeframe given
>> sufficiently sophisticated tools. I really don’t have a good idea where
>> those bounds are. But, if the potential error catching surface is
>> large…I say poor all the resources you can into that!
>>
>> For runtime tools…I would LOVE to see some kind of integration into
>> DTM, so again, we could automate the running and retrieving of results!!
>>
>> For that matter, how about fixing and/or finishing the COM interface to
>> DTM so automating the running of DTM is easier to achieve?
>>
>> Bill M.
>>
>>
>>
>> “Don Burn” wrote in message news:xxxxx@ntdev…
>>> The number of tools that would be nice out of Microsoft is too large to
>>> post, a few over the years I have suggested to them are:
>>>
>>> 1. Code coverage - yes Bullseye is nice, but even with it we need faut
>>> injection. Note they looked at fault injection and rejected it because
>>> people would have to modify their code (I guess annotations don’t count
>>> as modification)
>>>
>>> 2. Lock contention measurement - the number of times I have seen people
>>> propose a gazzilion locks for a driver because they do not want
>>> contention. A tool that shows how long locks are held (max and average)
>>> and how long someone waits for locks would probably slow down some of
>>> these lets have a seperate lock for everything.
>>>
>>> 3. Call usage verifier - this was a godd tool, it could have been
>>> expanded a lot. Come on Redmond either bring it forward, or release it
>>> back to OSR so they can have the OSR WDK (compared to the OSR DDK that
>>> CUV came from).
>>>
>>> Perhaps we need an email list for such tools, complete with a legal
>>> agreement that releases any thing people say, so Redmond or others can
>>> act on it.
>>>
>>>
>>> –
>>> Don Burn (MVP, Windows DDK)
>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>> Website: http://www.windrvr.com
>>> Blog: http://msmvps.com/blogs/WinDrvr
>>> Remove StopSpam to reply
>>>
>>>
>>>
>>>
>>> “Martin O’Brien” wrote in message
>>> news:xxxxx@ntdev…
>>>> +1
>>>>
>>>> I’d add that some complete and accurate WinDbg documentation might be
>>>> nice, as would the same for the KRM.
>>>>
>>>>
>>>> mm
>>>>
>>>> xxxxx@osr.com wrote:
>>>>> I agree with Mr. O’Brien. The focus of the DDC is clearly static
>>>>> analysis, because if we’re not using it, that’s obviously because
>>>>> we’re dumb out here and slow on the uptake.
>>>>>
>>>>> The distortion field is clearly alive and well. Sigh!
>>>>>
>>>>> As usual, Microsoft would be well served by REALLY understanding what
>>>>> we DO out here. They THINK they understand, but they don’t.
>>>>>
>>>>> As I’ve said before, I like and respect the folks who do SDV and
>>>>> PreFast. I really do. They’re very smart… most of them smarter than
>>>>> me. But they’d just be soooo surprised if they could spend some time,
>>>>> like maybe a solid 3 months, out in the real world trying to USE these
>>>>> tools to do useful work for pay.
>>>>>
>>>>> PreFast: Amusing, sometimes useful, but annoying. For maximum
>>>>> effectiveness and minimum annoyance have to “buy in” to using it
>>>>> throughout the life-cycle of your driver. You have to believe that
>>>>> SAL notations are a good thing. I am not convinced.
>>>>>
>>>>> SDV: Complicated, difficult, and VERY annoying. I have never – not
>>>>> once – been able to get SDV to disclose a meaningful error to me.
>>>>> This is true even when I try to create specific errors in my code that
>>>>> the documentation CLEARLY says SDV will detect and report.
>>>>>
>>>>> I intellectually understand who/why/where these tools could be
>>>>> useful. I suspect that it’ll take about another ten years before I
>>>>> find them useful in practical situations, however.
>>>>>
>>>>> If I were king of the group responsible for these tools, I would
>>>>> redirect their efforts to building clever and comprehensive DYNAMIC
>>>>> analysis and debugging tools that we can use out here in the real
>>>>> world, without significant up-front “adopt it for life or it’s not
>>>>> useful”, requirements.
>>>>>
>>>>> These dynamic tools are really what the community needs at this point.
>>>>> WAY MORE checks in KMDF Verifier. A LOCKING HIERARCHY VALIDATION
>>>>> scheme (OSR has a patent pending on one, we’ll be happy to license it)
>>>>> for ALL sorts of locks. MORE speculative checking options in Driver
>>>>> Verifier. Resurrect Call Usage Verifier – now discontinued. And…
>>>>> just to show you how easy it is to miss the simple stuff: WHY are
>>>>> things like IoGetNextIrpStackLocation still MACROs?? Why are ANY of
>>>>> the DDK functions macros? When they’re macros, they’re impossible to
>>>>> “hook” with Verifier. At LEAST make them functions, and mark them as
>>>>> “inline” in the free build. There is NO additional overhead to this,
>>>>> and much to be gained. This seems obvious to me (and, in fact, was an
>>>>> actual work item for the DDK back, oh, about a thousand years ago).
>>>>>
>>>>> If any of the static analysis folks would like to come to OSR to spend
>>>>> a couple of months writing code on real customer projects, and see
>>>>> “how the other half lives”, I’ll be happy to provide the office and
>>>>> project management. I’ll even give them free lunch for the duration
>>>>> of their stay (like every other OSR employee). Maybe they’ll gain
>>>>> some insight. Then again, maybe they’ll convince us that PreFast or
>>>>> SDV will make our drivers or file systems better and I’ll have to
>>>>> write a big long article in The NT Insider about how SDV changed my
>>>>> life.
>>>>>
>>>>> Peter
>>>>> OSR
>>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>

[Scott_Noone_OSR]

>It really decreases usefulness of checked build.

Agreed. And I’d add to that the widespread use of WPP, which makes getting
any tracing out of checked in box drivers impossible (unless there’s a
server with all of the necessary TMF files that I’m misssing someplace).

-scott

–
Scott Noone
Software Engineer
OSR Open Systems Resources, Inc.
http://www.osronline.com

“Michal Vodicka” wrote in message
news:xxxxx@ntdev…
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Skywing
> Sent: Thursday, September 25, 2008 4:22 PM
> To: Windows System Software Devs Interest List
> Subject: RE: Re:[ntdev] DDC - updated info for next week
>
> Every WinDbg release for the past few releases has
> successively broken more and more highly useful debugger
> extensons because they are changed to rely on private symbols.

I’d add NT_ASSERT abomination or how it is named which also depends on
private symbols. The macro which generates int 2ch and stores expression
as an annotation in the PDB. It really decreases usefulness of checked
build. One can see there was an assert but can’t see the expression
which caused failure. It was useful in most cases in the past when
RtlAssert was used and now it is gone. For no good reason IMO.

> It would be nice if, and my head’s probably in the clouds a
> bit here, if for a day, a week, whatever, Microsoft internal
> had to only use publicly available debugger extensions, the
> public symbol server, etc.

The test could be automated. It just need a set of test data (memory
dumps) and one person who sits and writes a script to test every command
and extension. Tiresome work but needs to be done only once. Then, the
script can be run output saved and verified. When everything is ready,
the script can be used after every WinDbg release and output compared
with the reference one. By human who can understand if a difference is a
bug or not. With a good compare tool it should be quick and easy task.

BTW, I can’t imagine WinDbg development without such test data scripts.
It seems they’re used with private symbols only.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

[Ken_Johnson]

Maybe we’re being encouraged to use psychic debugging skills more often.

• S

-----Original Message-----
From: Scott Noone
Sent: Friday, September 26, 2008 07:06
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Re:DDC - updated info for next week

>It really decreases usefulness of checked build.

Agreed. And I’d add to that the widespread use of WPP, which makes getting
any tracing out of checked in box drivers impossible (unless there’s a
server with all of the necessary TMF files that I’m misssing someplace).

-scott

–
Scott Noone
Software Engineer
OSR Open Systems Resources, Inc.
http://www.osronline.com

“Michal Vodicka” wrote in message
news:xxxxx@ntdev…
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Skywing
> Sent: Thursday, September 25, 2008 4:22 PM
> To: Windows System Software Devs Interest List
> Subject: RE: Re:[ntdev] DDC - updated info for next week
>
> Every WinDbg release for the past few releases has
> successively broken more and more highly useful debugger
> extensons because they are changed to rely on private symbols.

I’d add NT_ASSERT abomination or how it is named which also depends on
private symbols. The macro which generates int 2ch and stores expression
as an annotation in the PDB. It really decreases usefulness of checked
build. One can see there was an assert but can’t see the expression
which caused failure. It was useful in most cases in the past when
RtlAssert was used and now it is gone. For no good reason IMO.

> It would be nice if, and my head’s probably in the clouds a
> bit here, if for a day, a week, whatever, Microsoft internal
> had to only use publicly available debugger extensions, the
> public symbol server, etc.

The test could be automated. It just need a set of test data (memory
dumps) and one person who sits and writes a script to test every command
and extension. Tiresome work but needs to be done only once. Then, the
script can be run output saved and verified. When everything is ready,
the script can be used after every WinDbg release and output compared
with the reference one. By human who can understand if a difference is a
bug or not. With a good compare tool it should be quick and easy task.

BTW, I can’t imagine WinDbg development without such test data scripts.
It seems they’re used with private symbols only.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

[Don_Burn_1]

Bill,

To be honest I no longer directly use DTM. I used to try to run it but
have given up, since I have to tear my whole test lab apart to even try.
What I do have is a number of customers who I work with who run it, and from
their experiences I no longer go there. In particular:

One large firm, who has set aside the hardware and the MIS resources.
They don’t have too many problems, but have commented that every so often a
hotfix for the OS destabilizes the world. Their normal approach is to then
use MIS to figure out which hotfix and reconfigure the systems.

One medium size firm that pays a consulting firm to do it. They have
a fixed locked down test network and things mostly work.

Several small firms, that assemble test configurations when they need
to. Most of them swear they would become pure LINUX if the market could
support them, after each attempt at DTM. The ones who have very large
servers and setup a domain controller find it painful but doable. The ones
who have small systems and do not want a domain controller spend weeks (I’ve
been there with them through it) and sometimes finally say “to hell with
Redmond” and move on planning to come back in 6 months to a year.

I used to run HCT’s for the clients and ensure that the drivers would
pass WHQL when they got them. I no longer do because of the problems with
DTM. But DTM is actually a reflection of the bigger problem with WHQL and
driver testing. For the last 10 years at least WHQL has continued to
modify the test environment first HCT’s then DTM to make their life easer at
the expense of driver quality. The first principal of quality if find bugs
as early in the process as you can, but WHQL testing continues to evolve to
more and more complex environments which push the use of the tests later and
later into the cycle.

If Microsoft spent as much money on good tests for drivers as it has
in the last 10 years making WHQL testing more automated for them we would
have a lot more tests that driver writers could use, and a related increase
in the quality of drivers. Instead as we go forward they make the tests
more and more useless, for instance the fact that the various test harnesses
from HCT10 on do not run on a checked build to me is ridiculous.

At this point, if Redmond want driver quality the simplest first step
would be to fire everyone in WHQL and start over.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

“Bill McKenzie” wrote in message
news:xxxxx@ntdev…
> Don,
>
> I really have to disagree on DTM. When was the last time you used it? I
> used to have a lot of problems with it, but I find the latest DTM to be a
> VERY useful tool. I used it recently as the base of an automated testing
> framework for my company’s Windows client software. Using custom scripts
> I can test everything from the UI to the drivers in a completely automated
> fashion. It is great for controlling multiple platforms and collating
> results. Now, I don’t know what its limitations are as far as number of
> test platforms, we never used more than say 10. But, all in all for what
> I needed the tool was great. The only real problem I had was
> programmatically controlling DTM from its external COM interface. THAT is
> a disaster!
>
> Bill M.
>
> “Don Burn” wrote in message news:xxxxx@ntdev…
>> How about scrapping DTM and putting together a real test framework? DTM
>> is horribly over engineered for anyone other than WHQL, it just gets in
>> the way. For those of us who are small environments, setting it iup is
>> horribly complicated.
>>
>> For firms that have large test environments, it still gets in the way. I
>> know a major firm that has pools of hundreds of test machines and their
>> own tools to image and run tests, for years when they had a new driver
>> for Windows they would use the whole pool to stress test the driver. Now
>> thanks to DTM being in the way, they have a pool of 30 systems for
>> Windows testing, and reserve the hundreds for other OS’es that do have
>> the DTM problems.
>>
>> DTM IMHO has done more to set back the quality of Windows drivers than
>> anything Microsoft has done.
>>
>>
>> –
>> Don Burn (MVP, Windows DDK)
>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>> Website: http://www.windrvr.com
>> Blog: http://msmvps.com/blogs/WinDrvr
>> Remove StopSpam to reply
>>
>>
>>
>>
>> “Bill McKenzie” wrote in message
>> news:xxxxx@ntdev…
>>>I actually LOVE static analysis tools. I just don’t understand why they
>>>don’t get some real kernel devs on them and tailor them appropriately. I
>>>think PreFast is a beautiful tool! I don’t even bother with the SAL
>>>annotation much (except for the buffer size stuff that is pretty cool)
>>>and I still find it terribly useful. I mean, it just catches stupid
>>>errors mostly, but that is SO much better to catch at build time and you
>>>can easily hook it into the build process to automatically run. The
>>>false positives are a good bit annoying, they could trim that a LOT.
>>>
>>> SDV seems like a great idea, but horribly horribly broken. I, like
>>> Peter, have never gotten anything useful out of it. But the idea seems
>>> to have a lot of promise. Just needs some serious and directed effort
>>> behind it.
>>>
>>> Run time tools are awesome! I will use all I can get my hands on and I
>>> do. But static analysis tools that can be automated into a build are
>>> simply beautiful! I wish I had a better handle on just how broad a
>>> spectrum of errors those tools could find in a reasonable timeframe
>>> given sufficiently sophisticated tools. I really don’t have a good idea
>>> where those bounds are. But, if the potential error catching surface is
>>> large…I say poor all the resources you can into that!
>>>
>>> For runtime tools…I would LOVE to see some kind of integration into
>>> DTM, so again, we could automate the running and retrieving of results!!
>>>
>>> For that matter, how about fixing and/or finishing the COM interface to
>>> DTM so automating the running of DTM is easier to achieve?
>>>
>>> Bill M.
>>>
>>>
>>>
>>> “Don Burn” wrote in message news:xxxxx@ntdev…
>>>> The number of tools that would be nice out of Microsoft is too large to
>>>> post, a few over the years I have suggested to them are:
>>>>
>>>> 1. Code coverage - yes Bullseye is nice, but even with it we need faut
>>>> injection. Note they looked at fault injection and rejected it because
>>>> people would have to modify their code (I guess annotations don’t count
>>>> as modification)
>>>>
>>>> 2. Lock contention measurement - the number of times I have seen
>>>> people propose a gazzilion locks for a driver because they do not want
>>>> contention. A tool that shows how long locks are held (max and average)
>>>> and how long someone waits for locks would probably slow down some of
>>>> these lets have a seperate lock for everything.
>>>>
>>>> 3. Call usage verifier - this was a godd tool, it could have been
>>>> expanded a lot. Come on Redmond either bring it forward, or release it
>>>> back to OSR so they can have the OSR WDK (compared to the OSR DDK that
>>>> CUV came from).
>>>>
>>>> Perhaps we need an email list for such tools, complete with a legal
>>>> agreement that releases any thing people say, so Redmond or others can
>>>> act on it.
>>>>
>>>>
>>>> –
>>>> Don Burn (MVP, Windows DDK)
>>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>>> Website: http://www.windrvr.com
>>>> Blog: http://msmvps.com/blogs/WinDrvr
>>>> Remove StopSpam to reply
>>>>
>>>>
>>>>
>>>>
>>>> “Martin O’Brien” wrote in message
>>>> news:xxxxx@ntdev…
>>>>> +1
>>>>>
>>>>> I’d add that some complete and accurate WinDbg documentation might be
>>>>> nice, as would the same for the KRM.
>>>>>
>>>>>
>>>>> mm
>>>>>
>>>>> xxxxx@osr.com wrote:
>>>>>> I agree with Mr. O’Brien. The focus of the DDC is clearly static
>>>>>> analysis, because if we’re not using it, that’s obviously because
>>>>>> we’re dumb out here and slow on the uptake.
>>>>>>
>>>>>> The distortion field is clearly alive and well. Sigh!
>>>>>>
>>>>>> As usual, Microsoft would be well served by REALLY understanding what
>>>>>> we DO out here. They THINK they understand, but they don’t.
>>>>>>
>>>>>> As I’ve said before, I like and respect the folks who do SDV and
>>>>>> PreFast. I really do. They’re very smart… most of them smarter
>>>>>> than me. But they’d just be soooo surprised if they could spend some
>>>>>> time, like maybe a solid 3 months, out in the real world trying to
>>>>>> USE these tools to do useful work for pay.
>>>>>>
>>>>>> PreFast: Amusing, sometimes useful, but annoying. For maximum
>>>>>> effectiveness and minimum annoyance have to “buy in” to using it
>>>>>> throughout the life-cycle of your driver. You have to believe that
>>>>>> SAL notations are a good thing. I am not convinced.
>>>>>>
>>>>>> SDV: Complicated, difficult, and VERY annoying. I have never – not
>>>>>> once – been able to get SDV to disclose a meaningful error to me.
>>>>>> This is true even when I try to create specific errors in my code
>>>>>> that the documentation CLEARLY says SDV will detect and report.
>>>>>>
>>>>>> I intellectually understand who/why/where these tools could be
>>>>>> useful. I suspect that it’ll take about another ten years before I
>>>>>> find them useful in practical situations, however.
>>>>>>
>>>>>> If I were king of the group responsible for these tools, I would
>>>>>> redirect their efforts to building clever and comprehensive DYNAMIC
>>>>>> analysis and debugging tools that we can use out here in the real
>>>>>> world, without significant up-front “adopt it for life or it’s not
>>>>>> useful”, requirements.
>>>>>>
>>>>>> These dynamic tools are really what the community needs at this
>>>>>> point. WAY MORE checks in KMDF Verifier. A LOCKING HIERARCHY
>>>>>> VALIDATION scheme (OSR has a patent pending on one, we’ll be happy to
>>>>>> license it) for ALL sorts of locks. MORE speculative checking options
>>>>>> in Driver Verifier. Resurrect Call Usage Verifier – now
>>>>>> discontinued. And… just to show you how easy it is to miss the
>>>>>> simple stuff: WHY are things like IoGetNextIrpStackLocation still
>>>>>> MACROs?? Why are ANY of the DDK functions macros? When they’re
>>>>>> macros, they’re impossible to “hook” with Verifier. At LEAST make
>>>>>> them functions, and mark them as “inline” in the free build. There
>>>>>> is NO additional overhead to this, and much to be gained. This seems
>>>>>> obvious to me (and, in fact, was an actual work item for the DDK
>>>>>> back, oh, about a thousand years ago).
>>>>>>
>>>>>> If any of the static analysis folks would like to come to OSR to
>>>>>> spend a couple of months writing code on real customer projects, and
>>>>>> see “how the other half lives”, I’ll be happy to provide the office
>>>>>> and project management. I’ll even give them free lunch for the
>>>>>> duration of their stay (like every other OSR employee). Maybe
>>>>>> they’ll gain some insight. Then again, maybe they’ll convince us that
>>>>>> PreFast or SDV will make our drivers or file systems better and I’ll
>>>>>> have to write a big long article in The NT Insider about how SDV
>>>>>> changed my life.
>>>>>>
>>>>>> Peter
>>>>>> OSR
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>

[Peter_Viscarola_OSR]

add NT_ASSERT abomination

and

…the widespread use of WPP…

Yes. Absolutely.

BOTH of these things (NT_ASSERT in place of ASSERT and freakin’ WPP tracing) have had a DRASTIC effect on our ability to debug edge-condition situations, and have dramatically reduced the usefulness of the checked build.

While it’s not like this is the first time we’ve mentioned these two issues, we, as a community, haven’t been nearly vocal enough about our objections to either of these.

To be clear (in case anybody from msft who cares is reading), the issues are:

1. Replacing ASSERT with NT_ASSERT makes the output from the assert not meaningful without full symbols.

Back before NT_ASSERT, somebody then on the DDK team (me, actually) went through the checked build of the OS and made a list of all the significant asserts in the checked build (their text, and module names), described what the assert meant, and put that description into the DDK docs.

With the introduction of NT_ASSERT, we lost all that information.

You guys could fix this by inserting a DbgPrint before each NT_ASSERT in the checked build (only). Or, just change the freakin’ NT_ASSERT macro.

2. Drivers and even OS components now use WPP tracing, in place of custom-build DbgPrint schemes, results in the trace output being useless to us, because we don’t have access to the TMF files for these components. Before WPP tracing, somebody in PSS (it was last Eliyas, back in those days) would go through the most useful stuff in the checked build and publish the “debug mask” variable names and bit settings to be used to turn out verbose output. This was IMMEASURABLY helpful in debugging custom-written drivers.

Now that these components use WPP, there’s no way for us to see this same output. Sure, we can enable tracing, but without the TMF files we can’t decode the output.

I’ve asked about this, repeatedly, both BEFORE the move to WPP and as recently as about a year ago. And all I’ve EVER heard folks say is “Oh, yeah… That IS a problem. Hmmm… I wonder if we can ship those TMFs… I’ll look into that.” The problem is, nobody’s ever been willing take this on as an action item and expend some political capital to try to make it happen (and, we all know that’s what it would take to get it done).

NOT addressing these issues demonstrates exactly what some of us have been saying in this thread: The msft folks don’t really understand what it’s like to be out here, developing drivers for Windows. They don’t fully understand what we need. They might think they know, they might have some good ideas, but not being in our position it’s really hard for them to KNOW what our jobs are like and what tools/facilities/changes would help us.

Sigh!

Peter
OSR

[Bill_McKenzie-3]

Again, I tried DTM, I don’t know about 3 or 4 years ago, and it was an
abortion. Amazingly, however, they REALLY have improved it to the point
where I don’t think I want to live without it now.

As far as HCT vs DTM, I don’t even think there is a comparison. What used to
take me days or even sometimes weeks with HCT tests (IF I could get a magic
setup to ever work with HCTs and dedicate that magic setup to nothing but
HCTs for the rest of its life and pray it never breaks) I can do in hours
with DTM.

DTM isn’t bug free by a long shot, but I find it orders of magnitude easier
to setup and use than HCTs. Maybe it is the types of drivers we are
testing?

At any rate, if you haven’t tried the latest, you haven’t used DTM…it is
a LOT better now than even just one verion ago.

Bill M.

“Don Burn” wrote in message news:xxxxx@ntdev…
> Bill,
>
> To be honest I no longer directly use DTM. I used to try to run it
> but have given up, since I have to tear my whole test lab apart to even
> try. What I do have is a number of customers who I work with who run it,
> and from their experiences I no longer go there. In particular:
>
> One large firm, who has set aside the hardware and the MIS resources.
> They don’t have too many problems, but have commented that every so often
> a hotfix for the OS destabilizes the world. Their normal approach is to
> then use MIS to figure out which hotfix and reconfigure the systems.
>
> One medium size firm that pays a consulting firm to do it. They have
> a fixed locked down test network and things mostly work.
>
> Several small firms, that assemble test configurations when they
> need to. Most of them swear they would become pure LINUX if the market
> could support them, after each attempt at DTM. The ones who have very
> large servers and setup a domain controller find it painful but doable.
> The ones who have small systems and do not want a domain controller spend
> weeks (I’ve been there with them through it) and sometimes finally say “to
> hell with Redmond” and move on planning to come back in 6 months to a
> year.
>
> I used to run HCT’s for the clients and ensure that the drivers would
> pass WHQL when they got them. I no longer do because of the problems with
> DTM. But DTM is actually a reflection of the bigger problem with WHQL and
> driver testing. For the last 10 years at least WHQL has continued to
> modify the test environment first HCT’s then DTM to make their life easer
> at the expense of driver quality. The first principal of quality if find
> bugs as early in the process as you can, but WHQL testing continues to
> evolve to more and more complex environments which push the use of the
> tests later and later into the cycle.
>
> If Microsoft spent as much money on good tests for drivers as it has
> in the last 10 years making WHQL testing more automated for them we would
> have a lot more tests that driver writers could use, and a related
> increase in the quality of drivers. Instead as we go forward they make
> the tests more and more useless, for instance the fact that the various
> test harnesses from HCT10 on do not run on a checked build to me is
> ridiculous.
>
> At this point, if Redmond want driver quality the simplest first step
> would be to fire everyone in WHQL and start over.
>
>
> –
> Don Burn (MVP, Windows DDK)
> Windows 2k/XP/2k3 Filesystem and Driver Consulting
> Website: http://www.windrvr.com
> Blog: http://msmvps.com/blogs/WinDrvr
>
>
> “Bill McKenzie” wrote in message
> news:xxxxx@ntdev…
>> Don,
>>
>> I really have to disagree on DTM. When was the last time you used it? I
>> used to have a lot of problems with it, but I find the latest DTM to be a
>> VERY useful tool. I used it recently as the base of an automated testing
>> framework for my company’s Windows client software. Using custom
>> scripts I can test everything from the UI to the drivers in a completely
>> automated fashion. It is great for controlling multiple platforms and
>> collating results. Now, I don’t know what its limitations are as far as
>> number of test platforms, we never used more than say 10. But, all in
>> all for what I needed the tool was great. The only real problem I had
>> was programmatically controlling DTM from its external COM interface.
>> THAT is a disaster!
>>
>> Bill M.
>>
>> “Don Burn” wrote in message news:xxxxx@ntdev…
>>> How about scrapping DTM and putting together a real test framework? DTM
>>> is horribly over engineered for anyone other than WHQL, it just gets in
>>> the way. For those of us who are small environments, setting it iup is
>>> horribly complicated.
>>>
>>> For firms that have large test environments, it still gets in the way.
>>> I know a major firm that has pools of hundreds of test machines and
>>> their own tools to image and run tests, for years when they had a new
>>> driver for Windows they would use the whole pool to stress test the
>>> driver. Now thanks to DTM being in the way, they have a pool of 30
>>> systems for Windows testing, and reserve the hundreds for other OS’es
>>> that do have the DTM problems.
>>>
>>> DTM IMHO has done more to set back the quality of Windows drivers than
>>> anything Microsoft has done.
>>>
>>>
>>> –
>>> Don Burn (MVP, Windows DDK)
>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>> Website: http://www.windrvr.com
>>> Blog: http://msmvps.com/blogs/WinDrvr
>>> Remove StopSpam to reply
>>>
>>>
>>>
>>>
>>> “Bill McKenzie” wrote in message
>>> news:xxxxx@ntdev…
>>>>I actually LOVE static analysis tools. I just don’t understand why they
>>>>don’t get some real kernel devs on them and tailor them appropriately.
>>>>I think PreFast is a beautiful tool! I don’t even bother with the SAL
>>>>annotation much (except for the buffer size stuff that is pretty cool)
>>>>and I still find it terribly useful. I mean, it just catches stupid
>>>>errors mostly, but that is SO much better to catch at build time and you
>>>>can easily hook it into the build process to automatically run. The
>>>>false positives are a good bit annoying, they could trim that a LOT.
>>>>
>>>> SDV seems like a great idea, but horribly horribly broken. I, like
>>>> Peter, have never gotten anything useful out of it. But the idea
>>>> seems to have a lot of promise. Just needs some serious and directed
>>>> effort behind it.
>>>>
>>>> Run time tools are awesome! I will use all I can get my hands on and I
>>>> do. But static analysis tools that can be automated into a build are
>>>> simply beautiful! I wish I had a better handle on just how broad a
>>>> spectrum of errors those tools could find in a reasonable timeframe
>>>> given sufficiently sophisticated tools. I really don’t have a good
>>>> idea where those bounds are. But, if the potential error catching
>>>> surface is large…I say poor all the resources you can into that!
>>>>
>>>> For runtime tools…I would LOVE to see some kind of integration into
>>>> DTM, so again, we could automate the running and retrieving of
>>>> results!!
>>>>
>>>> For that matter, how about fixing and/or finishing the COM interface to
>>>> DTM so automating the running of DTM is easier to achieve?
>>>>
>>>> Bill M.
>>>>
>>>>
>>>>
>>>> “Don Burn” wrote in message news:xxxxx@ntdev…
>>>>> The number of tools that would be nice out of Microsoft is too large
>>>>> to post, a few over the years I have suggested to them are:
>>>>>
>>>>> 1. Code coverage - yes Bullseye is nice, but even with it we need
>>>>> faut injection. Note they looked at fault injection and rejected it
>>>>> because people would have to modify their code (I guess annotations
>>>>> don’t count as modification)
>>>>>
>>>>> 2. Lock contention measurement - the number of times I have seen
>>>>> people propose a gazzilion locks for a driver because they do not want
>>>>> contention. A tool that shows how long locks are held (max and
>>>>> average) and how long someone waits for locks would probably slow down
>>>>> some of these lets have a seperate lock for everything.
>>>>>
>>>>> 3. Call usage verifier - this was a godd tool, it could have been
>>>>> expanded a lot. Come on Redmond either bring it forward, or release
>>>>> it back to OSR so they can have the OSR WDK (compared to the OSR DDK
>>>>> that CUV came from).
>>>>>
>>>>> Perhaps we need an email list for such tools, complete with a legal
>>>>> agreement that releases any thing people say, so Redmond or others can
>>>>> act on it.
>>>>>
>>>>>
>>>>> –
>>>>> Don Burn (MVP, Windows DDK)
>>>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>>>> Website: http://www.windrvr.com
>>>>> Blog: http://msmvps.com/blogs/WinDrvr
>>>>> Remove StopSpam to reply
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> “Martin O’Brien” wrote in message
>>>>> news:xxxxx@ntdev…
>>>>>> +1
>>>>>>
>>>>>> I’d add that some complete and accurate WinDbg documentation might be
>>>>>> nice, as would the same for the KRM.
>>>>>>
>>>>>>
>>>>>> mm
>>>>>>
>>>>>> xxxxx@osr.com wrote:
>>>>>>> I agree with Mr. O’Brien. The focus of the DDC is clearly static
>>>>>>> analysis, because if we’re not using it, that’s obviously because
>>>>>>> we’re dumb out here and slow on the uptake.
>>>>>>>
>>>>>>> The distortion field is clearly alive and well. Sigh!
>>>>>>>
>>>>>>> As usual, Microsoft would be well served by REALLY understanding
>>>>>>> what we DO out here. They THINK they understand, but they don’t.
>>>>>>>
>>>>>>> As I’ve said before, I like and respect the folks who do SDV and
>>>>>>> PreFast. I really do. They’re very smart… most of them smarter
>>>>>>> than me. But they’d just be soooo surprised if they could spend
>>>>>>> some time, like maybe a solid 3 months, out in the real world trying
>>>>>>> to USE these tools to do useful work for pay.
>>>>>>>
>>>>>>> PreFast: Amusing, sometimes useful, but annoying. For maximum
>>>>>>> effectiveness and minimum annoyance have to “buy in” to using it
>>>>>>> throughout the life-cycle of your driver. You have to believe that
>>>>>>> SAL notations are a good thing. I am not convinced.
>>>>>>>
>>>>>>> SDV: Complicated, difficult, and VERY annoying. I have never – not
>>>>>>> once – been able to get SDV to disclose a meaningful error to me.
>>>>>>> This is true even when I try to create specific errors in my code
>>>>>>> that the documentation CLEARLY says SDV will detect and report.
>>>>>>>
>>>>>>> I intellectually understand who/why/where these tools could be
>>>>>>> useful. I suspect that it’ll take about another ten years before I
>>>>>>> find them useful in practical situations, however.
>>>>>>>
>>>>>>> If I were king of the group responsible for these tools, I would
>>>>>>> redirect their efforts to building clever and comprehensive DYNAMIC
>>>>>>> analysis and debugging tools that we can use out here in the real
>>>>>>> world, without significant up-front “adopt it for life or it’s not
>>>>>>> useful”, requirements.
>>>>>>>
>>>>>>> These dynamic tools are really what the community needs at this
>>>>>>> point. WAY MORE checks in KMDF Verifier. A LOCKING HIERARCHY
>>>>>>> VALIDATION scheme (OSR has a patent pending on one, we’ll be happy
>>>>>>> to license it) for ALL sorts of locks. MORE speculative checking
>>>>>>> options in Driver Verifier. Resurrect Call Usage Verifier – now
>>>>>>> discontinued. And… just to show you how easy it is to miss the
>>>>>>> simple stuff: WHY are things like IoGetNextIrpStackLocation still
>>>>>>> MACROs?? Why are ANY of the DDK functions macros? When they’re
>>>>>>> macros, they’re impossible to “hook” with Verifier. At LEAST make
>>>>>>> them functions, and mark them as “inline” in the free build. There
>>>>>>> is NO additional overhead to this, and much to be gained. This
>>>>>>> seems obvious to me (and, in fact, was an actual work item for the
>>>>>>> DDK back, oh, about a thousand years ago).
>>>>>>>
>>>>>>> If any of the static analysis folks would like to come to OSR to
>>>>>>> spend a couple of months writing code on real customer projects, and
>>>>>>> see “how the other half lives”, I’ll be happy to provide the office
>>>>>>> and project management. I’ll even give them free lunch for the
>>>>>>> duration of their stay (like every other OSR employee). Maybe
>>>>>>> they’ll gain some insight. Then again, maybe they’ll convince us
>>>>>>> that PreFast or SDV will make our drivers or file systems better and
>>>>>>> I’ll have to write a big long article in The NT Insider about how
>>>>>>> SDV changed my life.
>>>>>>>
>>>>>>> Peter
>>>>>>> OSR
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>

[Don_Burn_1]

I’VE DEALT WITH THAT PIECE OF SHIT KNOWN AS THE LATEST WLK WITH MY CUSTOMERS
MY OPINION STANDS.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

“Bill McKenzie” wrote in message
news:xxxxx@ntdev…
> Again, I tried DTM, I don’t know about 3 or 4 years ago, and it was an
> abortion. Amazingly, however, they REALLY have improved it to the point
> where I don’t think I want to live without it now.
>
> As far as HCT vs DTM, I don’t even think there is a comparison. What used
> to take me days or even sometimes weeks with HCT tests (IF I could get a
> magic setup to ever work with HCTs and dedicate that magic setup to
> nothing but HCTs for the rest of its life and pray it never breaks) I can
> do in hours with DTM.
>
> DTM isn’t bug free by a long shot, but I find it orders of magnitude
> easier to setup and use than HCTs. Maybe it is the types of drivers we
> are testing?
>
> At any rate, if you haven’t tried the latest, you haven’t used DTM…it
> is a LOT better now than even just one verion ago.
>
> Bill M.
>
> “Don Burn” wrote in message news:xxxxx@ntdev…
>> Bill,
>>
>> To be honest I no longer directly use DTM. I used to try to run it
>> but have given up, since I have to tear my whole test lab apart to even
>> try. What I do have is a number of customers who I work with who run it,
>> and from their experiences I no longer go there. In particular:
>>
>> One large firm, who has set aside the hardware and the MIS
>> resources. They don’t have too many problems, but have commented that
>> every so often a hotfix for the OS destabilizes the world. Their normal
>> approach is to then use MIS to figure out which hotfix and reconfigure
>> the systems.
>>
>> One medium size firm that pays a consulting firm to do it. They
>> have a fixed locked down test network and things mostly work.
>>
>> Several small firms, that assemble test configurations when they
>> need to. Most of them swear they would become pure LINUX if the market
>> could support them, after each attempt at DTM. The ones who have very
>> large servers and setup a domain controller find it painful but doable.
>> The ones who have small systems and do not want a domain controller spend
>> weeks (I’ve been there with them through it) and sometimes finally say
>> “to hell with Redmond” and move on planning to come back in 6 months to a
>> year.
>>
>> I used to run HCT’s for the clients and ensure that the drivers
>> would pass WHQL when they got them. I no longer do because of the
>> problems with DTM. But DTM is actually a reflection of the bigger
>> problem with WHQL and driver testing. For the last 10 years at least
>> WHQL has continued to modify the test environment first HCT’s then DTM to
>> make their life easer at the expense of driver quality. The first
>> principal of quality if find bugs as early in the process as you can, but
>> WHQL testing continues to evolve to more and more complex environments
>> which push the use of the tests later and later into the cycle.
>>
>> If Microsoft spent as much money on good tests for drivers as it has
>> in the last 10 years making WHQL testing more automated for them we would
>> have a lot more tests that driver writers could use, and a related
>> increase in the quality of drivers. Instead as we go forward they make
>> the tests more and more useless, for instance the fact that the various
>> test harnesses from HCT10 on do not run on a checked build to me is
>> ridiculous.
>>
>> At this point, if Redmond want driver quality the simplest first
>> step would be to fire everyone in WHQL and start over.
>>
>>
>> –
>> Don Burn (MVP, Windows DDK)
>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>> Website: http://www.windrvr.com
>> Blog: http://msmvps.com/blogs/WinDrvr
>>
>>
>> “Bill McKenzie” wrote in message
>> news:xxxxx@ntdev…
>>> Don,
>>>
>>> I really have to disagree on DTM. When was the last time you used it?
>>> I used to have a lot of problems with it, but I find the latest DTM to
>>> be a VERY useful tool. I used it recently as the base of an automated
>>> testing framework for my company’s Windows client software. Using
>>> custom scripts I can test everything from the UI to the drivers in a
>>> completely automated fashion. It is great for controlling multiple
>>> platforms and collating results. Now, I don’t know what its limitations
>>> are as far as number of test platforms, we never used more than say 10.
>>> But, all in all for what I needed the tool was great. The only real
>>> problem I had was programmatically controlling DTM from its external COM
>>> interface. THAT is a disaster!
>>>
>>> Bill M.
>>>
>>> “Don Burn” wrote in message news:xxxxx@ntdev…
>>>> How about scrapping DTM and putting together a real test framework?
>>>> DTM is horribly over engineered for anyone other than WHQL, it just
>>>> gets in the way. For those of us who are small environments, setting
>>>> it iup is horribly complicated.
>>>>
>>>> For firms that have large test environments, it still gets in the way.
>>>> I know a major firm that has pools of hundreds of test machines and
>>>> their own tools to image and run tests, for years when they had a new
>>>> driver for Windows they would use the whole pool to stress test the
>>>> driver. Now thanks to DTM being in the way, they have a pool of 30
>>>> systems for Windows testing, and reserve the hundreds for other OS’es
>>>> that do have the DTM problems.
>>>>
>>>> DTM IMHO has done more to set back the quality of Windows drivers than
>>>> anything Microsoft has done.
>>>>
>>>>
>>>> –
>>>> Don Burn (MVP, Windows DDK)
>>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>>> Website: http://www.windrvr.com
>>>> Blog: http://msmvps.com/blogs/WinDrvr
>>>> Remove StopSpam to reply
>>>>
>>>>
>>>>
>>>>
>>>> “Bill McKenzie” wrote in message
>>>> news:xxxxx@ntdev…
>>>>>I actually LOVE static analysis tools. I just don’t understand why
>>>>>they don’t get some real kernel devs on them and tailor them
>>>>>appropriately. I think PreFast is a beautiful tool! I don’t even
>>>>>bother with the SAL annotation much (except for the buffer size stuff
>>>>>that is pretty cool) and I still find it terribly useful. I mean, it
>>>>>just catches stupid errors mostly, but that is SO much better to catch
>>>>>at build time and you can easily hook it into the build process to
>>>>>automatically run. The false positives are a good bit annoying, they
>>>>>could trim that a LOT.
>>>>>
>>>>> SDV seems like a great idea, but horribly horribly broken. I, like
>>>>> Peter, have never gotten anything useful out of it. But the idea
>>>>> seems to have a lot of promise. Just needs some serious and directed
>>>>> effort behind it.
>>>>>
>>>>> Run time tools are awesome! I will use all I can get my hands on and
>>>>> I do. But static analysis tools that can be automated into a build are
>>>>> simply beautiful! I wish I had a better handle on just how broad a
>>>>> spectrum of errors those tools could find in a reasonable timeframe
>>>>> given sufficiently sophisticated tools. I really don’t have a good
>>>>> idea where those bounds are. But, if the potential error catching
>>>>> surface is large…I say poor all the resources you can into that!
>>>>>
>>>>> For runtime tools…I would LOVE to see some kind of integration into
>>>>> DTM, so again, we could automate the running and retrieving of
>>>>> results!!
>>>>>
>>>>> For that matter, how about fixing and/or finishing the COM interface
>>>>> to DTM so automating the running of DTM is easier to achieve?
>>>>>
>>>>> Bill M.
>>>>>
>>>>>
>>>>>
>>>>> “Don Burn” wrote in message news:xxxxx@ntdev…
>>>>>> The number of tools that would be nice out of Microsoft is too large
>>>>>> to post, a few over the years I have suggested to them are:
>>>>>>
>>>>>> 1. Code coverage - yes Bullseye is nice, but even with it we need
>>>>>> faut injection. Note they looked at fault injection and rejected it
>>>>>> because people would have to modify their code (I guess annotations
>>>>>> don’t count as modification)
>>>>>>
>>>>>> 2. Lock contention measurement - the number of times I have seen
>>>>>> people propose a gazzilion locks for a driver because they do not
>>>>>> want contention. A tool that shows how long locks are held (max and
>>>>>> average) and how long someone waits for locks would probably slow
>>>>>> down some of these lets have a seperate lock for everything.
>>>>>>
>>>>>> 3. Call usage verifier - this was a godd tool, it could have been
>>>>>> expanded a lot. Come on Redmond either bring it forward, or release
>>>>>> it back to OSR so they can have the OSR WDK (compared to the OSR DDK
>>>>>> that CUV came from).
>>>>>>
>>>>>> Perhaps we need an email list for such tools, complete with a legal
>>>>>> agreement that releases any thing people say, so Redmond or others
>>>>>> can act on it.
>>>>>>
>>>>>>
>>>>>> –
>>>>>> Don Burn (MVP, Windows DDK)
>>>>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>>>>> Website: http://www.windrvr.com
>>>>>> Blog: http://msmvps.com/blogs/WinDrvr
>>>>>> Remove StopSpam to reply
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> “Martin O’Brien” wrote in message
>>>>>> news:xxxxx@ntdev…
>>>>>>> +1
>>>>>>>
>>>>>>> I’d add that some complete and accurate WinDbg documentation might
>>>>>>> be nice, as would the same for the KRM.
>>>>>>>
>>>>>>>
>>>>>>> mm
>>>>>>>
>>>>>>> xxxxx@osr.com wrote:
>>>>>>>> I agree with Mr. O’Brien. The focus of the DDC is clearly static
>>>>>>>> analysis, because if we’re not using it, that’s obviously because
>>>>>>>> we’re dumb out here and slow on the uptake.
>>>>>>>>
>>>>>>>> The distortion field is clearly alive and well. Sigh!
>>>>>>>>
>>>>>>>> As usual, Microsoft would be well served by REALLY understanding
>>>>>>>> what we DO out here. They THINK they understand, but they don’t.
>>>>>>>>
>>>>>>>> As I’ve said before, I like and respect the folks who do SDV and
>>>>>>>> PreFast. I really do. They’re very smart… most of them smarter
>>>>>>>> than me. But they’d just be soooo surprised if they could spend
>>>>>>>> some time, like maybe a solid 3 months, out in the real world
>>>>>>>> trying to USE these tools to do useful work for pay.
>>>>>>>>
>>>>>>>> PreFast: Amusing, sometimes useful, but annoying. For maximum
>>>>>>>> effectiveness and minimum annoyance have to “buy in” to using it
>>>>>>>> throughout the life-cycle of your driver. You have to believe that
>>>>>>>> SAL notations are a good thing. I am not convinced.
>>>>>>>>
>>>>>>>> SDV: Complicated, difficult, and VERY annoying. I have never –
>>>>>>>> not once – been able to get SDV to disclose a meaningful error to
>>>>>>>> me. This is true even when I try to create specific errors in my
>>>>>>>> code that the documentation CLEARLY says SDV will detect and
>>>>>>>> report.
>>>>>>>>
>>>>>>>> I intellectually understand who/why/where these tools could be
>>>>>>>> useful. I suspect that it’ll take about another ten years before I
>>>>>>>> find them useful in practical situations, however.
>>>>>>>>
>>>>>>>> If I were king of the group responsible for these tools, I would
>>>>>>>> redirect their efforts to building clever and comprehensive DYNAMIC
>>>>>>>> analysis and debugging tools that we can use out here in the real
>>>>>>>> world, without significant up-front “adopt it for life or it’s not
>>>>>>>> useful”, requirements.
>>>>>>>>
>>>>>>>> These dynamic tools are really what the community needs at this
>>>>>>>> point. WAY MORE checks in KMDF Verifier. A LOCKING HIERARCHY
>>>>>>>> VALIDATION scheme (OSR has a patent pending on one, we’ll be happy
>>>>>>>> to license it) for ALL sorts of locks. MORE speculative checking
>>>>>>>> options in Driver Verifier. Resurrect Call Usage Verifier – now
>>>>>>>> discontinued. And… just to show you how easy it is to miss the
>>>>>>>> simple stuff: WHY are things like IoGetNextIrpStackLocation still
>>>>>>>> MACROs?? Why are ANY of the DDK functions macros? When they’re
>>>>>>>> macros, they’re impossible to “hook” with Verifier. At LEAST make
>>>>>>>> them functions, and mark them as “inline” in the free build. There
>>>>>>>> is NO additional overhead to this, and much to be gained. This
>>>>>>>> seems obvious to me (and, in fact, was an actual work item for the
>>>>>>>> DDK back, oh, about a thousand years ago).
>>>>>>>>
>>>>>>>> If any of the static analysis folks would like to come to OSR to
>>>>>>>> spend a couple of months writing code on real customer projects,
>>>>>>>> and see “how the other half lives”, I’ll be happy to provide the
>>>>>>>> office and project management. I’ll even give them free lunch for
>>>>>>>> the duration of their stay (like every other OSR employee). Maybe
>>>>>>>> they’ll gain some insight. Then again, maybe they’ll convince us
>>>>>>>> that PreFast or SDV will make our drivers or file systems better
>>>>>>>> and I’ll have to write a big long article in The NT Insider about
>>>>>>>> how SDV changed my life.
>>>>>>>>
>>>>>>>> Peter
>>>>>>>> OSR
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>

[Michal_Vodicka-2]

> -----Original Message-----

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@osr.com
Sent: Friday, September 26, 2008 4:21 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] DDC - updated info for next week

You guys could fix this by inserting a DbgPrint before each
NT_ASSERT in the checked build (only). Or, just change the
freakin’ NT_ASSERT macro.

…and replace __annotation there with DbgPrint for checked build (at
least). All this evil started with introducing undocumented __annotation
keyword. It demonstrates how too much power replaces careful thinking.
The result is useless implementation of a good idea (WPP using own
preprocessor) and the checked build deterioration we see. Just because
they had power to change compiler for their needs. I’m pretty sure the
same problem can be solved using standard tools because we had to did it
for our firmware where was no place for debug strings in the ROM.

I’ve asked about this, repeatedly, both BEFORE the move to
WPP and as recently as about a year ago. And all I’ve EVER
heard folks say is “Oh, yeah… That IS a problem. Hmmm… I
wonder if we can ship those TMFs… I’ll look into that.”
The problem is, nobody’s ever been willing take this on as an
action item and expend some political capital to try to make
it happen (and, we all know that’s what it would take to get it done).

DDC can be a good opportunity to raise this issue again. When we were at
MVP global summit at spring, I had a good feeling they were really
interested about our feedback. On the other hand, I’m not sure if any of
issues discussed there was already solved.

All depends on people. We see MS as a company but such decisions and
changes are made by individuals. You’re right it is necessary to
convince right people to invest time and resources to make such a fix.
Not easy thing. They should understand it is a problem which can lead to
lower quality of 3rd party drivers and worse perception of Windows by
end users.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

[OSR_Community_User]

Is the shouting really necessary Don?

-p

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Friday, September 26, 2008 8:57 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] DDC - updated info for next week

I’VE DEALT WITH THAT PIECE OF SHIT KNOWN AS THE LATEST WLK WITH MY CUSTOMERS
MY OPINION STANDS.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

“Bill McKenzie” wrote in message
news:xxxxx@ntdev…
> Again, I tried DTM, I don’t know about 3 or 4 years ago, and it was an
> abortion. Amazingly, however, they REALLY have improved it to the point
> where I don’t think I want to live without it now.
>
> As far as HCT vs DTM, I don’t even think there is a comparison. What used
> to take me days or even sometimes weeks with HCT tests (IF I could get a
> magic setup to ever work with HCTs and dedicate that magic setup to
> nothing but HCTs for the rest of its life and pray it never breaks) I can
> do in hours with DTM.
>
> DTM isn’t bug free by a long shot, but I find it orders of magnitude
> easier to setup and use than HCTs. Maybe it is the types of drivers we
> are testing?
>
> At any rate, if you haven’t tried the latest, you haven’t used DTM…it
> is a LOT better now than even just one verion ago.
>
> Bill M.
>
> “Don Burn” wrote in message news:xxxxx@ntdev…
>> Bill,
>>
>> To be honest I no longer directly use DTM. I used to try to run it
>> but have given up, since I have to tear my whole test lab apart to even
>> try. What I do have is a number of customers who I work with who run it,
>> and from their experiences I no longer go there. In particular:
>>
>> One large firm, who has set aside the hardware and the MIS
>> resources. They don’t have too many problems, but have commented that
>> every so often a hotfix for the OS destabilizes the world. Their normal
>> approach is to then use MIS to figure out which hotfix and reconfigure
>> the systems.
>>
>> One medium size firm that pays a consulting firm to do it. They
>> have a fixed locked down test network and things mostly work.
>>
>> Several small firms, that assemble test configurations when they
>> need to. Most of them swear they would become pure LINUX if the market
>> could support them, after each attempt at DTM. The ones who have very
>> large servers and setup a domain controller find it painful but doable.
>> The ones who have small systems and do not want a domain controller spend
>> weeks (I’ve been there with them through it) and sometimes finally say
>> “to hell with Redmond” and move on planning to come back in 6 months to a
>> year.
>>
>> I used to run HCT’s for the clients and ensure that the drivers
>> would pass WHQL when they got them. I no longer do because of the
>> problems with DTM. But DTM is actually a reflection of the bigger
>> problem with WHQL and driver testing. For the last 10 years at least
>> WHQL has continued to modify the test environment first HCT’s then DTM to
>> make their life easer at the expense of driver quality. The first
>> principal of quality if find bugs as early in the process as you can, but
>> WHQL testing continues to evolve to more and more complex environments
>> which push the use of the tests later and later into the cycle.
>>
>> If Microsoft spent as much money on good tests for drivers as it has
>> in the last 10 years making WHQL testing more automated for them we would
>> have a lot more tests that driver writers could use, and a related
>> increase in the quality of drivers. Instead as we go forward they make
>> the tests more and more useless, for instance the fact that the various
>> test harnesses from HCT10 on do not run on a checked build to me is
>> ridiculous.
>>
>> At this point, if Redmond want driver quality the simplest first
>> step would be to fire everyone in WHQL and start over.
>>
>>
>> –
>> Don Burn (MVP, Windows DDK)
>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>> Website: http://www.windrvr.com
>> Blog: http://msmvps.com/blogs/WinDrvr
>>
>>
>> “Bill McKenzie” wrote in message
>> news:xxxxx@ntdev…
>>> Don,
>>>
>>> I really have to disagree on DTM. When was the last time you used it?
>>> I used to have a lot of problems with it, but I find the latest DTM to
>>> be a VERY useful tool. I used it recently as the base of an automated
>>> testing framework for my company’s Windows client software. Using
>>> custom scripts I can test everything from the UI to the drivers in a
>>> completely automated fashion. It is great for controlling multiple
>>> platforms and collating results. Now, I don’t know what its limitations
>>> are as far as number of test platforms, we never used more than say 10.
>>> But, all in all for what I needed the tool was great. The only real
>>> problem I had was programmatically controlling DTM from its external COM
>>> interface. THAT is a disaster!
>>>
>>> Bill M.
>>>
>>> “Don Burn” wrote in message news:xxxxx@ntdev…
>>>> How about scrapping DTM and putting together a real test framework?
>>>> DTM is horribly over engineered for anyone other than WHQL, it just
>>>> gets in the way. For those of us who are small environments, setting
>>>> it iup is horribly complicated.
>>>>
>>>> For firms that have large test environments, it still gets in the way.
>>>> I know a major firm that has pools of hundreds of test machines and
>>>> their own tools to image and run tests, for years when they had a new
>>>> driver for Windows they would use the whole pool to stress test the
>>>> driver. Now thanks to DTM being in the way, they have a pool of 30
>>>> systems for Windows testing, and reserve the hundreds for other OS’es
>>>> that do have the DTM problems.
>>>>
>>>> DTM IMHO has done more to set back the quality of Windows drivers than
>>>> anything Microsoft has done.
>>>>
>>>>
>>>> –
>>>> Don Burn (MVP, Windows DDK)
>>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>>> Website: http://www.windrvr.com
>>>> Blog: http://msmvps.com/blogs/WinDrvr
>>>> Remove StopSpam to reply
>>>>
>>>>
>>>>
>>>>
>>>> “Bill McKenzie” wrote in message
>>>> news:xxxxx@ntdev…
>>>>>I actually LOVE static analysis tools. I just don’t understand why
>>>>>they don’t get some real kernel devs on them and tailor them
>>>>>appropriately. I think PreFast is a beautiful tool! I don’t even
>>>>>bother with the SAL annotation much (except for the buffer size stuff
>>>>>that is pretty cool) and I still find it terribly useful. I mean, it
>>>>>just catches stupid errors mostly, but that is SO much better to catch
>>>>>at build time and you can easily hook it into the build process to
>>>>>automatically run. The false positives are a good bit annoying, they
>>>>>could trim that a LOT.
>>>>>
>>>>> SDV seems like a great idea, but horribly horribly broken. I, like
>>>>> Peter, have never gotten anything useful out of it. But the idea
>>>>> seems to have a lot of promise. Just needs some serious and directed
>>>>> effort behind it.
>>>>>
>>>>> Run time tools are awesome! I will use all I can get my hands on and
>>>>> I do. But static analysis tools that can be automated into a build are
>>>>> simply beautiful! I wish I had a better handle on just how broad a
>>>>> spectrum of errors those tools could find in a reasonable timeframe
>>>>> given sufficiently sophisticated tools. I really don’t have a good
>>>>> idea where those bounds are. But, if the potential error catching
>>>>> surface is large…I say poor all the resources you can into that!
>>>>>
>>>>> For runtime tools…I would LOVE to see some kind of integration into
>>>>> DTM, so again, we could automate the running and retrieving of
>>>>> results!!
>>>>>
>>>>> For that matter, how about fixing and/or finishing the COM interface
>>>>> to DTM so automating the running of DTM is easier to achieve?
>>>>>
>>>>> Bill M.
>>>>>
>>>>>
>>>>>
>>>>> “Don Burn” wrote in message news:xxxxx@ntdev…
>>>>>> The number of tools that would be nice out of Microsoft is too large
>>>>>> to post, a few over the years I have suggested to them are:
>>>>>>
>>>>>> 1. Code coverage - yes Bullseye is nice, but even with it we need
>>>>>> faut injection. Note they looked at fault injection and rejected it
>>>>>> because people would have to modify their code (I guess annotations
>>>>>> don’t count as modification)
>>>>>>
>>>>>> 2. Lock contention measurement - the number of times I have seen
>>>>>> people propose a gazzilion locks for a driver because they do not
>>>>>> want contention. A tool that shows how long locks are held (max and
>>>>>> average) and how long someone waits for locks would probably slow
>>>>>> down some of these lets have a seperate lock for everything.
>>>>>>
>>>>>> 3. Call usage verifier - this was a godd tool, it could have been
>>>>>> expanded a lot. Come on Redmond either bring it forward, or release
>>>>>> it back to OSR so they can have the OSR WDK (compared to the OSR DDK
>>>>>> that CUV came from).
>>>>>>
>>>>>> Perhaps we need an email list for such tools, complete with a legal
>>>>>> agreement that releases any thing people say, so Redmond or others
>>>>>> can act on it.
>>>>>>
>>>>>>
>>>>>> –
>>>>>> Don Burn (MVP, Windows DDK)
>>>>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>>>>> Website: http://www.windrvr.com
>>>>>> Blog: http://msmvps.com/blogs/WinDrvr
>>>>>> Remove StopSpam to reply
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> “Martin O’Brien” wrote in message
>>>>>> news:xxxxx@ntdev…
>>>>>>> +1
>>>>>>>
>>>>>>> I’d add that some complete and accurate WinDbg documentation might
>>>>>>> be nice, as would the same for the KRM.
>>>>>>>
>>>>>>>
>>>>>>> mm
>>>>>>>
>>>>>>> xxxxx@osr.com wrote:
>>>>>>>> I agree with Mr. O’Brien. The focus of the DDC is clearly static
>>>>>>>> analysis, because if we’re not using it, that’s obviously because
>>>>>>>> we’re dumb out here and slow on the uptake.
>>>>>>>>
>>>>>>>> The distortion field is clearly alive and well. Sigh!
>>>>>>>>
>>>>>>>> As usual, Microsoft would be well served by REALLY understanding
>>>>>>>> what we DO out here. They THINK they understand, but they don’t.
>>>>>>>>
>>>>>>>> As I’ve said before, I like and respect the folks who do SDV and
>>>>>>>> PreFast. I really do. They’re very smart… most of them smarter
>>>>>>>> than me. But they’d just be soooo surprised if they could spend
>>>>>>>> some time, like maybe a solid 3 months, out in the real world
>>>>>>>> trying to USE these tools to do useful work for pay.
>>>>>>>>
>>>>>>>> PreFast: Amusing, sometimes useful, but annoying. For maximum
>>>>>>>> effectiveness and minimum annoyance have to “buy in” to using it
>>>>>>>> throughout the life-cycle of your driver. You have to believe that
>>>>>>>> SAL notations are a good thing. I am not convinced.
>>>>>>>>
>>>>>>>> SDV: Complicated, difficult, and VERY annoying. I have never –
>>>>>>>> not once – been able to get SDV to disclose a meaningful error to
>>>>>>>> me. This is true even when I try to create specific errors in my
>>>>>>>> code that the documentation CLEARLY says SDV will detect and
>>>>>>>> report.
>>>>>>>>
>>>>>>>> I intellectually understand who/why/where these tools could be
>>>>>>>> useful. I suspect that it’ll take about another ten years before I
>>>>>>>> find them useful in practical situations, however.
>>>>>>>>
>>>>>>>> If I were king of the group responsible for these tools, I would
>>>>>>>> redirect their efforts to building clever and comprehensive DYNAMIC
>>>>>>>> analysis and debugging tools that we can use out here in the real
>>>>>>>> world, without significant up-front “adopt it for life or it’s not
>>>>>>>> useful”, requirements.
>>>>>>>>
>>>>>>>> These dynamic tools are really what the community needs at this
>>>>>>>> point. WAY MORE checks in KMDF Verifier. A LOCKING HIERARCHY
>>>>>>>> VALIDATION scheme (OSR has a patent pending on one, we’ll be happy
>>>>>>>> to license it) for ALL sorts of locks. MORE speculative checking
>>>>>>>> options in Driver Verifier. Resurrect Call Usage Verifier – now
>>>>>>>> discontinued. And… just to show you how easy it is to miss the
>>>>>>>> simple stuff: WHY are things like IoGetNextIrpStackLocation still
>>>>>>>> MACROs?? Why are ANY of the DDK functions macros? When they’re
>>>>>>>> macros, they’re impossible to “hook” with Verifier. At LEAST make
>>>>>>>> them functions, and mark them as “inline” in the free build. There
>>>>>>>> is NO additional overhead to this, and much to be gained. This
>>>>>>>> seems obvious to me (and, in fact, was an actual work item for the
>>>>>>>> DDK back, oh, about a thousand years ago).
>>>>>>>>
>>>>>>>> If any of the static analysis folks would like to come to OSR to
>>>>>>>> spend a couple of months writing code on real customer projects,
>>>>>>>> and see “how the other half lives”, I’ll be happy to provide the
>>>>>>>> office and project management. I’ll even give them free lunch for
>>>>>>>> the duration of their stay (like every other OSR employee). Maybe
>>>>>>>> they’ll gain some insight. Then again, maybe they’ll convince us
>>>>>>>> that PreFast or SDV will make our drivers or file systems better
>>>>>>>> and I’ll have to write a big long article in The NT Insider about
>>>>>>>> how SDV changed my life.
>>>>>>>>
>>>>>>>> Peter
>>>>>>>> OSR
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

[Mark_Roddy]

When DTM is involved: yes.

On Fri, Sep 26, 2008 at 1:01 PM, Peter Wieland <
xxxxx@windows.microsoft.com> wrote:

Is the shouting really necessary Don?

-p

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:
xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Friday, September 26, 2008 8:57 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] DDC - updated info for next week

I’VE DEALT WITH THAT PIECE OF SHIT KNOWN AS THE LATEST WLK WITH MY
CUSTOMERS
MY OPINION STANDS.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

“Bill McKenzie” wrote in message
> news:xxxxx@ntdev…
> > Again, I tried DTM, I don’t know about 3 or 4 years ago, and it was an
> > abortion. Amazingly, however, they REALLY have improved it to the point
> > where I don’t think I want to live without it now.
> >
> > As far as HCT vs DTM, I don’t even think there is a comparison. What used
> > to take me days or even sometimes weeks with HCT tests (IF I could get a
> > magic setup to ever work with HCTs and dedicate that magic setup to
> > nothing but HCTs for the rest of its life and pray it never breaks) I can
> > do in hours with DTM.
> >
> > DTM isn’t bug free by a long shot, but I find it orders of magnitude
> > easier to setup and use than HCTs. Maybe it is the types of drivers we
> > are testing?
> >
> > At any rate, if you haven’t tried the latest, you haven’t used DTM…it
> > is a LOT better now than even just one verion ago.
> >
> > Bill M.
> >
> > “Don Burn” wrote in message news:xxxxx@ntdev…
> >> Bill,
> >>
> >> To be honest I no longer directly use DTM. I used to try to run it
> >> but have given up, since I have to tear my whole test lab apart to even
> >> try. What I do have is a number of customers who I work with who run it,
> >> and from their experiences I no longer go there. In particular:
> >>
> >> One large firm, who has set aside the hardware and the MIS
> >> resources. They don’t have too many problems, but have commented that
> >> every so often a hotfix for the OS destabilizes the world. Their normal
> >> approach is to then use MIS to figure out which hotfix and reconfigure
> >> the systems.
> >>
> >> One medium size firm that pays a consulting firm to do it. They
> >> have a fixed locked down test network and things mostly work.
> >>
> >> Several small firms, that assemble test configurations when they
> >> need to. Most of them swear they would become pure LINUX if the market
> >> could support them, after each attempt at DTM. The ones who have very
> >> large servers and setup a domain controller find it painful but doable.
> >> The ones who have small systems and do not want a domain controller
> spend
> >> weeks (I’ve been there with them through it) and sometimes finally say
> >> “to hell with Redmond” and move on planning to come back in 6 months to
> a
> >> year.
> >>
> >> I used to run HCT’s for the clients and ensure that the drivers
> >> would pass WHQL when they got them. I no longer do because of the
> >> problems with DTM. But DTM is actually a reflection of the bigger
> >> problem with WHQL and driver testing. For the last 10 years at least
> >> WHQL has continued to modify the test environment first HCT’s then DTM
> to
> >> make their life easer at the expense of driver quality. The first
> >> principal of quality if find bugs as early in the process as you can,
> but
> >> WHQL testing continues to evolve to more and more complex environments
> >> which push the use of the tests later and later into the cycle.
> >>
> >> If Microsoft spent as much money on good tests for drivers as it
> has
> >> in the last 10 years making WHQL testing more automated for them we
> would
> >> have a lot more tests that driver writers could use, and a related
> >> increase in the quality of drivers. Instead as we go forward they make
> >> the tests more and more useless, for instance the fact that the various
> >> test harnesses from HCT10 on do not run on a checked build to me is
> >> ridiculous.
> >>
> >> At this point, if Redmond want driver quality the simplest first
> >> step would be to fire everyone in WHQL and start over.
> >>
> >>
> >> –
> >> Don Burn (MVP, Windows DDK)
> >> Windows 2k/XP/2k3 Filesystem and Driver Consulting
> >> Website: http://www.windrvr.com
> >> Blog: http://msmvps.com/blogs/WinDrvr
> >>
> >>
> >> “Bill McKenzie” wrote in message
> >> news:xxxxx@ntdev…
> >>> Don,
> >>>
> >>> I really have to disagree on DTM. When was the last time you used it?
> >>> I used to have a lot of problems with it, but I find the latest DTM to
> >>> be a VERY useful tool. I used it recently as the base of an automated
> >>> testing framework for my company’s Windows client software. Using
> >>> custom scripts I can test everything from the UI to the drivers in a
> >>> completely automated fashion. It is great for controlling multiple
> >>> platforms and collating results. Now, I don’t know what its
> limitations
> >>> are as far as number of test platforms, we never used more than say 10.
> >>> But, all in all for what I needed the tool was great. The only real
> >>> problem I had was programmatically controlling DTM from its external
> COM
> >>> interface. THAT is a disaster!
> >>>
> >>> Bill M.
> >>>
> >>> “Don Burn” wrote in message news:xxxxx@ntdev…
> >>>> How about scrapping DTM and putting together a real test framework?
> >>>> DTM is horribly over engineered for anyone other than WHQL, it just
> >>>> gets in the way. For those of us who are small environments, setting
> >>>> it iup is horribly complicated.
> >>>>
> >>>> For firms that have large test environments, it still gets in the way.
> >>>> I know a major firm that has pools of hundreds of test machines and
> >>>> their own tools to image and run tests, for years when they had a new
> >>>> driver for Windows they would use the whole pool to stress test the
> >>>> driver. Now thanks to DTM being in the way, they have a pool of 30
> >>>> systems for Windows testing, and reserve the hundreds for other OS’es
> >>>> that do have the DTM problems.
> >>>>
> >>>> DTM IMHO has done more to set back the quality of Windows drivers than
> >>>> anything Microsoft has done.
> >>>>
> >>>>
> >>>> –
> >>>> Don Burn (MVP, Windows DDK)
> >>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
> >>>> Website: http://www.windrvr.com
> >>>> Blog: http://msmvps.com/blogs/WinDrvr
> >>>> Remove StopSpam to reply
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> “Bill McKenzie” wrote in message
> >>>> news:xxxxx@ntdev…
> >>>>>I actually LOVE static analysis tools. I just don’t understand why
> >>>>>they don’t get some real kernel devs on them and tailor them
> >>>>>appropriately. I think PreFast is a beautiful tool! I don’t even
> >>>>>bother with the SAL annotation much (except for the buffer size stuff
> >>>>>that is pretty cool) and I still find it terribly useful. I mean, it
> >>>>>just catches stupid errors mostly, but that is SO much better to catch
> >>>>>at build time and you can easily hook it into the build process to
> >>>>>automatically run. The false positives are a good bit annoying, they
> >>>>>could trim that a LOT.
> >>>>>
> >>>>> SDV seems like a great idea, but horribly horribly broken. I, like
> >>>>> Peter, have never gotten anything useful out of it. But the idea
> >>>>> seems to have a lot of promise. Just needs some serious and directed
> >>>>> effort behind it.
> >>>>>
> >>>>> Run time tools are awesome! I will use all I can get my hands on and
> >>>>> I do. But static analysis tools that can be automated into a build
> are
> >>>>> simply beautiful! I wish I had a better handle on just how broad a
> >>>>> spectrum of errors those tools could find in a reasonable timeframe
> >>>>> given sufficiently sophisticated tools. I really don’t have a good
> >>>>> idea where those bounds are. But, if the potential error catching
> >>>>> surface is large…I say poor all the resources you can into that!
> >>>>>
> >>>>> For runtime tools…I would LOVE to see some kind of integration into
> >>>>> DTM, so again, we could automate the running and retrieving of
> >>>>> results!!
> >>>>>
> >>>>> For that matter, how about fixing and/or finishing the COM interface
> >>>>> to DTM so automating the running of DTM is easier to achieve?
> >>>>>
> >>>>> Bill M.
> >>>>>
> >>>>>
> >>>>>
> >>>>> “Don Burn” wrote in message news:xxxxx@ntdev…
> >>>>>> The number of tools that would be nice out of Microsoft is too large
> >>>>>> to post, a few over the years I have suggested to them are:
> >>>>>>
> >>>>>> 1. Code coverage - yes Bullseye is nice, but even with it we need
> >>>>>> faut injection. Note they looked at fault injection and rejected it
> >>>>>> because people would have to modify their code (I guess annotations
> >>>>>> don’t count as modification)
> >>>>>>
> >>>>>> 2. Lock contention measurement - the number of times I have seen
> >>>>>> people propose a gazzilion locks for a driver because they do not
> >>>>>> want contention. A tool that shows how long locks are held (max and
> >>>>>> average) and how long someone waits for locks would probably slow
> >>>>>> down some of these lets have a seperate lock for everything.
> >>>>>>
> >>>>>> 3. Call usage verifier - this was a godd tool, it could have been
> >>>>>> expanded a lot. Come on Redmond either bring it forward, or release
> >>>>>> it back to OSR so they can have the OSR WDK (compared to the OSR DDK
> >>>>>> that CUV came from).
> >>>>>>
> >>>>>> Perhaps we need an email list for such tools, complete with a legal
> >>>>>> agreement that releases any thing people say, so Redmond or others
> >>>>>> can act on it.
> >>>>>>
> >>>>>>
> >>>>>> –
> >>>>>> Don Burn (MVP, Windows DDK)
> >>>>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
> >>>>>> Website: http://www.windrvr.com
> >>>>>> Blog: http://msmvps.com/blogs/WinDrvr
> >>>>>> Remove StopSpam to reply
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> “Martin O’Brien” wrote in message
> >>>>>> news:xxxxx@ntdev…
> >>>>>>> +1
> >>>>>>>
> >>>>>>> I’d add that some complete and accurate WinDbg documentation might
> >>>>>>> be nice, as would the same for the KRM.
> >>>>>>>
> >>>>>>>
> >>>>>>> mm
> >>>>>>>
> >>>>>>> xxxxx@osr.com wrote:
> >>>>>>>> I agree with Mr. O’Brien. The focus of the DDC is clearly static
> >>>>>>>> analysis, because if we’re not using it, that’s obviously because
> >>>>>>>> we’re dumb out here and slow on the uptake.
> >>>>>>>>
> >>>>>>>> The distortion field is clearly alive and well. Sigh!
> >>>>>>>>
> >>>>>>>> As usual, Microsoft would be well served by REALLY understanding
> >>>>>>>> what we DO out here. They THINK they understand, but they don’t.
> >>>>>>>>
> >>>>>>>> As I’ve said before, I like and respect the folks who do SDV and
> >>>>>>>> PreFast. I really do. They’re very smart… most of them smarter
> >>>>>>>> than me. But they’d just be soooo surprised if they could spend
> >>>>>>>> some time, like maybe a solid 3 months, out in the real world
> >>>>>>>> trying to USE these tools to do useful work for pay.
> >>>>>>>>
> >>>>>>>> PreFast: Amusing, sometimes useful, but annoying. For maximum
> >>>>>>>> effectiveness and minimum annoyance have to “buy in” to using it
> >>>>>>>> throughout the life-cycle of your driver. You have to believe
> that
> >>>>>>>> SAL notations are a good thing. I am not convinced.
> >>>>>>>>
> >>>>>>>> SDV: Complicated, difficult, and VERY annoying. I have never –
> >>>>>>>> not once – been able to get SDV to disclose a meaningful error to
> >>>>>>>> me. This is true even when I try to create specific errors in my
> >>>>>>>> code that the documentation CLEARLY says SDV will detect and
> >>>>>>>> report.
> >>>>>>>>
> >>>>>>>> I intellectually understand who/why/where these tools could be
> >>>>>>>> useful. I suspect that it’ll take about another ten years before I
> >>>>>>>> find them useful in practical situations, however.
> >>>>>>>>
> >>>>>>>> If I were king of the group responsible for these tools, I would
> >>>>>>>> redirect their efforts to building clever and comprehensive
> DYNAMIC
> >>>>>>>> analysis and debugging tools that we can use out here in the real
> >>>>>>>> world, without significant up-front “adopt it for life or it’s not
> >>>>>>>> useful”, requirements.
> >>>>>>>>
> >>>>>>>> These dynamic tools are really what the community needs at this
> >>>>>>>> point. WAY MORE checks in KMDF Verifier. A LOCKING HIERARCHY
> >>>>>>>> VALIDATION scheme (OSR has a patent pending on one, we’ll be happy
> >>>>>>>> to license it) for ALL sorts of locks. MORE speculative checking
> >>>>>>>> options in Driver Verifier. Resurrect Call Usage Verifier – now
> >>>>>>>> discontinued. And… just to show you how easy it is to miss the
> >>>>>>>> simple stuff: WHY are things like IoGetNextIrpStackLocation still
> >>>>>>>> MACROs?? Why are ANY of the DDK functions macros? When they’re
> >>>>>>>> macros, they’re impossible to “hook” with Verifier. At LEAST make
> >>>>>>>> them functions, and mark them as “inline” in the free build.
> There
> >>>>>>>> is NO additional overhead to this, and much to be gained. This
> >>>>>>>> seems obvious to me (and, in fact, was an actual work item for the
> >>>>>>>> DDK back, oh, about a thousand years ago).
> >>>>>>>>
> >>>>>>>> If any of the static analysis folks would like to come to OSR to
> >>>>>>>> spend a couple of months writing code on real customer projects,
> >>>>>>>> and see “how the other half lives”, I’ll be happy to provide the
> >>>>>>>> office and project management. I’ll even give them free lunch for
> >>>>>>>> the duration of their stay (like every other OSR employee). Maybe
> >>>>>>>> they’ll gain some insight. Then again, maybe they’ll convince us
> >>>>>>>> that PreFast or SDV will make our drivers or file systems better
> >>>>>>>> and I’ll have to write a big long article in The NT Insider about
> >>>>>>>> how SDV changed my life.
> >>>>>>>>
> >>>>>>>> Peter
> >>>>>>>> OSR
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >
> >
> >
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

–
Mark Roddy

[Don_Burn_1]

Peter,

A lot of us have complained about DTM for 5.5 years since Jean
Valentine first presented it at WinHEC, but little has been done to address
problems pointed out back then. Perhaps shouting should have been done
more? HCT’s while pretty bad could be set up in less than an hour, now I
tell my customers (who don’'t have full time MIS staffs) to plan for 2
weeks, personally that indicates something terribly wrong in my opinion.

I fully support the idea of driver testing and logo’ing but it needs
to test something. I have seen little evidence that most of the WHQL
testing (this does not include forcing drivers to run under Verifier or
things like Prefast) does anything to test the driver for quality.
Microsoft has done a lot of fantastic work in the last 5.5 years, the whole
WDF effort is a great example, but WHQL has not done anything to improve the
quality of Windows. Considering that Windows is hammered by proponents of
tha other OS for quality this says to me that WHQL is giving the illusion
that Redmond is working on it without actually doing anything, a classic
example of the reality distortion field at work.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

“Peter Wieland” wrote in message
news:xxxxx@ntdev…
Is the shouting really necessary Don?

-p

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Friday, September 26, 2008 8:57 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] DDC - updated info for next week

I’VE DEALT WITH THAT PIECE OF SHIT KNOWN AS THE LATEST WLK WITH MY CUSTOMERS
MY OPINION STANDS.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

“Bill McKenzie” wrote in message
news:xxxxx@ntdev…
> Again, I tried DTM, I don’t know about 3 or 4 years ago, and it was an
> abortion. Amazingly, however, they REALLY have improved it to the point
> where I don’t think I want to live without it now.
>
> As far as HCT vs DTM, I don’t even think there is a comparison. What used
> to take me days or even sometimes weeks with HCT tests (IF I could get a
> magic setup to ever work with HCTs and dedicate that magic setup to
> nothing but HCTs for the rest of its life and pray it never breaks) I can
> do in hours with DTM.
>
> DTM isn’t bug free by a long shot, but I find it orders of magnitude
> easier to setup and use than HCTs. Maybe it is the types of drivers we
> are testing?
>
> At any rate, if you haven’t tried the latest, you haven’t used DTM…it
> is a LOT better now than even just one verion ago.
>
> Bill M.
>
> “Don Burn” wrote in message news:xxxxx@ntdev…
>> Bill,
>>
>> To be honest I no longer directly use DTM. I used to try to run it
>> but have given up, since I have to tear my whole test lab apart to even
>> try. What I do have is a number of customers who I work with who run it,
>> and from their experiences I no longer go there. In particular:
>>
>> One large firm, who has set aside the hardware and the MIS
>> resources. They don’t have too many problems, but have commented that
>> every so often a hotfix for the OS destabilizes the world. Their normal
>> approach is to then use MIS to figure out which hotfix and reconfigure
>> the systems.
>>
>> One medium size firm that pays a consulting firm to do it. They
>> have a fixed locked down test network and things mostly work.
>>
>> Several small firms, that assemble test configurations when they
>> need to. Most of them swear they would become pure LINUX if the market
>> could support them, after each attempt at DTM. The ones who have very
>> large servers and setup a domain controller find it painful but doable.
>> The ones who have small systems and do not want a domain controller spend
>> weeks (I’ve been there with them through it) and sometimes finally say
>> “to hell with Redmond” and move on planning to come back in 6 months to a
>> year.
>>
>> I used to run HCT’s for the clients and ensure that the drivers
>> would pass WHQL when they got them. I no longer do because of the
>> problems with DTM. But DTM is actually a reflection of the bigger
>> problem with WHQL and driver testing. For the last 10 years at least
>> WHQL has continued to modify the test environment first HCT’s then DTM to
>> make their life easer at the expense of driver quality. The first
>> principal of quality if find bugs as early in the process as you can, but
>> WHQL testing continues to evolve to more and more complex environments
>> which push the use of the tests later and later into the cycle.
>>
>> If Microsoft spent as much money on good tests for drivers as it has
>> in the last 10 years making WHQL testing more automated for them we would
>> have a lot more tests that driver writers could use, and a related
>> increase in the quality of drivers. Instead as we go forward they make
>> the tests more and more useless, for instance the fact that the various
>> test harnesses from HCT10 on do not run on a checked build to me is
>> ridiculous.
>>
>> At this point, if Redmond want driver quality the simplest first
>> step would be to fire everyone in WHQL and start over.
>>
>>
>> –
>> Don Burn (MVP, Windows DDK)
>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>> Website: http://www.windrvr.com
>> Blog: http://msmvps.com/blogs/WinDrvr
>>
>>
>> “Bill McKenzie” wrote in message
>> news:xxxxx@ntdev…
>>> Don,
>>>
>>> I really have to disagree on DTM. When was the last time you used it?
>>> I used to have a lot of problems with it, but I find the latest DTM to
>>> be a VERY useful tool. I used it recently as the base of an automated
>>> testing framework for my company’s Windows client software. Using
>>> custom scripts I can test everything from the UI to the drivers in a
>>> completely automated fashion. It is great for controlling multiple
>>> platforms and collating results. Now, I don’t know what its limitations
>>> are as far as number of test platforms, we never used more than say 10.
>>> But, all in all for what I needed the tool was great. The only real
>>> problem I had was programmatically controlling DTM from its external COM
>>> interface. THAT is a disaster!
>>>
>>> Bill M.
>>>
>>> “Don Burn” wrote in message news:xxxxx@ntdev…
>>>> How about scrapping DTM and putting together a real test framework?
>>>> DTM is horribly over engineered for anyone other than WHQL, it just
>>>> gets in the way. For those of us who are small environments, setting
>>>> it iup is horribly complicated.
>>>>
>>>> For firms that have large test environments, it still gets in the way.
>>>> I know a major firm that has pools of hundreds of test machines and
>>>> their own tools to image and run tests, for years when they had a new
>>>> driver for Windows they would use the whole pool to stress test the
>>>> driver. Now thanks to DTM being in the way, they have a pool of 30
>>>> systems for Windows testing, and reserve the hundreds for other OS’es
>>>> that do have the DTM problems.
>>>>
>>>> DTM IMHO has done more to set back the quality of Windows drivers than
>>>> anything Microsoft has done.
>>>>
>>>>
>>>> –
>>>> Don Burn (MVP, Windows DDK)
>>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>>> Website: http://www.windrvr.com
>>>> Blog: http://msmvps.com/blogs/WinDrvr
>>>> Remove StopSpam to reply
>>>>
>>>>
>>>>
>>>>
>>>> “Bill McKenzie” wrote in message
>>>> news:xxxxx@ntdev…
>>>>>I actually LOVE static analysis tools. I just don’t understand why
>>>>>they don’t get some real kernel devs on them and tailor them
>>>>>appropriately. I think PreFast is a beautiful tool! I don’t even
>>>>>bother with the SAL annotation much (except for the buffer size stuff
>>>>>that is pretty cool) and I still find it terribly useful. I mean, it
>>>>>just catches stupid errors mostly, but that is SO much better to catch
>>>>>at build time and you can easily hook it into the build process to
>>>>>automatically run. The false positives are a good bit annoying, they
>>>>>could trim that a LOT.
>>>>>
>>>>> SDV seems like a great idea, but horribly horribly broken. I, like
>>>>> Peter, have never gotten anything useful out of it. But the idea
>>>>> seems to have a lot of promise. Just needs some serious and directed
>>>>> effort behind it.
>>>>>
>>>>> Run time tools are awesome! I will use all I can get my hands on and
>>>>> I do. But static analysis tools that can be automated into a build are
>>>>> simply beautiful! I wish I had a better handle on just how broad a
>>>>> spectrum of errors those tools could find in a reasonable timeframe
>>>>> given sufficiently sophisticated tools. I really don’t have a good
>>>>> idea where those bounds are. But, if the potential error catching
>>>>> surface is large…I say poor all the resources you can into that!
>>>>>
>>>>> For runtime tools…I would LOVE to see some kind of integration into
>>>>> DTM, so again, we could automate the running and retrieving of
>>>>> results!!
>>>>>
>>>>> For that matter, how about fixing and/or finishing the COM interface
>>>>> to DTM so automating the running of DTM is easier to achieve?
>>>>>
>>>>> Bill M.
>>>>>
>>>>>
>>>>>
>>>>> “Don Burn” wrote in message news:xxxxx@ntdev…
>>>>>> The number of tools that would be nice out of Microsoft is too large
>>>>>> to post, a few over the years I have suggested to them are:
>>>>>>
>>>>>> 1. Code coverage - yes Bullseye is nice, but even with it we need
>>>>>> faut injection. Note they looked at fault injection and rejected it
>>>>>> because people would have to modify their code (I guess annotations
>>>>>> don’t count as modification)
>>>>>>
>>>>>> 2. Lock contention measurement - the number of times I have seen
>>>>>> people propose a gazzilion locks for a driver because they do not
>>>>>> want contention. A tool that shows how long locks are held (max and
>>>>>> average) and how long someone waits for locks would probably slow
>>>>>> down some of these lets have a seperate lock for everything.
>>>>>>
>>>>>> 3. Call usage verifier - this was a godd tool, it could have been
>>>>>> expanded a lot. Come on Redmond either bring it forward, or release
>>>>>> it back to OSR so they can have the OSR WDK (compared to the OSR DDK
>>>>>> that CUV came from).
>>>>>>
>>>>>> Perhaps we need an email list for such tools, complete with a legal
>>>>>> agreement that releases any thing people say, so Redmond or others
>>>>>> can act on it.
>>>>>>
>>>>>>
>>>>>> –
>>>>>> Don Burn (MVP, Windows DDK)
>>>>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>>>>> Website: http://www.windrvr.com
>>>>>> Blog: http://msmvps.com/blogs/WinDrvr
>>>>>> Remove StopSpam to reply
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> “Martin O’Brien” wrote in message
>>>>>> news:xxxxx@ntdev…
>>>>>>> +1
>>>>>>>
>>>>>>> I’d add that some complete and accurate WinDbg documentation might
>>>>>>> be nice, as would the same for the KRM.
>>>>>>>
>>>>>>>
>>>>>>> mm
>>>>>>>
>>>>>>> xxxxx@osr.com wrote:
>>>>>>>> I agree with Mr. O’Brien. The focus of the DDC is clearly static
>>>>>>>> analysis, because if we’re not using it, that’s obviously because
>>>>>>>> we’re dumb out here and slow on the uptake.
>>>>>>>>
>>>>>>>> The distortion field is clearly alive and well. Sigh!
>>>>>>>>
>>>>>>>> As usual, Microsoft would be well served by REALLY understanding
>>>>>>>> what we DO out here. They THINK they understand, but they don’t.
>>>>>>>>
>>>>>>>> As I’ve said before, I like and respect the folks who do SDV and
>>>>>>>> PreFast. I really do. They’re very smart… most of them smarter
>>>>>>>> than me. But they’d just be soooo surprised if they could spend
>>>>>>>> some time, like maybe a solid 3 months, out in the real world
>>>>>>>> trying to USE these tools to do useful work for pay.
>>>>>>>>
>>>>>>>> PreFast: Amusing, sometimes useful, but annoying. For maximum
>>>>>>>> effectiveness and minimum annoyance have to “buy in” to using it
>>>>>>>> throughout the life-cycle of your driver. You have to believe that
>>>>>>>> SAL notations are a good thing. I am not convinced.
>>>>>>>>
>>>>>>>> SDV: Complicated, difficult, and VERY annoying. I have never –
>>>>>>>> not once – been able to get SDV to disclose a meaningful error to
>>>>>>>> me. This is true even when I try to create specific errors in my
>>>>>>>> code that the documentation CLEARLY says SDV will detect and
>>>>>>>> report.
>>>>>>>>
>>>>>>>> I intellectually understand who/why/where these tools could be
>>>>>>>> useful. I suspect that it’ll take about another ten years before I
>>>>>>>> find them useful in practical situations, however.
>>>>>>>>
>>>>>>>> If I were king of the group responsible for these tools, I would
>>>>>>>> redirect their efforts to building clever and comprehensive DYNAMIC
>>>>>>>> analysis and debugging tools that we can use out here in the real
>>>>>>>> world, without significant up-front “adopt it for life or it’s not
>>>>>>>> useful”, requirements.
>>>>>>>>
>>>>>>>> These dynamic tools are really what the community needs at this
>>>>>>>> point. WAY MORE checks in KMDF Verifier. A LOCKING HIERARCHY
>>>>>>>> VALIDATION scheme (OSR has a patent pending on one, we’ll be happy
>>>>>>>> to license it) for ALL sorts of locks. MORE speculative checking
>>>>>>>> options in Driver Verifier. Resurrect Call Usage Verifier – now
>>>>>>>> discontinued. And… just to show you how easy it is to miss the
>>>>>>>> simple stuff: WHY are things like IoGetNextIrpStackLocation still
>>>>>>>> MACROs?? Why are ANY of the DDK functions macros? When they’re
>>>>>>>> macros, they’re impossible to “hook” with Verifier. At LEAST make
>>>>>>>> them functions, and mark them as “inline” in the free build. There
>>>>>>>> is NO additional overhead to this, and much to be gained. This
>>>>>>>> seems obvious to me (and, in fact, was an actual work item for the
>>>>>>>> DDK back, oh, about a thousand years ago).
>>>>>>>>
>>>>>>>> If any of the static analysis folks would like to come to OSR to
>>>>>>>> spend a couple of months writing code on real customer projects,
>>>>>>>> and see “how the other half lives”, I’ll be happy to provide the
>>>>>>>> office and project management. I’ll even give them free lunch for
>>>>>>>> the duration of their stay (like every other OSR employee). Maybe
>>>>>>>> they’ll gain some insight. Then again, maybe they’ll convince us
>>>>>>>> that PreFast or SDV will make our drivers or file systems better
>>>>>>>> and I’ll have to write a big long article in The NT Insider about
>>>>>>>> how SDV changed my life.
>>>>>>>>
>>>>>>>> Peter
>>>>>>>> OSR
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

[Ken_Johnson]

However, the whole ?if someone disagrees, say it again, louder? is not particularly productive.

• S

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Mark Roddy
Sent: Friday, September 26, 2008 1:15 PM
To: Windows System Software Devs Interest List
Subject: Re: Re:[ntdev] DDC - updated info for next week

When DTM is involved: yes.
On Fri, Sep 26, 2008 at 1:01 PM, Peter Wieland > wrote:
Is the shouting really necessary Don?

-p

-----Original Message-----
From: xxxxx@lists.osr.commailto:xxxxx [mailto:xxxxx@lists.osr.commailto:xxxxx] On Behalf Of Don Burn
Sent: Friday, September 26, 2008 8:57 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] DDC - updated info for next week

I’VE DEALT WITH THAT PIECE OF SHIT KNOWN AS THE LATEST WLK WITH MY CUSTOMERS
MY OPINION STANDS.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
Website: http://www.windrvr.comhttp:</http:>
Blog: http://msmvps.com/blogs/WinDrvr

“Bill McKenzie” > wrote in message
news:xxxxx@ntdev…
> Again, I tried DTM, I don’t know about 3 or 4 years ago, and it was an
> abortion. Amazingly, however, they REALLY have improved it to the point
> where I don’t think I want to live without it now.
>
> As far as HCT vs DTM, I don’t even think there is a comparison. What used
> to take me days or even sometimes weeks with HCT tests (IF I could get a
> magic setup to ever work with HCTs and dedicate that magic setup to
> nothing but HCTs for the rest of its life and pray it never breaks) I can
> do in hours with DTM.
>
> DTM isn’t bug free by a long shot, but I find it orders of magnitude
> easier to setup and use than HCTs. Maybe it is the types of drivers we
> are testing?
>
> At any rate, if you haven’t tried the latest, you haven’t used DTM…it
> is a LOT better now than even just one verion ago.
>
> Bill M.
>
> “Don Burn” > wrote in message news:xxxxx@ntdev…
>> Bill,
>>
>> To be honest I no longer directly use DTM. I used to try to run it
>> but have given up, since I have to tear my whole test lab apart to even
>> try. What I do have is a number of customers who I work with who run it,
>> and from their experiences I no longer go there. In particular:
>>
>> One large firm, who has set aside the hardware and the MIS
>> resources. They don’t have too many problems, but have commented that
>> every so often a hotfix for the OS destabilizes the world. Their normal
>> approach is to then use MIS to figure out which hotfix and reconfigure
>> the systems.
>>
>> One medium size firm that pays a consulting firm to do it. They
>> have a fixed locked down test network and things mostly work.
>>
>> Several small firms, that assemble test configurations when they
>> need to. Most of them swear they would become pure LINUX if the market
>> could support them, after each attempt at DTM. The ones who have very
>> large servers and setup a domain controller find it painful but doable.
>> The ones who have small systems and do not want a domain controller spend
>> weeks (I’ve been there with them through it) and sometimes finally say
>> “to hell with Redmond” and move on planning to come back in 6 months to a
>> year.
>>
>> I used to run HCT’s for the clients and ensure that the drivers
>> would pass WHQL when they got them. I no longer do because of the
>> problems with DTM. But DTM is actually a reflection of the bigger
>> problem with WHQL and driver testing. For the last 10 years at least
>> WHQL has continued to modify the test environment first HCT’s then DTM to
>> make their life easer at the expense of driver quality. The first
>> principal of quality if find bugs as early in the process as you can, but
>> WHQL testing continues to evolve to more and more complex environments
>> which push the use of the tests later and later into the cycle.
>>
>> If Microsoft spent as much money on good tests for drivers as it has
>> in the last 10 years making WHQL testing more automated for them we would
>> have a lot more tests that driver writers could use, and a related
>> increase in the quality of drivers. Instead as we go forward they make
>> the tests more and more useless, for instance the fact that the various
>> test harnesses from HCT10 on do not run on a checked build to me is
>> ridiculous.
>>
>> At this point, if Redmond want driver quality the simplest first
>> step would be to fire everyone in WHQL and start over.
>>
>>
>> –
>> Don Burn (MVP, Windows DDK)
>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>> Website: http://www.windrvr.comhttp:</http:>
>> Blog: http://msmvps.com/blogs/WinDrvr
>>
>>
>> “Bill McKenzie” > wrote in message
>> news:xxxxx@ntdev…
>>> Don,
>>>
>>> I really have to disagree on DTM. When was the last time you used it?
>>> I used to have a lot of problems with it, but I find the latest DTM to
>>> be a VERY useful tool. I used it recently as the base of an automated
>>> testing framework for my company’s Windows client software. Using
>>> custom scripts I can test everything from the UI to the drivers in a
>>> completely automated fashion. It is great for controlling multiple
>>> platforms and collating results. Now, I don’t know what its limitations
>>> are as far as number of test platforms, we never used more than say 10.
>>> But, all in all for what I needed the tool was great. The only real
>>> problem I had was programmatically controlling DTM from its external COM
>>> interface. THAT is a disaster!
>>>
>>> Bill M.
>>>
>>> “Don Burn” > wrote in message news:xxxxx@ntdev…
>>>> How about scrapping DTM and putting together a real test framework?
>>>> DTM is horribly over engineered for anyone other than WHQL, it just
>>>> gets in the way. For those of us who are small environments, setting
>>>> it iup is horribly complicated.
>>>>
>>>> For firms that have large test environments, it still gets in the way.
>>>> I know a major firm that has pools of hundreds of test machines and
>>>> their own tools to image and run tests, for years when they had a new
>>>> driver for Windows they would use the whole pool to stress test the
>>>> driver. Now thanks to DTM being in the way, they have a pool of 30
>>>> systems for Windows testing, and reserve the hundreds for other OS’es
>>>> that do have the DTM problems.
>>>>
>>>> DTM IMHO has done more to set back the quality of Windows drivers than
>>>> anything Microsoft has done.
>>>>
>>>>
>>>> –
>>>> Don Burn (MVP, Windows DDK)
>>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>>> Website: http://www.windrvr.comhttp:</http:>
>>>> Blog: http://msmvps.com/blogs/WinDrvr
>>>> Remove StopSpam to reply
>>>>
>>>>
>>>>
>>>>
>>>> “Bill McKenzie” > wrote in message
>>>> news:xxxxx@ntdev…
>>>>>I actually LOVE static analysis tools. I just don’t understand why
>>>>>they don’t get some real kernel devs on them and tailor them
>>>>>appropriately. I think PreFast is a beautiful tool! I don’t even
>>>>>bother with the SAL annotation much (except for the buffer size stuff
>>>>>that is pretty cool) and I still find it terribly useful. I mean, it
>>>>>just catches stupid errors mostly, but that is SO much better to catch
>>>>>at build time and you can easily hook it into the build process to
>>>>>automatically run. The false positives are a good bit annoying, they
>>>>>could trim that a LOT.
>>>>>
>>>>> SDV seems like a great idea, but horribly horribly broken. I, like
>>>>> Peter, have never gotten anything useful out of it. But the idea
>>>>> seems to have a lot of promise. Just needs some serious and directed
>>>>> effort behind it.
>>>>>
>>>>> Run time tools are awesome! I will use all I can get my hands on and
>>>>> I do. But static analysis tools that can be automated into a build are
>>>>> simply beautiful! I wish I had a better handle on just how broad a
>>>>> spectrum of errors those tools could find in a reasonable timeframe
>>>>> given sufficiently sophisticated tools. I really don’t have a good
>>>>> idea where those bounds are. But, if the potential error catching
>>>>> surface is large…I say poor all the resources you can into that!
>>>>>
>>>>> For runtime tools…I would LOVE to see some kind of integration into
>>>>> DTM, so again, we could automate the running and retrieving of
>>>>> results!!
>>>>>
>>>>> For that matter, how about fixing and/or finishing the COM interface
>>>>> to DTM so automating the running of DTM is easier to achieve?
>>>>>
>>>>> Bill M.
>>>>>
>>>>>
>>>>>
>>>>> “Don Burn” > wrote in message news:xxxxx@ntdev…
>>>>>> The number of tools that would be nice out of Microsoft is too large
>>>>>> to post, a few over the years I have suggested to them are:
>>>>>>
>>>>>> 1. Code coverage - yes Bullseye is nice, but even with it we need
>>>>>> faut injection. Note they looked at fault injection and rejected it
>>>>>> because people would have to modify their code (I guess annotations
>>>>>> don’t count as modification)
>>>>>>
>>>>>> 2. Lock contention measurement - the number of times I have seen
>>>>>> people propose a gazzilion locks for a driver because they do not
>>>>>> want contention. A tool that shows how long locks are held (max and
>>>>>> average) and how long someone waits for locks would probably slow
>>>>>> down some of these lets have a seperate lock for everything.
>>>>>>
>>>>>> 3. Call usage verifier - this was a godd tool, it could have been
>>>>>> expanded a lot. Come on Redmond either bring it forward, or release
>>>>>> it back to OSR so they can have the OSR WDK (compared to the OSR DDK
>>>>>> that CUV came from).
>>>>>>
>>>>>> Perhaps we need an email list for such tools, complete with a legal
>>>>>> agreement that releases any thing people say, so Redmond or others
>>>>>> can act on it.
>>>>>>
>>>>>>
>>>>>> –
>>>>>> Don Burn (MVP, Windows DDK)
>>>>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>>>>> Website: http://www.windrvr.comhttp:</http:>
>>>>>> Blog: http://msmvps.com/blogs/WinDrvr
>>>>>> Remove StopSpam to reply
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> “Martin O’Brien” > wrote in message
>>>>>> news:xxxxx@ntdev…
>>>>>>> +1
>>>>>>>
>>>>>>> I’d add that some complete and accurate WinDbg documentation might
>>>>>>> be nice, as would the same for the KRM.
>>>>>>>
>>>>>>>
>>>>>>> mm
>>>>>>>
>>>>>>> xxxxx@osr.commailto:xxxxx wrote:
>>>>>>>> I agree with Mr. O’Brien. The focus of the DDC is clearly static
>>>>>>>> analysis, because if we’re not using it, that’s obviously because
>>>>>>>> we’re dumb out here and slow on the uptake.
>>>>>>>>
>>>>>>>> The distortion field is clearly alive and well. Sigh!
>>>>>>>>
>>>>>>>> As usual, Microsoft would be well served by REALLY understanding
>>>>>>>> what we DO out here. They THINK they understand, but they don’t.
>>>>>>>>
>>>>>>>> As I’ve said before, I like and respect the folks who do SDV and
>>>>>>>> PreFast. I really do. They’re very smart… most of them smarter
>>>>>>>> than me. But they’d just be soooo surprised if they could spend
>>>>>>>> some time, like maybe a solid 3 months, out in the real world
>>>>>>>> trying to USE these tools to do useful work for pay.
>>>>>>>>
>>>>>>>> PreFast: Amusing, sometimes useful, but annoying. For maximum
>>>>>>>> effectiveness and minimum annoyance have to “buy in” to using it
>>>>>>>> throughout the life-cycle of your driver. You have to believe that
>>>>>>>> SAL notations are a good thing. I am not convinced.
>>>>>>>>
>>>>>>>> SDV: Complicated, difficult, and VERY annoying. I have never –
>>>>>>>> not once – been able to get SDV to disclose a meaningful error to
>>>>>>>> me. This is true even when I try to create specific errors in my
>>>>>>>> code that the documentation CLEARLY says SDV will detect and
>>>>>>>> report.
>>>>>>>>
>>>>>>>> I intellectually understand who/why/where these tools could be
>>>>>>>> useful. I suspect that it’ll take about another ten years before I
>>>>>>>> find them useful in practical situations, however.
>>>>>>>>
>>>>>>>> If I were king of the group responsible for these tools, I would
>>>>>>>> redirect their efforts to building clever and comprehensive DYNAMIC
>>>>>>>> analysis and debugging tools that we can use out here in the real
>>>>>>>> world, without significant up-front “adopt it for life or it’s not
>>>>>>>> useful”, requirements.
>>>>>>>>
>>>>>>>> These dynamic tools are really what the community needs at this
>>>>>>>> point. WAY MORE checks in KMDF Verifier. A LOCKING HIERARCHY
>>>>>>>> VALIDATION scheme (OSR has a patent pending on one, we’ll be happy
>>>>>>>> to license it) for ALL sorts of locks. MORE speculative checking
>>>>>>>> options in Driver Verifier. Resurrect Call Usage Verifier – now
>>>>>>>> discontinued. And… just to show you how easy it is to miss the
>>>>>>>> simple stuff: WHY are things like IoGetNextIrpStackLocation still
>>>>>>>> MACROs?? Why are ANY of the DDK functions macros? When they’re
>>>>>>>> macros, they’re impossible to “hook” with Verifier. At LEAST make
>>>>>>>> them functions, and mark them as “inline” in the free build. There
>>>>>>>> is NO additional overhead to this, and much to be gained. This
>>>>>>>> seems obvious to me (and, in fact, was an actual work item for the
>>>>>>>> DDK back, oh, about a thousand years ago).
>>>>>>>>
>>>>>>>> If any of the static analysis folks would like to come to OSR to
>>>>>>>> spend a couple of months writing code on real customer projects,
>>>>>>>> and see “how the other half lives”, I’ll be happy to provide the
>>>>>>>> office and project management. I’ll even give them free lunch for
>>>>>>>> the duration of their stay (like every other OSR employee). Maybe
>>>>>>>> they’ll gain some insight. Then again, maybe they’ll convince us
>>>>>>>> that PreFast or SDV will make our drivers or file systems better
>>>>>>>> and I’ll have to write a big long article in The NT Insider about
>>>>>>>> how SDV changed my life.
>>>>>>>>
>>>>>>>> Peter
>>>>>>>> OSR
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

–
Mark Roddy
— NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx></mailto:xxxxx></mailto:xxxxx>

[Mark_Roddy]

WPP is a perfect example of a well intentioned but vastly overcomplicated
solution to a simple problem that had a well known and easily implementable
solution.

They didn’t need anything more complicated than a kernel ringbuffer to
capture the output from debugprint, a spooler service to transfer the
contents of that ringbuffer to a log file in real time, and a debugger
command to display the contents of the ringbuffer in the correct sequence
from a live system or a crash dump. Instead we got WPP or ETW or whatever it
is called. So everyone implements their own private debug logging, or
struggles with the overcomplicated and nearly unusable WPP/ETW, and the
problem of a reliable easy to use and universal kernel logging facility
remains unresolved.

On Fri, Sep 26, 2008 at 12:51 PM, Michal Vodicka wrote:

> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of
> > xxxxx@osr.com
> > Sent: Friday, September 26, 2008 4:21 PM
> > To: Windows System Software Devs Interest List
> > Subject: RE:[ntdev] DDC - updated info for next week
> >
> > You guys could fix this by inserting a DbgPrint before each
> > NT_ASSERT in the checked build (only). Or, just change the
> > freakin’ NT_ASSERT macro.
>
> …and replace __annotation there with DbgPrint for checked build (at
> least). All this evil started with introducing undocumented__annotation
> keyword. It demonstrates how too much power replaces careful thinking.
> The result is useless implementation of a good idea (WPP using own
> preprocessor) and the checked build deterioration we see. Just because
> they had power to change compiler for their needs. I’m pretty sure the
> same problem can be solved using standard tools because we had to did it
> for our firmware where was no place for debug strings in the ROM.
>
> > I’ve asked about this, repeatedly, both BEFORE the move to
> > WPP and as recently as about a year ago. And all I’ve EVER
> > heard folks say is “Oh, yeah… That IS a problem. Hmmm… I
> > wonder if we can ship those TMFs… I’ll look into that.”
> > The problem is, nobody’s ever been willing take this on as an
> > action item and expend some political capital to try to make
> > it happen (and, we all know that’s what it would take to get it done).
>
> DDC can be a good opportunity to raise this issue again. When we were at
> MVP global summit at spring, I had a good feeling they were really
> interested about our feedback. On the other hand, I’m not sure if any of
> issues discussed there was already solved.
>
> All depends on people. We see MS as a company but such decisions and
> changes are made by individuals. You’re right it is necessary to
> convince right people to invest time and resources to make such a fix.
> Not easy thing. They should understand it is a problem which can lead to
> lower quality of 3rd party drivers and worse perception of Windows by
> end users.
>
> Best regards,
>
> Michal Vodicka
> UPEK, Inc.
> [xxxxx@upek.com, http://www.upek.com]
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

–
Mark Roddy

[Michal_Vodicka-2]

What “if someone ignores”?

In my case DTM leads to quality improvements. That’s becase my coworker
who makes DTM tests is more and more mad about it and I’m affraid to
tell him we need to sign my driver again. He could kill me once so
having bug free code is vitally important for me

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com http:</http:>]

---

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Skywing
Sent: Friday, September 26, 2008 7:17 PM
To: Windows System Software Devs Interest List
Subject: RE: Re:[ntdev] DDC - updated info for next week

However, the whole “if someone disagrees, say it again, louder”
is not particularly productive.

• S

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark Roddy
Sent: Friday, September 26, 2008 1:15 PM
To: Windows System Software Devs Interest List
Subject: Re: Re:[ntdev] DDC - updated info for next week

When DTM is involved: yes.

On Fri, Sep 26, 2008 at 1:01 PM, Peter Wieland
wrote:

Is the shouting really necessary Don?

-p

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Friday, September 26, 2008 8:57 AM
To: Windows System Software Devs Interest List

Subject: Re:[ntdev] DDC - updated info for next week

I’VE DEALT WITH THAT PIECE OF SHIT KNOWN AS THE LATEST WLK WITH
MY CUSTOMERS
MY OPINION STANDS.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
Website: http://www.windrvr.com http:</http:>
Blog: http://msmvps.com/blogs/WinDrvr

“Bill McKenzie” wrote in message
news:xxxxx@ntdev…
> Again, I tried DTM, I don’t know about 3 or 4 years ago, and
it was an
> abortion. Amazingly, however, they REALLY have improved it to
the point
> where I don’t think I want to live without it now.
>
> As far as HCT vs DTM, I don’t even think there is a
comparison. What used
> to take me days or even sometimes weeks with HCT tests (IF I
could get a
> magic setup to ever work with HCTs and dedicate that magic
setup to
> nothing but HCTs for the rest of its life and pray it never
breaks) I can
> do in hours with DTM.
>
> DTM isn’t bug free by a long shot, but I find it orders of
magnitude
> easier to setup and use than HCTs. Maybe it is the types of
drivers we
> are testing?
>
> At any rate, if you haven’t tried the latest, you haven’t used
DTM…it
> is a LOT better now than even just one verion ago.
>
> Bill M.
>
> “Don Burn” wrote in message
news:xxxxx@ntdev…
>> Bill,
>>
>> To be honest I no longer directly use DTM. I used to try
to run it
>> but have given up, since I have to tear my whole test lab
apart to even
>> try. What I do have is a number of customers who I work with
who run it,
>> and from their experiences I no longer go there. In
particular:
>>
>> One large firm, who has set aside the hardware and the
MIS
>> resources. They don’t have too many problems, but have
commented that
>> every so often a hotfix for the OS destabilizes the world.
Their normal
>> approach is to then use MIS to figure out which hotfix and
reconfigure
>> the systems.
>>
>> One medium size firm that pays a consulting firm to do
it. They
>> have a fixed locked down test network and things mostly work.
>>
>> Several small firms, that assemble test configurations
when they
>> need to. Most of them swear they would become pure LINUX if
the market
>> could support them, after each attempt at DTM. The ones who
have very
>> large servers and setup a domain controller find it painful
but doable.
>> The ones who have small systems and do not want a domain
controller spend
>> weeks (I’ve been there with them through it) and sometimes
finally say
>> “to hell with Redmond” and move on planning to come back in 6
months to a
>> year.
>>
>> I used to run HCT’s for the clients and ensure that the
drivers
>> would pass WHQL when they got them. I no longer do because
of the
>> problems with DTM. But DTM is actually a reflection of the
bigger
>> problem with WHQL and driver testing. For the last 10 years
at least
>> WHQL has continued to modify the test environment first HCT’s
then DTM to
>> make their life easer at the expense of driver quality. The
first
>> principal of quality if find bugs as early in the process as
you can, but
>> WHQL testing continues to evolve to more and more complex
environments
>> which push the use of the tests later and later into the
cycle.
>>
>> If Microsoft spent as much money on good tests for
drivers as it has
>> in the last 10 years making WHQL testing more automated for
them we would
>> have a lot more tests that driver writers could use, and a
related
>> increase in the quality of drivers. Instead as we go forward
they make
>> the tests more and more useless, for instance the fact that
the various
>> test harnesses from HCT10 on do not run on a checked build to
me is
>> ridiculous.
>>
>> At this point, if Redmond want driver quality the
simplest first
>> step would be to fire everyone in WHQL and start over.
>>
>>
>> –
>> Don Burn (MVP, Windows DDK)
>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>> Website: http://www.windrvr.com http:</http:>
>> Blog: http://msmvps.com/blogs/WinDrvr
>>
>>
>> “Bill McKenzie” wrote in message
>> news:xxxxx@ntdev…
>>> Don,
>>>
>>> I really have to disagree on DTM. When was the last time
you used it?
>>> I used to have a lot of problems with it, but I find the
latest DTM to
>>> be a VERY useful tool. I used it recently as the base of an
automated
>>> testing framework for my company’s Windows client software.
Using
>>> custom scripts I can test everything from the UI to the
drivers in a
>>> completely automated fashion. It is great for controlling
multiple
>>> platforms and collating results. Now, I don’t know what its
limitations
>>> are as far as number of test platforms, we never used more
than say 10.
>>> But, all in all for what I needed the tool was great. The
only real
>>> problem I had was programmatically controlling DTM from its
external COM
>>> interface. THAT is a disaster!
>>>
>>> Bill M.
>>>
>>> “Don Burn” wrote in message
news:xxxxx@ntdev…
>>>> How about scrapping DTM and putting together a real test
framework?
>>>> DTM is horribly over engineered for anyone other than WHQL,
it just
>>>> gets in the way. For those of us who are small
environments, setting
>>>> it iup is horribly complicated.
>>>>
>>>> For firms that have large test environments, it still gets
in the way.
>>>> I know a major firm that has pools of hundreds of test
machines and
>>>> their own tools to image and run tests, for years when they
had a new
>>>> driver for Windows they would use the whole pool to stress
test the
>>>> driver. Now thanks to DTM being in the way, they have a
pool of 30
>>>> systems for Windows testing, and reserve the hundreds for
other OS’es
>>>> that do have the DTM problems.
>>>>
>>>> DTM IMHO has done more to set back the quality of Windows
drivers than
>>>> anything Microsoft has done.
>>>>
>>>>
>>>> –
>>>> Don Burn (MVP, Windows DDK)
>>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>>> Website: http://www.windrvr.com http:</http:>
>>>> Blog: http://msmvps.com/blogs/WinDrvr
>>>> Remove StopSpam to reply
>>>>
>>>>
>>>>
>>>>
>>>> “Bill McKenzie” wrote in message
>>>> news:xxxxx@ntdev…
>>>>>I actually LOVE static analysis tools. I just don’t
understand why
>>>>>they don’t get some real kernel devs on them and tailor
them
>>>>>appropriately. I think PreFast is a beautiful tool! I
don’t even
>>>>>bother with the SAL annotation much (except for the buffer
size stuff
>>>>>that is pretty cool) and I still find it terribly useful.
I mean, it
>>>>>just catches stupid errors mostly, but that is SO much
better to catch
>>>>>at build time and you can easily hook it into the build
process to
>>>>>automatically run. The false positives are a good bit
annoying, they
>>>>>could trim that a LOT.
>>>>>
>>>>> SDV seems like a great idea, but horribly horribly broken.
I, like
>>>>> Peter, have never gotten anything useful out of it. But
the idea
>>>>> seems to have a lot of promise. Just needs some serious
and directed
>>>>> effort behind it.
>>>>>
>>>>> Run time tools are awesome! I will use all I can get my
hands on and
>>>>> I do. But static analysis tools that can be automated into
a build are
>>>>> simply beautiful! I wish I had a better handle on just
how broad a
>>>>> spectrum of errors those tools could find in a reasonable
timeframe
>>>>> given sufficiently sophisticated tools. I really don’t
have a good
>>>>> idea where those bounds are. But, if the potential error
catching
>>>>> surface is large…I say poor all the resources you can
into that!
>>>>>
>>>>> For runtime tools…I would LOVE to see some kind of
integration into
>>>>> DTM, so again, we could automate the running and
retrieving of
>>>>> results!!
>>>>>
>>>>> For that matter, how about fixing and/or finishing the COM
interface
>>>>> to DTM so automating the running of DTM is easier to
achieve?
>>>>>
>>>>> Bill M.
>>>>>
>>>>>
>>>>>
>>>>> “Don Burn” wrote in message
news:xxxxx@ntdev…
>>>>>> The number of tools that would be nice out of Microsoft
is too large
>>>>>> to post, a few over the years I have suggested to them
are:
>>>>>>
>>>>>> 1. Code coverage - yes Bullseye is nice, but even with
it we need
>>>>>> faut injection. Note they looked at fault injection and
rejected it
>>>>>> because people would have to modify their code (I guess
annotations
>>>>>> don’t count as modification)
>>>>>>
>>>>>> 2. Lock contention measurement - the number of times I
have seen
>>>>>> people propose a gazzilion locks for a driver because
they do not
>>>>>> want contention. A tool that shows how long locks are
held (max and
>>>>>> average) and how long someone waits for locks would
probably slow
>>>>>> down some of these lets have a seperate lock for
everything.
>>>>>>
>>>>>> 3. Call usage verifier - this was a godd tool, it could
have been
>>>>>> expanded a lot. Come on Redmond either bring it forward,
or release
>>>>>> it back to OSR so they can have the OSR WDK (compared to
the OSR DDK
>>>>>> that CUV came from).
>>>>>>
>>>>>> Perhaps we need an email list for such tools, complete
with a legal
>>>>>> agreement that releases any thing people say, so Redmond
or others
>>>>>> can act on it.
>>>>>>
>>>>>>
>>>>>> –
>>>>>> Don Burn (MVP, Windows DDK)
>>>>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>>>>> Website: http://www.windrvr.com http:</http:>

>>>>>> Blog: http://msmvps.com/blogs/WinDrvr
>>>>>> Remove StopSpam to reply
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> “Martin O’Brien” wrote in
message
>>>>>> news:xxxxx@ntdev…
>>>>>>> +1
>>>>>>>
>>>>>>> I’d add that some complete and accurate WinDbg
documentation might
>>>>>>> be nice, as would the same for the KRM.
>>>>>>>
>>>>>>>
>>>>>>> mm
>>>>>>>
>>>>>>> xxxxx@osr.com wrote:
>>>>>>>> I agree with Mr. O’Brien. The focus of the DDC is
clearly static
>>>>>>>> analysis, because if we’re not using it, that’s
obviously because
>>>>>>>> we’re dumb out here and slow on the uptake.
>>>>>>>>
>>>>>>>> The distortion field is clearly alive and well. Sigh!
>>>>>>>>
>>>>>>>> As usual, Microsoft would be well served by REALLY
understanding
>>>>>>>> what we DO out here. They THINK they understand, but
they don’t.
>>>>>>>>
>>>>>>>> As I’ve said before, I like and respect the folks who
do SDV and
>>>>>>>> PreFast. I really do. They’re very smart… most of
them smarter
>>>>>>>> than me. But they’d just be soooo surprised if they
could spend
>>>>>>>> some time, like maybe a solid 3 months, out in the real
world
>>>>>>>> trying to USE these tools to do useful work for pay.
>>>>>>>>
>>>>>>>> PreFast: Amusing, sometimes useful, but annoying. For
maximum
>>>>>>>> effectiveness and minimum annoyance have to “buy in” to
using it
>>>>>>>> throughout the life-cycle of your driver. You have to
believe that
>>>>>>>> SAL notations are a good thing. I am not convinced.
>>>>>>>>
>>>>>>>> SDV: Complicated, difficult, and VERY annoying. I have
never –
>>>>>>>> not once – been able to get SDV to disclose a
meaningful error to
>>>>>>>> me. This is true even when I try to create specific
errors in my
>>>>>>>> code that the documentation CLEARLY says SDV will
detect and
>>>>>>>> report.
>>>>>>>>
>>>>>>>> I intellectually understand who/why/where these tools
could be
>>>>>>>> useful. I suspect that it’ll take about another ten
years before I
>>>>>>>> find them useful in practical situations, however.
>>>>>>>>
>>>>>>>> If I were king of the group responsible for these
tools, I would
>>>>>>>> redirect their efforts to building clever and
comprehensive DYNAMIC
>>>>>>>> analysis and debugging tools that we can use out here
in the real
>>>>>>>> world, without significant up-front “adopt it for life
or it’s not
>>>>>>>> useful”, requirements.
>>>>>>>>
>>>>>>>> These dynamic tools are really what the community needs
at this
>>>>>>>> point. WAY MORE checks in KMDF Verifier. A LOCKING
HIERARCHY
>>>>>>>> VALIDATION scheme (OSR has a patent pending on one,
we’ll be happy
>>>>>>>> to license it) for ALL sorts of locks. MORE speculative
checking
>>>>>>>> options in Driver Verifier. Resurrect Call Usage
Verifier – now
>>>>>>>> discontinued. And… just to show you how easy it is
to miss the
>>>>>>>> simple stuff: WHY are things like
IoGetNextIrpStackLocation still
>>>>>>>> MACROs?? Why are ANY of the DDK functions macros?
When they’re
>>>>>>>> macros, they’re impossible to “hook” with Verifier. At
LEAST make
>>>>>>>> them functions, and mark them as “inline” in the free
build. There
>>>>>>>> is NO additional overhead to this, and much to be
gained. This
>>>>>>>> seems obvious to me (and, in fact, was an actual work
item for the
>>>>>>>> DDK back, oh, about a thousand years ago).
>>>>>>>>
>>>>>>>> If any of the static analysis folks would like to come
to OSR to
>>>>>>>> spend a couple of months writing code on real customer
projects,
>>>>>>>> and see “how the other half lives”, I’ll be happy to
provide the
>>>>>>>> office and project management. I’ll even give them
free lunch for
>>>>>>>> the duration of their stay (like every other OSR
employee). Maybe
>>>>>>>> they’ll gain some insight. Then again, maybe they’ll
convince us
>>>>>>>> that PreFast or SDV will make our drivers or file
systems better
>>>>>>>> and I’ll have to write a big long article in The NT
Insider about
>>>>>>>> how SDV changed my life.
>>>>>>>>
>>>>>>>> Peter
>>>>>>>> OSR
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>

—

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars
visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars
visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

–
Mark Roddy

— NTDEV is sponsored by OSR For our schedule of WDF, WDM,
debugging and other seminars visit: http://www.osr.com/seminars To
unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars
visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

[Michal_Vodicka-2]

---

From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Mark Roddy
Sent: Friday, September 26, 2008 7:23 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] DDC - updated info for next week

WPP is a perfect example of a well intentioned but vastly
overcomplicated solution to a simple problem that had a well known and
easily implementable solution.

They didn’t need anything more complicated than a kernel
ringbuffer to capture the output from debugprint, a spooler service to
transfer the contents of that ringbuffer to a log file in real time, and
a debugger command to display the contents of the ringbuffer in the
correct sequence from a live system or a crash dump.

This is all done by DebugView which is now MS. The only thing which’d be
helpful is standardized control of debug output so user/developer can
easily say what traces wants to display. If whole OS and 3rd patry
drivers use the same scheme, it’d be really useful. App or just registry
editing. It’d be progress against the necessity to change debug mask in
memory or trying to find undocumented registry settings.

Instead, we have WPP which brings nothing but pain. And uncontrollable
MS drivers and software which still spew to debug output. If MS use
debug output internally, it’d be solved sooner or later. With WPP it
will probably remain forever :-#

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com http:</http:>]