Bill,
I have never done anything with an 802.11 device, so no. But try some
of the simpler things like disk, and quickly become suicidal or homicidal.
–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr
“Bill McKenzie” wrote in message
news:xxxxx@ntdev…
> HCT’s while pretty bad could be set up in less than an hour, now I
>> tell my customers (who don’‘t have full time MIS staffs) to plan for 2
>> weeks, personally that indicates something terribly wrong in my opinion.
>
> Okay I’ll bite. Did you ever test 802.11 devices with HCT’s? OMG if you
> got a setup working EVER you did great! We had a guy at Conexant that got
> a setup to work at his house and so we had to have him take the hardware
> home anytime we needed to re-WHQL. I have literally WHQL’d dozens of
> drivers of varying varieties (WDM filter, AVStreaming, USB drivers of
> several types, and others) with DTM and have never had anywhere near the
> issues I had with the HCTs. DTM is trivial to setup in comparison and I
> have used both personally. I still have problems with DTM not creating
> submissions correctly and having to re-start the UI and such, but those
> minor bugs are an annoyance. The fact that I can make a change in a
> driver, re-run the DTM tests, submit my new package and receive my newly
> signed driver(s) all in less than a day in some instances is astounding.
> The tool itself has been very well thought out and architected. I can
> test anything on Windows with DTM easily and it allows me to control the
> tests through reboots and user login changes, it allows me to push whole
> operating systems to platforms if I so choose, I can select to run on
> given platform types…its just a nice control vehicle. Some of the tests
> probably need a LOT of work, but compared to a lot of driver tools and for
> what it is I find it to work marvelously.
>
> Bill M.
>
>
> “Don Burn” wrote in message news:xxxxx@ntdev…
>> Peter,
>>
>> A lot of us have complained about DTM for 5.5 years since Jean
>> Valentine first presented it at WinHEC, but little has been done to
>> address problems pointed out back then. Perhaps shouting should have
>> been done more? HCT’s while pretty bad could be set up in less than an
>> hour, now I tell my customers (who don’'t have full time MIS staffs) to
>> plan for 2 weeks, personally that indicates something terribly wrong in
>> my opinion.
>>
>> I fully support the idea of driver testing and logo’ing but it needs
>> to test something. I have seen little evidence that most of the WHQL
>> testing (this does not include forcing drivers to run under Verifier or
>> things like Prefast) does anything to test the driver for quality.
>> Microsoft has done a lot of fantastic work in the last 5.5 years, the
>> whole WDF effort is a great example, but WHQL has not done anything to
>> improve the quality of Windows. Considering that Windows is hammered by
>> proponents of tha other OS for quality this says to me that WHQL is
>> giving the illusion that Redmond is working on it without actually doing
>> anything, a classic example of the reality distortion field at work.
>>
>>
>> –
>> Don Burn (MVP, Windows DDK)
>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>> Website: http://www.windrvr.com
>> Blog: http://msmvps.com/blogs/WinDrvr
>>
>>
>>
>> “Peter Wieland” wrote in message
>> news:xxxxx@ntdev…
>> Is the shouting really necessary Don?
>>
>> -p
>>
>> -----Original Message-----
>> From: xxxxx@lists.osr.com
>> [mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
>> Sent: Friday, September 26, 2008 8:57 AM
>> To: Windows System Software Devs Interest List
>> Subject: Re:[ntdev] DDC - updated info for next week
>>
>> I’VE DEALT WITH THAT PIECE OF SHIT KNOWN AS THE LATEST WLK WITH MY
>> CUSTOMERS
>> MY OPINION STANDS.
>>
>>
>> –
>> Don Burn (MVP, Windows DDK)
>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>> Website: http://www.windrvr.com
>> Blog: http://msmvps.com/blogs/WinDrvr
>>
>>
>> “Bill McKenzie” wrote in message
>> news:xxxxx@ntdev…
>>> Again, I tried DTM, I don’t know about 3 or 4 years ago, and it was an
>>> abortion. Amazingly, however, they REALLY have improved it to the point
>>> where I don’t think I want to live without it now.
>>>
>>> As far as HCT vs DTM, I don’t even think there is a comparison. What
>>> used
>>> to take me days or even sometimes weeks with HCT tests (IF I could get a
>>> magic setup to ever work with HCTs and dedicate that magic setup to
>>> nothing but HCTs for the rest of its life and pray it never breaks) I
>>> can
>>> do in hours with DTM.
>>>
>>> DTM isn’t bug free by a long shot, but I find it orders of magnitude
>>> easier to setup and use than HCTs. Maybe it is the types of drivers we
>>> are testing?
>>>
>>> At any rate, if you haven’t tried the latest, you haven’t used DTM…it
>>> is a LOT better now than even just one verion ago.
>>>
>>> Bill M.
>>>
>>> “Don Burn” wrote in message news:xxxxx@ntdev…
>>>> Bill,
>>>>
>>>> To be honest I no longer directly use DTM. I used to try to run it
>>>> but have given up, since I have to tear my whole test lab apart to even
>>>> try. What I do have is a number of customers who I work with who run
>>>> it,
>>>> and from their experiences I no longer go there. In particular:
>>>>
>>>> One large firm, who has set aside the hardware and the MIS
>>>> resources. They don’t have too many problems, but have commented that
>>>> every so often a hotfix for the OS destabilizes the world. Their
>>>> normal
>>>> approach is to then use MIS to figure out which hotfix and reconfigure
>>>> the systems.
>>>>
>>>> One medium size firm that pays a consulting firm to do it. They
>>>> have a fixed locked down test network and things mostly work.
>>>>
>>>> Several small firms, that assemble test configurations when they
>>>> need to. Most of them swear they would become pure LINUX if the market
>>>> could support them, after each attempt at DTM. The ones who have very
>>>> large servers and setup a domain controller find it painful but doable.
>>>> The ones who have small systems and do not want a domain controller
>>>> spend
>>>> weeks (I’ve been there with them through it) and sometimes finally say
>>>> “to hell with Redmond” and move on planning to come back in 6 months to
>>>> a
>>>> year.
>>>>
>>>> I used to run HCT’s for the clients and ensure that the drivers
>>>> would pass WHQL when they got them. I no longer do because of the
>>>> problems with DTM. But DTM is actually a reflection of the bigger
>>>> problem with WHQL and driver testing. For the last 10 years at least
>>>> WHQL has continued to modify the test environment first HCT’s then DTM
>>>> to
>>>> make their life easer at the expense of driver quality. The first
>>>> principal of quality if find bugs as early in the process as you can,
>>>> but
>>>> WHQL testing continues to evolve to more and more complex environments
>>>> which push the use of the tests later and later into the cycle.
>>>>
>>>> If Microsoft spent as much money on good tests for drivers as it
>>>> has
>>>> in the last 10 years making WHQL testing more automated for them we
>>>> would
>>>> have a lot more tests that driver writers could use, and a related
>>>> increase in the quality of drivers. Instead as we go forward they make
>>>> the tests more and more useless, for instance the fact that the various
>>>> test harnesses from HCT10 on do not run on a checked build to me is
>>>> ridiculous.
>>>>
>>>> At this point, if Redmond want driver quality the simplest first
>>>> step would be to fire everyone in WHQL and start over.
>>>>
>>>>
>>>> –
>>>> Don Burn (MVP, Windows DDK)
>>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>>> Website: http://www.windrvr.com
>>>> Blog: http://msmvps.com/blogs/WinDrvr
>>>>
>>>>
>>>> “Bill McKenzie” wrote in message
>>>> news:xxxxx@ntdev…
>>>>> Don,
>>>>>
>>>>> I really have to disagree on DTM. When was the last time you used it?
>>>>> I used to have a lot of problems with it, but I find the latest DTM to
>>>>> be a VERY useful tool. I used it recently as the base of an automated
>>>>> testing framework for my company’s Windows client software. Using
>>>>> custom scripts I can test everything from the UI to the drivers in a
>>>>> completely automated fashion. It is great for controlling multiple
>>>>> platforms and collating results. Now, I don’t know what its
>>>>> limitations
>>>>> are as far as number of test platforms, we never used more than say
>>>>> 10.
>>>>> But, all in all for what I needed the tool was great. The only real
>>>>> problem I had was programmatically controlling DTM from its external
>>>>> COM
>>>>> interface. THAT is a disaster!
>>>>>
>>>>> Bill M.
>>>>>
>>>>> “Don Burn” wrote in message news:xxxxx@ntdev…
>>>>>> How about scrapping DTM and putting together a real test framework?
>>>>>> DTM is horribly over engineered for anyone other than WHQL, it just
>>>>>> gets in the way. For those of us who are small environments,
>>>>>> setting
>>>>>> it iup is horribly complicated.
>>>>>>
>>>>>> For firms that have large test environments, it still gets in the
>>>>>> way.
>>>>>> I know a major firm that has pools of hundreds of test machines and
>>>>>> their own tools to image and run tests, for years when they had a new
>>>>>> driver for Windows they would use the whole pool to stress test the
>>>>>> driver. Now thanks to DTM being in the way, they have a pool of 30
>>>>>> systems for Windows testing, and reserve the hundreds for other OS’es
>>>>>> that do have the DTM problems.
>>>>>>
>>>>>> DTM IMHO has done more to set back the quality of Windows drivers
>>>>>> than
>>>>>> anything Microsoft has done.
>>>>>>
>>>>>>
>>>>>> –
>>>>>> Don Burn (MVP, Windows DDK)
>>>>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>>>>> Website: http://www.windrvr.com
>>>>>> Blog: http://msmvps.com/blogs/WinDrvr
>>>>>> Remove StopSpam to reply
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> “Bill McKenzie” wrote in message
>>>>>> news:xxxxx@ntdev…
>>>>>>>I actually LOVE static analysis tools. I just don’t understand why
>>>>>>>they don’t get some real kernel devs on them and tailor them
>>>>>>>appropriately. I think PreFast is a beautiful tool! I don’t even
>>>>>>>bother with the SAL annotation much (except for the buffer size stuff
>>>>>>>that is pretty cool) and I still find it terribly useful. I mean, it
>>>>>>>just catches stupid errors mostly, but that is SO much better to
>>>>>>>catch
>>>>>>>at build time and you can easily hook it into the build process to
>>>>>>>automatically run. The false positives are a good bit annoying, they
>>>>>>>could trim that a LOT.
>>>>>>>
>>>>>>> SDV seems like a great idea, but horribly horribly broken. I, like
>>>>>>> Peter, have never gotten anything useful out of it. But the idea
>>>>>>> seems to have a lot of promise. Just needs some serious and
>>>>>>> directed
>>>>>>> effort behind it.
>>>>>>>
>>>>>>> Run time tools are awesome! I will use all I can get my hands on
>>>>>>> and
>>>>>>> I do. But static analysis tools that can be automated into a build
>>>>>>> are
>>>>>>> simply beautiful! I wish I had a better handle on just how broad a
>>>>>>> spectrum of errors those tools could find in a reasonable timeframe
>>>>>>> given sufficiently sophisticated tools. I really don’t have a good
>>>>>>> idea where those bounds are. But, if the potential error catching
>>>>>>> surface is large…I say poor all the resources you can into that!
>>>>>>>
>>>>>>> For runtime tools…I would LOVE to see some kind of integration
>>>>>>> into
>>>>>>> DTM, so again, we could automate the running and retrieving of
>>>>>>> results!!
>>>>>>>
>>>>>>> For that matter, how about fixing and/or finishing the COM interface
>>>>>>> to DTM so automating the running of DTM is easier to achieve?
>>>>>>>
>>>>>>> Bill M.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> “Don Burn” wrote in message news:xxxxx@ntdev…
>>>>>>>> The number of tools that would be nice out of Microsoft is too
>>>>>>>> large
>>>>>>>> to post, a few over the years I have suggested to them are:
>>>>>>>>
>>>>>>>> 1. Code coverage - yes Bullseye is nice, but even with it we need
>>>>>>>> faut injection. Note they looked at fault injection and rejected
>>>>>>>> it
>>>>>>>> because people would have to modify their code (I guess annotations
>>>>>>>> don’t count as modification)
>>>>>>>>
>>>>>>>> 2. Lock contention measurement - the number of times I have seen
>>>>>>>> people propose a gazzilion locks for a driver because they do not
>>>>>>>> want contention. A tool that shows how long locks are held (max and
>>>>>>>> average) and how long someone waits for locks would probably slow
>>>>>>>> down some of these lets have a seperate lock for everything.
>>>>>>>>
>>>>>>>> 3. Call usage verifier - this was a godd tool, it could have been
>>>>>>>> expanded a lot. Come on Redmond either bring it forward, or
>>>>>>>> release
>>>>>>>> it back to OSR so they can have the OSR WDK (compared to the OSR
>>>>>>>> DDK
>>>>>>>> that CUV came from).
>>>>>>>>
>>>>>>>> Perhaps we need an email list for such tools, complete with a legal
>>>>>>>> agreement that releases any thing people say, so Redmond or others
>>>>>>>> can act on it.
>>>>>>>>
>>>>>>>>
>>>>>>>> –
>>>>>>>> Don Burn (MVP, Windows DDK)
>>>>>>>> Windows 2k/XP/2k3 Filesystem and Driver Consulting
>>>>>>>> Website: http://www.windrvr.com
>>>>>>>> Blog: http://msmvps.com/blogs/WinDrvr
>>>>>>>> Remove StopSpam to reply
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> “Martin O’Brien” wrote in message
>>>>>>>> news:xxxxx@ntdev…
>>>>>>>>> +1
>>>>>>>>>
>>>>>>>>> I’d add that some complete and accurate WinDbg documentation might
>>>>>>>>> be nice, as would the same for the KRM.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> mm
>>>>>>>>>
>>>>>>>>> xxxxx@osr.com wrote:
>>>>>>>>>> I agree with Mr. O’Brien. The focus of the DDC is clearly static
>>>>>>>>>> analysis, because if we’re not using it, that’s obviously because
>>>>>>>>>> we’re dumb out here and slow on the uptake.
>>>>>>>>>>
>>>>>>>>>> The distortion field is clearly alive and well. Sigh!
>>>>>>>>>>
>>>>>>>>>> As usual, Microsoft would be well served by REALLY understanding
>>>>>>>>>> what we DO out here. They THINK they understand, but they don’t.
>>>>>>>>>>
>>>>>>>>>> As I’ve said before, I like and respect the folks who do SDV and
>>>>>>>>>> PreFast. I really do. They’re very smart… most of them smarter
>>>>>>>>>> than me. But they’d just be soooo surprised if they could spend
>>>>>>>>>> some time, like maybe a solid 3 months, out in the real world
>>>>>>>>>> trying to USE these tools to do useful work for pay.
>>>>>>>>>>
>>>>>>>>>> PreFast: Amusing, sometimes useful, but annoying. For maximum
>>>>>>>>>> effectiveness and minimum annoyance have to “buy in” to using it
>>>>>>>>>> throughout the life-cycle of your driver. You have to believe
>>>>>>>>>> that
>>>>>>>>>> SAL notations are a good thing. I am not convinced.
>>>>>>>>>>
>>>>>>>>>> SDV: Complicated, difficult, and VERY annoying. I have never –
>>>>>>>>>> not once – been able to get SDV to disclose a meaningful error
>>>>>>>>>> to
>>>>>>>>>> me. This is true even when I try to create specific errors in my
>>>>>>>>>> code that the documentation CLEARLY says SDV will detect and
>>>>>>>>>> report.
>>>>>>>>>>
>>>>>>>>>> I intellectually understand who/why/where these tools could be
>>>>>>>>>> useful. I suspect that it’ll take about another ten years before
>>>>>>>>>> I
>>>>>>>>>> find them useful in practical situations, however.
>>>>>>>>>>
>>>>>>>>>> If I were king of the group responsible for these tools, I would
>>>>>>>>>> redirect their efforts to building clever and comprehensive
>>>>>>>>>> DYNAMIC
>>>>>>>>>> analysis and debugging tools that we can use out here in the real
>>>>>>>>>> world, without significant up-front “adopt it for life or it’s
>>>>>>>>>> not
>>>>>>>>>> useful”, requirements.
>>>>>>>>>>
>>>>>>>>>> These dynamic tools are really what the community needs at this
>>>>>>>>>> point. WAY MORE checks in KMDF Verifier. A LOCKING HIERARCHY
>>>>>>>>>> VALIDATION scheme (OSR has a patent pending on one, we’ll be
>>>>>>>>>> happy
>>>>>>>>>> to license it) for ALL sorts of locks. MORE speculative checking
>>>>>>>>>> options in Driver Verifier. Resurrect Call Usage Verifier – now
>>>>>>>>>> discontinued. And… just to show you how easy it is to miss the
>>>>>>>>>> simple stuff: WHY are things like IoGetNextIrpStackLocation still
>>>>>>>>>> MACROs?? Why are ANY of the DDK functions macros? When they’re
>>>>>>>>>> macros, they’re impossible to “hook” with Verifier. At LEAST
>>>>>>>>>> make
>>>>>>>>>> them functions, and mark them as “inline” in the free build.
>>>>>>>>>> There
>>>>>>>>>> is NO additional overhead to this, and much to be gained. This
>>>>>>>>>> seems obvious to me (and, in fact, was an actual work item for
>>>>>>>>>> the
>>>>>>>>>> DDK back, oh, about a thousand years ago).
>>>>>>>>>>
>>>>>>>>>> If any of the static analysis folks would like to come to OSR to
>>>>>>>>>> spend a couple of months writing code on real customer projects,
>>>>>>>>>> and see “how the other half lives”, I’ll be happy to provide the
>>>>>>>>>> office and project management. I’ll even give them free lunch
>>>>>>>>>> for
>>>>>>>>>> the duration of their stay (like every other OSR employee).
>>>>>>>>>> Maybe
>>>>>>>>>> they’ll gain some insight. Then again, maybe they’ll convince us
>>>>>>>>>> that PreFast or SDV will make our drivers or file systems better
>>>>>>>>>> and I’ll have to write a big long article in The NT Insider about
>>>>>>>>>> how SDV changed my life.
>>>>>>>>>>
>>>>>>>>>> Peter
>>>>>>>>>> OSR
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>>
>>
>>
>
>
>
