What is wrong with WHQL (was WHQL Certification)

NTDEV
1

The problem is that the change in WHQL is slower than the change in the
industry. Consider the following:

HT desktops from name brands were $750 last month.
A 64-bit desktop from HP was $1200, and now we have 64-bit laptops
With careful shopping I can get 4GB of memory for my server for roughly
$1200, so PAE is now mainstream.
Windows XP and 2003 now support headless, and we have TS on the other
end, but many tools including Microsoft assume a single desktop model
With laptop bases with PCI, as well as USB and 1394 every peripheral
needs to support hot plug but few drivers do

Now politically, I realize that WHQL cannot change the rules overnight and
say that all drivers must be MP, PAE, 64-bit and hot-plug tommorrow. But
perhaps it is time for WHQL to change their methodology on testing, in
particular:

  1. There should be tests for MP, PAE, 64-bit and hot-plug with the
    concept that right now failing them produces a warning (perhaps not public
    even) that the driver is remiss, and should be fixed because the times are
    changing.

  2. WHQL tests should be runnable standalone and report clearly the
    actual problem.

  3. WHQL test sources should be made available.

The DDK team has four times in the last two years had meetings with
developers to solicit input on their plans and what could be done better. I
challenge WHQL to have similar meetings at WinHEC and DevCon to get input on
how to make the testing better and more meaningful.

I ASK OTHER MEMBERS OF THE FORUM WHO CONCUR TO MAKE SURE THEIR FEELINGS ARE
KNOWN, SO WE CAN HOPEFULLY SEE IMPROVEMENT IN WHQL.

Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting

Jake Oshins wrote:

It will change, but slowly. Today, only high-end machines support HT, and
only those with Intel processors. There are still a whole raft of
chipsets
that either don’t support mobile-HT (which is to say that they either
don’t
support APIC mode, or they don’t support APIC mode with power management
enabled) or they don’t support Intel processors. Think for a moment about
the chipsets that get matched with Transmeta processors. For now, at
least,
those still can only be run uni-proc.

The problem is really a large one. I’m writing this message with my very
favorite computer, a Thinkpad X31. And all the drivers for it, including
the TrackPoint and couple of others, have only very recently ever been
tried
on a machine with more than one logical processor.

2

> 1. There should be tests for MP, PAE, 64-bit and hot-plug with the

concept that right now failing them produces a warning (perhaps not public
even) that the driver is remiss, and should be fixed because the times are
changing.

Agree.

  1. WHQL tests should be runnable standalone and report clearly the
    actual problem.

Agree.

  1. WHQL test sources should be made available.

Agree.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

3

> ----------

From: xxxxx@acm.org[SMTP:xxxxx@acm.org]
Reply To: xxxxx@lists.osr.com
Sent: Saturday, January 24, 2004 3:49 PM
To: xxxxx@lists.osr.com
Subject: [ntdev] What is wrong with WHQL (was WHQL Certification)

  1. There should be tests for MP, PAE, 64-bit and hot-plug with the
    concept that right now failing them produces a warning (perhaps not public
    even) that the driver is remiss, and should be fixed because the times are
    changing.

Basically yes.

Another idea: what if driver signature contains info about certification
type? There could be drivers signed for UP only and when installed on SMP
machine, OS would display a warning as for non-signed drivers now.

Yet another one: WHQL test made on UP machine could force UP-only flag in PE
header. It would solve mobile-machines issue and drivers for general use
would be tested on SMP machines with no such requirement.

On the other hand, it isn’t easy to find SMP machine which fulfils all
requirements for tests; especially ACPI stress tests requirements (both S1
and S3 support).

  1. WHQL tests should be runnable standalone and report clearly the
    actual problem.

Isn’t the first part already possible? I mean, command line for tests is
visible and maybe it could be used standalone (never tried it). With HCT
installed, of course.

And yes, clear reports would really help. The best would be layered approach
– every component involved since the lowest failed layer adds its error
code and description. The result is usually easily understandable problem
desciption. Real life example taken from RADIUS server logs:

23:11:25 : EX: [0xe74a0001] Rejecting user “”. * [0xe74a0001]
Error checking Token ID “JC0000-00B7”. * [0xe7460004] One time password
verification failed. * [0xe7460005] Incorrect one time password.

C++ exceptions are great for this kind of reporting in user mode land; above
example is basically exception catched and dumped by the top layer.

> 3. WHQL test sources should be made available.
>
I agree.

Best regards,

Michal Vodicka
STMicroelectronics Design and Application s.r.o.
[michal.vodicka@st.com, http:://www.st.com]

4

> On the other hand, it isn’t easy to find SMP machine which fulfils all

requirements for tests; especially ACPI stress tests requirements (both S1
and S3 support).

Maybe WHQL would include the “ACPI cheat” driver which will implement S1 on any
machine?

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

5

“Michal Vodicka” wrote in message
news:xxxxx@ntdev…
>
> On the other hand, it isn’t easy to find SMP machine which fulfils all
> requirements for tests; especially ACPI stress tests requirements (both S1
> and S3 support).
>

Dell 530 MT supports all power states and is a MP machine (set BIOS to
suspend state=S3). It can go to 4 procs with hyper threading. I am not sure
it supports 6GB.

I am still trying to get someone to tell me a 64 bit machine.

6

> “Michal Vodicka” wrote in message
> news:xxxxx@ntdev…
> >
> > On the other hand, it isn’t easy to find SMP machine which
> fulfils all
> > requirements for tests; especially ACPI stress tests
> requirements (both S1
> > and S3 support).
> >
>
> Dell 530 MT supports all power states and is a MP machine (set BIOS to
> suspend state=S3). It can go to 4 procs with hyper threading.
> I am not sure
> it supports 6GB.
>
> I am still trying to get someone to tell me a 64 bit machine.

Not sure about a dual 64-bit machine with S3 state, but an Opteron base
motherboard should be able to do it. If S3 isn’t a strict requirement,
Tyan’s boards does S1 and C3 according to their specs.

Polywell does a machine with AGP 8X and dual opterons:
http://www.polywell.com/us/workstations/polystation2020a.asp
So does NTSI: http://www.ntsi.com/opteronws.html

I’m not that impressed with the motherboard that the above uses, as it’s
only got ONE of the Opteron’s operate the memory, so the second one has to
request all of it’s memory over the Hypertransport bus. This is bad both
because it means that the second processor gets slower access to memory, and
because it congests the memory bandwidth on the first processor. But it’s
capable of 8GB of RAM if you have 2GB sticks.

Here’s the Tyan Opteron web-page:
http://www.tyan.com/products/html/opteron.html

I’ll try to find some better references from one of my ex-collegues…

Of course, if you want an Itanium machine, then I’m unable to supply ANY
recommendations (other than “Avoid it” ;-).

–
Mats

7

(1) Yes I agree on all counts, test, test, test, please, please, please.

Thanks
Mike Jones

“Maxim S. Shatskih” wrote in message
news:xxxxx@ntdev…
> > 1. There should be tests for MP, PAE, 64-bit and hot-plug with
the
> > concept that right now failing them produces a warning (perhaps not
public
> > even) that the driver is remiss, and should be fixed because the times
are
> > changing.
>
> Agree.
>
> > 2. WHQL tests should be runnable standalone and report clearly
the
> > actual problem.
>
> Agree.
>
> > 3. WHQL test sources should be made available.
>
> Agree.
>
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>