QA criteria

NTDEV
1

Hello everyone,

I have a general question about QA of drivers.

We have a driver which will plan to support on W2K Prof., W2K Server, Win XP
Prof. and Win 2003 Server. We have, of course, purchased an MSDN
subscription so that we have all these OSes at our disposal for testing.

Now my question is related to service packs. For W2K, 4 different SPs have
been released. The most conservative approach would be to test all 5
combinations: W2K w/o SP, W2K w/ SP1, …, up to W2K w/ SP4. We would have
to repeat this twice, once for Prof. and once for Server. Incidentally, our
driver is pretty generic and is not dependent on any features specific to a
particular version of the OS - as far as I know.

I’m hoping we don’t have to do all of the above. It appears that MS only
included the latest SP for each OS in the CDs that we got. This suggests
that MS doesn’t think we need to test with every SP. I’m thinking that if
we test the two “extremes”, i.e., OS with no SP and the same OS with the
latest SP, that we have covered things pretty well.

I’m looking for opinions on whether this is reasonable. Also, I’d like to
hear what practices others are following.

Thanks,

Monish Shah
CEO, Indra Networks, Inc.

2

I would focus tests on what is mostly currently used: SP3 and SP4. I
would of course run a few tests also on SP0-SP2 but just for
completeness as the chances that customers are still running Windows
2000 with SP2 or less are pretty thin.

If you cannot afford to test on all service packs I would also start
from the top (sp4) and move backward and state clearly in the doc/setup
what are the min systems requirements.

Automating as much as possible your tests will save you time too. We use
ghost for the OS images and lots of scripts to exercise the driver so
testing on different configs becomes fairly trivial.

BTW, if you’re not yes using driver driver I recommend that you turn it
on during the tests as it will help catch a lots of bugs: the more you
find before you ship the less support calls you’re going to get.

Cheers,

Marco

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Monish Shah
Sent: Tuesday, June 07, 2005 10:22 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] QA criteria

Hello everyone,

I have a general question about QA of drivers.

We have a driver which will plan to support on W2K Prof., W2K Server,
Win XP Prof. and Win 2003 Server. We have, of course, purchased an MSDN
subscription so that we have all these OSes at our disposal for testing.

Now my question is related to service packs. For W2K, 4 different SPs
have been released. The most conservative approach would be to test all
5
combinations: W2K w/o SP, W2K w/ SP1, …, up to W2K w/ SP4. We would
have to repeat this twice, once for Prof. and once for Server.
Incidentally, our driver is pretty generic and is not dependent on any
features specific to a particular version of the OS - as far as I know.

I’m hoping we don’t have to do all of the above. It appears that MS
only included the latest SP for each OS in the CDs that we got. This
suggests that MS doesn’t think we need to test with every SP. I’m
thinking that if we test the two “extremes”, i.e., OS with no SP and the
same OS with the latest SP, that we have covered things pretty well.

I’m looking for opinions on whether this is reasonable. Also, I’d like
to hear what practices others are following.

Thanks,

Monish Shah
CEO, Indra Networks, Inc.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@neovalens.com To
unsubscribe send a blank email to xxxxx@lists.osr.com

3

Hi,
We always run our automatic tests on every OS we support with every SP that
is available (W2K,W2K+SP1,…,W2K+SP4,WXP,WXP+SP1,WXP+SP2,W2K3,W2K3+SP1).
On top of that for 2K we also test the server addition (we would also test
XP’s home edition, if we wanted to support it).

Every OS/SP combination is tested with on NTFS, FAT and FAT32, even though
our product is not related to filesystems. We thought at first that this was
not necessary, but after discovering a bugcheck that occurred on WIN2K+NTFS
once in a while because we put two backslashes in the WPP log file path
instead of one and didn’t occur on any other combination, we were convinced
that it was worth it.

Even if you don’t believe that your driver is OS dependent, you should still
test on all the OS/SP and even FS combinations you support. This is atleast
what I think is best.

Shahar

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Monish Shah
Sent: Tuesday, June 07, 2005 10:22 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] QA criteria

Hello everyone,

I have a general question about QA of drivers.

We have a driver which will plan to support on W2K Prof., W2K Server, Win XP
Prof. and Win 2003 Server. We have, of course, purchased an MSDN
subscription so that we have all these OSes at our disposal for testing.

Now my question is related to service packs. For W2K, 4 different SPs have
been released. The most conservative approach would be to test all 5
combinations: W2K w/o SP, W2K w/ SP1, …, up to W2K w/ SP4. We would have
to repeat this twice, once for Prof. and once for Server. Incidentally, our
driver is pretty generic and is not dependent on any features specific to a
particular version of the OS - as far as I know.

I’m hoping we don’t have to do all of the above. It appears that MS only
included the latest SP for each OS in the CDs that we got. This suggests
that MS doesn’t think we need to test with every SP. I’m thinking that if
we test the two “extremes”, i.e., OS with no SP and the same OS with the
latest SP, that we have covered things pretty well.

I’m looking for opinions on whether this is reasonable. Also, I’d like to
hear what practices others are following.

Thanks,

Monish Shah
CEO, Indra Networks, Inc.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@safend.com To unsubscribe
send a blank email to xxxxx@lists.osr.com

4

Hello Marco,

Thanks for your feedback.
…

On the following point:

BTW, if you’re not yes using driver driver I recommend that you turn it
on during the tests as it will help catch a lots of bugs: the more you
find before you ship the less support calls you’re going to get.

I think you mean driver verifier, right? If yes, we are using it and it did
help catch bugs. In many cases, crashes that were difficult to reproduce
without it became much easier to reproduce and debug with it. I’m quite
impressed with it.

Cheers,

Marco

Thanks,

Monish

5

While I agree that you should test as much as possible, I wonder what the
point is in testing all intermediate service pack releases. The problem is
that customer systems are more likely to be “OS major release” + “some
service pack” + “some set of hotfix updates”. How are you going to test
that? Base and base+latest, where ‘base+latest’ has an interim phase as a
new service pack emerges from Microsoft that has two service packs
associated with it, would seem to be sufficient.

=====================
Mark Roddy DDK MVP
Windows 2003/XP/2000 Consulting
Hollis Technology Solutions 603-321-1032
www.hollistech.com

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Shahar Talmi
Sent: Tuesday, June 07, 2005 5:38 AM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] QA criteria

Hi,
We always run our automatic tests on every OS we support with
every SP that is available
(W2K,W2K+SP1,…,W2K+SP4,WXP,WXP+SP1,WXP+SP2,W2K3,W2K3+SP1).
On top of that for 2K we also test the server addition (we
would also test XP’s home edition, if we wanted to support it).

Every OS/SP combination is tested with on NTFS, FAT and
FAT32, even though our product is not related to filesystems.
We thought at first that this was not necessary, but after
discovering a bugcheck that occurred on WIN2K+NTFS once in a
while because we put two backslashes in the WPP log file path
instead of one and didn’t occur on any other combination, we
were convinced that it was worth it.

Even if you don’t believe that your driver is OS dependent,
you should still test on all the OS/SP and even FS
combinations you support. This is atleast what I think is best.

Shahar

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Monish Shah
Sent: Tuesday, June 07, 2005 10:22 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] QA criteria

Hello everyone,

I have a general question about QA of drivers.

We have a driver which will plan to support on W2K Prof., W2K
Server, Win XP Prof. and Win 2003 Server. We have, of
course, purchased an MSDN subscription so that we have all
these OSes at our disposal for testing.

Now my question is related to service packs. For W2K, 4
different SPs have been released. The most conservative
approach would be to test all 5
combinations: W2K w/o SP, W2K w/ SP1, …, up to W2K w/ SP4.
We would have to repeat this twice, once for Prof. and once
for Server. Incidentally, our driver is pretty generic and
is not dependent on any features specific to a particular
version of the OS - as far as I know.

I’m hoping we don’t have to do all of the above. It appears
that MS only included the latest SP for each OS in the CDs
that we got. This suggests that MS doesn’t think we need to
test with every SP. I’m thinking that if we test the two
“extremes”, i.e., OS with no SP and the same OS with the
latest SP, that we have covered things pretty well.

I’m looking for opinions on whether this is reasonable.
Also, I’d like to hear what practices others are following.

Thanks,

Monish Shah
CEO, Indra Networks, Inc.

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as: xxxxx@safend.com
To unsubscribe send a blank email to xxxxx@lists.osr.com

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as:
xxxxx@hollistech.com To unsubscribe send a blank email to
xxxxx@lists.osr.com

6

Hello Shahar,

Hi,
We always run our automatic tests on every OS we support with every SP
that
is available (W2K,W2K+SP1,…,W2K+SP4,WXP,WXP+SP1,WXP+SP2,W2K3,W2K3+SP1).
On top of that for 2K we also test the server addition (we would also test
XP’s home edition, if we wanted to support it).

Thanks for your feedback. Have you run into bugs that were only
reproducible on a particular SP and not others?

Every OS/SP combination is tested with on NTFS, FAT and FAT32, even though
our product is not related to filesystems. We thought at first that this
was
not necessary, but after discovering a bugcheck that occurred on
WIN2K+NTFS
once in a while because we put two backslashes in the WPP log file path
instead of one and didn’t occur on any other combination, we were
convinced
that it was worth it.

Geez, you just multiplied our test combination matrix by 3! :frowning:

Anyway, thanks for the feedback. I don’t want to ship bugs, so we will try
all file systems.

Even if you don’t believe that your driver is OS dependent, you should
still
test on all the OS/SP and even FS combinations you support. This is
atleast
what I think is best.

Shahar

Thanks,

Monish

7

If you can’t do everything, you should at least do the RTM version and the
last couple of service packs. As Mark Roddy pointed out it is truly
impossilbe to test everything because of hotfixes and the possible
combinations people use. Using the RTM version will catch things that may
have been impacted by service pack bug fixes.

One thing not mentioned is you should test against the checked build of the
OS as wells as the release build. You may want to reduce the checked build
testing to the latest service packs only, but you want to run against it to
catch those bugs it finds that driver verifier does not. For Windows 2003
you should run a version of your driver with Call Usage Verifier enabled, to
be sure these checks are exercised.

Finally, run your driver under both performance profiling and code coverage
and analyze the results. The performance profiling may show hotspots in
your driver you need to fix. The code coverage will point out how little of
your driver is exercised by the tests.

–
Don Burn (MVP, Windows DDK)
Windows 2k/XP/2k3 Filesystem and Driver Consulting
Remove StopSpam from the email to reply

“Monish Shah” wrote in message
news:xxxxx@ntdev…
> Hello everyone,
>
> I have a general question about QA of drivers.
>
> We have a driver which will plan to support on W2K Prof., W2K Server, Win
> XP Prof. and Win 2003 Server. We have, of course, purchased an MSDN
> subscription so that we have all these OSes at our disposal for testing.
>
> Now my question is related to service packs. For W2K, 4 different SPs
> have been released. The most conservative approach would be to test all 5
> combinations: W2K w/o SP, W2K w/ SP1, …, up to W2K w/ SP4. We would
> have to repeat this twice, once for Prof. and once for Server.
> Incidentally, our driver is pretty generic and is not dependent on any
> features specific to a particular version of the OS - as far as I know.
>
> I’m hoping we don’t have to do all of the above. It appears that MS only
> included the latest SP for each OS in the CDs that we got. This suggests
> that MS doesn’t think we need to test with every SP. I’m thinking that if
> we test the two “extremes”, i.e., OS with no SP and the same OS with the
> latest SP, that we have covered things pretty well.
>
> I’m looking for opinions on whether this is reasonable. Also, I’d like to
> hear what practices others are following.
>
> Thanks,
>
> Monish Shah
> CEO, Indra Networks, Inc.
>

8

Usually, testing with the base OS + the latest SP is enough, provided you
did no “unkosher” things like hooking etc.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Monish Shah”
To: “Windows System Software Devs Interest List”
Sent: Tuesday, June 07, 2005 12:21 PM
Subject: [ntdev] QA criteria

> Hello everyone,
>
> I have a general question about QA of drivers.
>
> We have a driver which will plan to support on W2K Prof., W2K Server, Win XP
> Prof. and Win 2003 Server. We have, of course, purchased an MSDN
> subscription so that we have all these OSes at our disposal for testing.
>
> Now my question is related to service packs. For W2K, 4 different SPs have
> been released. The most conservative approach would be to test all 5
> combinations: W2K w/o SP, W2K w/ SP1, …, up to W2K w/ SP4. We would have
> to repeat this twice, once for Prof. and once for Server. Incidentally, our
> driver is pretty generic and is not dependent on any features specific to a
> particular version of the OS - as far as I know.
>
> I’m hoping we don’t have to do all of the above. It appears that MS only
> included the latest SP for each OS in the CDs that we got. This suggests
> that MS doesn’t think we need to test with every SP. I’m thinking that if
> we test the two “extremes”, i.e., OS with no SP and the same OS with the
> latest SP, that we have covered things pretty well.
>
> I’m looking for opinions on whether this is reasonable. Also, I’d like to
> hear what practices others are following.
>
> Thanks,
>
> Monish Shah
> CEO, Indra Networks, Inc.
>
>
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com

9

From the real life: the amount of issues specific to the SP is very, very
low. The amount of issues specific to FS kind (if the product is not
filesystem-related) is also very, very low. The amount of issues specific to
Home/Pro or to the flavours of the server Windows is practically zero (don’t
remember any that was not related to some smart service running in Advanced
Server only - and this just means “test on Advanced Server”).

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

----- Original Message -----
From: “Mark Roddy”
To: “Windows System Software Devs Interest List”
Sent: Tuesday, June 07, 2005 3:22 PM
Subject: RE: [ntdev] QA criteria

> While I agree that you should test as much as possible, I wonder what the
> point is in testing all intermediate service pack releases. The problem is
> that customer systems are more likely to be “OS major release” + “some
> service pack” + “some set of hotfix updates”. How are you going to test
> that? Base and base+latest, where ‘base+latest’ has an interim phase as a
> new service pack emerges from Microsoft that has two service packs
> associated with it, would seem to be sufficient.
>
>
> =====================
> Mark Roddy DDK MVP
> Windows 2003/XP/2000 Consulting
> Hollis Technology Solutions 603-321-1032
> www.hollistech.com
>
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of Shahar Talmi
> > Sent: Tuesday, June 07, 2005 5:38 AM
> > To: Windows System Software Devs Interest List
> > Subject: RE: [ntdev] QA criteria
> >
> > Hi,
> > We always run our automatic tests on every OS we support with
> > every SP that is available
> > (W2K,W2K+SP1,…,W2K+SP4,WXP,WXP+SP1,WXP+SP2,W2K3,W2K3+SP1).
> > On top of that for 2K we also test the server addition (we
> > would also test XP’s home edition, if we wanted to support it).
> >
> > Every OS/SP combination is tested with on NTFS, FAT and
> > FAT32, even though our product is not related to filesystems.
> > We thought at first that this was not necessary, but after
> > discovering a bugcheck that occurred on WIN2K+NTFS once in a
> > while because we put two backslashes in the WPP log file path
> > instead of one and didn’t occur on any other combination, we
> > were convinced that it was worth it.
> >
> > Even if you don’t believe that your driver is OS dependent,
> > you should still test on all the OS/SP and even FS
> > combinations you support. This is atleast what I think is best.
> >
> > Shahar
> >
> > -----Original Message-----
> > From: xxxxx@lists.osr.com
> > [mailto:xxxxx@lists.osr.com] On Behalf Of Monish Shah
> > Sent: Tuesday, June 07, 2005 10:22 AM
> > To: Windows System Software Devs Interest List
> > Subject: [ntdev] QA criteria
> >
> > Hello everyone,
> >
> > I have a general question about QA of drivers.
> >
> > We have a driver which will plan to support on W2K Prof., W2K
> > Server, Win XP Prof. and Win 2003 Server. We have, of
> > course, purchased an MSDN subscription so that we have all
> > these OSes at our disposal for testing.
> >
> > Now my question is related to service packs. For W2K, 4
> > different SPs have been released. The most conservative
> > approach would be to test all 5
> > combinations: W2K w/o SP, W2K w/ SP1, …, up to W2K w/ SP4.
> > We would have to repeat this twice, once for Prof. and once
> > for Server. Incidentally, our driver is pretty generic and
> > is not dependent on any features specific to a particular
> > version of the OS - as far as I know.
> >
> > I’m hoping we don’t have to do all of the above. It appears
> > that MS only included the latest SP for each OS in the CDs
> > that we got. This suggests that MS doesn’t think we need to
> > test with every SP. I’m thinking that if we test the two
> > “extremes”, i.e., OS with no SP and the same OS with the
> > latest SP, that we have covered things pretty well.
> >
> > I’m looking for opinions on whether this is reasonable.
> > Also, I’d like to hear what practices others are following.
> >
> > Thanks,
> >
> > Monish Shah
> > CEO, Indra Networks, Inc.
> >
> >
> > —
> > Questions? First check the Kernel Driver FAQ at
> > http://www.osronline.com/article.cfm?id=256
> >
> > You are currently subscribed to ntdev as: xxxxx@safend.com
> > To unsubscribe send a blank email to xxxxx@lists.osr.com
> >
> >
> > —
> > Questions? First check the Kernel Driver FAQ at
> > http://www.osronline.com/article.cfm?id=256
> >
> > You are currently subscribed to ntdev as:
> > xxxxx@hollistech.com To unsubscribe send a blank email to
> > xxxxx@lists.osr.com
> >
>
>
>
> —
> Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@storagecraft.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com

10

> ----------

From: xxxxx@lists.osr.com[SMTP:xxxxx@lists.osr.com] on behalf of Maxim S. Shatskih[SMTP:xxxxx@storagecraft.com]
Reply To: Windows System Software Devs Interest List
Sent: Tuesday, June 07, 2005 6:10 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] QA criteria

From the real life: the amount of issues specific to the SP is very, very
low.

It depends. For USB at XP there is real difference between RTM and SP1, next differences with following two important hotfixes and SP2 brings new changes. I had issues which were reproducible with one hotfix only. There were race conditions and different hotfixes had different timing.

Best regards,

Michal Vodicka
UPEK, Inc.
[xxxxx@upek.com, http://www.upek.com]

11

In particular, Mark’s advice is very true if it means that you can
test the latest stuff more carefully because you don’t burn time
testing other configurations. For my money, the deeper you can test,
the better - do as Mark suggests and spend as much time as possible
working out all of the corner cases.

-sd

On Jun 7, 2005, at 6:22 AM, Mark Roddy wrote:

While I agree that you should test as much as possible, I wonder
what the
point is in testing all intermediate service pack releases. The
problem is
that customer systems are more likely to be “OS major release” + “some
service pack” + “some set of hotfix updates”. How are you going to
test
that? Base and base+latest, where ‘base+latest’ has an interim
phase as a
new service pack emerges from Microsoft that has two service packs
associated with it, would seem to be sufficient.

=====================
Mark Roddy DDK MVP
Windows 2003/XP/2000 Consulting
Hollis Technology Solutions 603-321-1032
www.hollistech.com

> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Shahar Talmi
> Sent: Tuesday, June 07, 2005 5:38 AM
> To: Windows System Software Devs Interest List
> Subject: RE: [ntdev] QA criteria
>
> Hi,
> We always run our automatic tests on every OS we support with
> every SP that is available
> (W2K,W2K+SP1,…,W2K+SP4,WXP,WXP+SP1,WXP+SP2,W2K3,W2K3+SP1).
> On top of that for 2K we also test the server addition (we
> would also test XP’s home edition, if we wanted to support it).
>
> Every OS/SP combination is tested with on NTFS, FAT and
> FAT32, even though our product is not related to filesystems.
> We thought at first that this was not necessary, but after
> discovering a bugcheck that occurred on WIN2K+NTFS once in a
> while because we put two backslashes in the WPP log file path
> instead of one and didn’t occur on any other combination, we
> were convinced that it was worth it.
>
> Even if you don’t believe that your driver is OS dependent,
> you should still test on all the OS/SP and even FS
> combinations you support. This is atleast what I think is best.
>
> Shahar
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Monish Shah
> Sent: Tuesday, June 07, 2005 10:22 AM
> To: Windows System Software Devs Interest List
> Subject: [ntdev] QA criteria
>
> Hello everyone,
>
> I have a general question about QA of drivers.
>
> We have a driver which will plan to support on W2K Prof., W2K
> Server, Win XP Prof. and Win 2003 Server. We have, of
> course, purchased an MSDN subscription so that we have all
> these OSes at our disposal for testing.
>
> Now my question is related to service packs. For W2K, 4
> different SPs have been released. The most conservative
> approach would be to test all 5
> combinations: W2K w/o SP, W2K w/ SP1, …, up to W2K w/ SP4.
> We would have to repeat this twice, once for Prof. and once
> for Server. Incidentally, our driver is pretty generic and
> is not dependent on any features specific to a particular
> version of the OS - as far as I know.
>
> I’m hoping we don’t have to do all of the above. It appears
> that MS only included the latest SP for each OS in the CDs
> that we got. This suggests that MS doesn’t think we need to
> test with every SP. I’m thinking that if we test the two
> “extremes”, i.e., OS with no SP and the same OS with the
> latest SP, that we have covered things pretty well.
>
> I’m looking for opinions on whether this is reasonable.
> Also, I’d like to hear what practices others are following.
>
> Thanks,
>
> Monish Shah
> CEO, Indra Networks, Inc.
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@safend.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as:
> xxxxx@hollistech.com To unsubscribe send a blank email to
> xxxxx@lists.osr.com
>
>

Questions? First check the Kernel Driver FAQ at http://
www.osronline.com/article.cfm?id=256

You are currently subscribed to ntdev as:
xxxxx@positivenetworks.net
To unsubscribe send a blank email to xxxxx@lists.osr.com

12

I agree. It’s a wise step to test with every OS and Service Pack
that more recent than some cutoff date, which is usually
determined by management or by the marketing cats. For example,
someone today might cut it at Win2K SP4, or at WinXP SP1. The
other thing is, I find it wise to make sure that different DDKs
build it too, and that you test on a variety of machine
configurations. So, let’s see, UP,2x,4x,P3,P4 and even P2 and
Pentium (I was bitten recently by the realization that some but
not all Pentiums do not have the 8-byte compare-and-exchange
instruction), W2K through W2K3 SP1, DDK 2260, 2260.1106, 3270
and 3270.1830, NTFS, FAT32, FAT, maybe 32-bit and 64-bit PCI
plus PCI Express: the number of configurations skyrockets very
quickly, and it’s a good idea to let the QA group handle it! I
have seen QA people successfully test drivers and their
user-side GUI apps using Ghost to quickly load multiple
configurations on one machine, or having a library of VMWare
images and downloading them from a server on request.

Now, one thing I don’t think it’s advisable to do, that is to
test on intermediate Windows builds. Sometimes I have done work
on a Beta version of Windows, but that’s a royal pain because
the build’s half-life is very short and it’s usually not
synchronized with the product’s development cycle. I find it a
good idea to wait for the Service Pack RTM, and then test it.
Yet in the past I was bitten by customers who expected me to
support every build that comes out of Microsoft; go figure!

Alberto.

----- Original Message -----
From: “Shahar Talmi”
To: “Windows System Software Devs Interest List”

Sent: Tuesday, June 07, 2005 5:37 AM
Subject: RE: [ntdev] QA criteria

> Hi,
> We always run our automatic tests on every OS we support with
> every SP that
> is available
> (W2K,W2K+SP1,…,W2K+SP4,WXP,WXP+SP1,WXP+SP2,W2K3,W2K3+SP1).
> On top of that for 2K we also test the server addition (we
> would also test
> XP’s home edition, if we wanted to support it).
>
> Every OS/SP combination is tested with on NTFS, FAT and FAT32,
> even though
> our product is not related to filesystems. We thought at first
> that this was
> not necessary, but after discovering a bugcheck that occurred
> on WIN2K+NTFS
> once in a while because we put two backslashes in the WPP log
> file path
> instead of one and didn’t occur on any other combination, we
> were convinced
> that it was worth it.
>
> Even if you don’t believe that your driver is OS dependent,
> you should still
> test on all the OS/SP and even FS combinations you support.
> This is atleast
> what I think is best.
>
> Shahar
>
> -----Original Message-----
> From: xxxxx@lists.osr.com
> [mailto:xxxxx@lists.osr.com] On Behalf Of Monish
> Shah
> Sent: Tuesday, June 07, 2005 10:22 AM
> To: Windows System Software Devs Interest List
> Subject: [ntdev] QA criteria
>
> Hello everyone,
>
> I have a general question about QA of drivers.
>
> We have a driver which will plan to support on W2K Prof., W2K
> Server, Win XP
> Prof. and Win 2003 Server. We have, of course, purchased an
> MSDN
> subscription so that we have all these OSes at our disposal
> for testing.
>
> Now my question is related to service packs. For W2K, 4
> different SPs have
> been released. The most conservative approach would be to
> test all 5
> combinations: W2K w/o SP, W2K w/ SP1, …, up to W2K w/ SP4.
> We would have
> to repeat this twice, once for Prof. and once for Server.
> Incidentally, our
> driver is pretty generic and is not dependent on any features
> specific to a
> particular version of the OS - as far as I know.
>
> I’m hoping we don’t have to do all of the above. It appears
> that MS only
> included the latest SP for each OS in the CDs that we got.
> This suggests
> that MS doesn’t think we need to test with every SP. I’m
> thinking that if
> we test the two “extremes”, i.e., OS with no SP and the same
> OS with the
> latest SP, that we have covered things pretty well.
>
> I’m looking for opinions on whether this is reasonable. Also,
> I’d like to
> hear what practices others are following.
>
> Thanks,
>
> Monish Shah
> CEO, Indra Networks, Inc.
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@safend.com To
> unsubscribe
> send a blank email to xxxxx@lists.osr.com
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@ieee.org
> To unsubscribe send a blank email to
> xxxxx@lists.osr.com

13

> instruction), W2K through W2K3 SP1, DDK 2260, 2260.1106, 3270

and 3270.1830, NTFS, FAT32, FAT, maybe 32-bit and 64-bit PCI

I would choose 1 and only 1 DDK for the whole project. Well, if we are speaking
about the toolkits (like Compuware produces) - then supporting several DDKs is
a good idea.

Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

14

> I would choose 1 and only 1 DDK for the whole project. Well, if we are
speaking

about the toolkits (like Compuware produces) - then supporting several
DDKs is
a good idea.

We found a while back that you can get into trouble doing this, at least if
you support back to NT4. I don’t recall the specifics of the problem any
more, but the solution was to build with the latest/final DDK release for
each OS version.

But then, we tend to build HALs, which are somewhat strange device drivers.

Loren

15

Also, today it may be feasible to use 3790.1830 for everything
(except maybe NT4, but that’s virtually dead by now, or at least
it should be), but not that long ago one had to use 2260.1106
for 32-bit and 3790 for 64-bit. In the case of a product such as
a Debugger or a BoundsChecker, which is addressed to the kernel
developer community, of course they must be tested against
multiple DDKs. This is even truer with DriverWorks, for example,
because one must make sure that all the sample drivers, as well
as any wizard-generated driver, builds and executes under many
combinations of platform and DDK.

Alberto.

----- Original Message -----
From: “Loren Wilton”
To: “Windows System Software Devs Interest List”

Sent: Wednesday, June 08, 2005 3:22 AM
Subject: Re: [ntdev] QA criteria

>> I would choose 1 and only 1 DDK for the whole project. Well,
>> if we are
> speaking
>> about the toolkits (like Compuware produces) - then
>> supporting several
> DDKs is
>> a good idea.
>
> We found a while back that you can get into trouble doing
> this, at least if
> you support back to NT4. I don’t recall the specifics of the
> problem any
> more, but the solution was to build with the latest/final DDK
> release for
> each OS version.
>
> But then, we tend to build HALs, which are somewhat strange
> device drivers.
>
> Loren
>
>
> —
> Questions? First check the Kernel Driver FAQ at
> http://www.osronline.com/article.cfm?id=256
>
> You are currently subscribed to ntdev as: xxxxx@ieee.org
> To unsubscribe send a blank email to
> xxxxx@lists.osr.com