WLK or WDTF or DTM for legacy drivers?

NTDEV
1

Does anyone have thoughts about the ease
and effectiveness of using the WLK (Windows
Logo Kit), WDTF (Windows Device Test
Framework), DTM (Driver Test Manager)
for testing of legacy logical/software drivers?

I’d like to be able to test such drivers on real
hardware and not just VMs. So I’m looking
for a large-scale test framework in which I can
provision hardware systems with an OS and
potentially various applications, run tests on
my software drivers, and trap any bugchecks.
Are there any good commercial testing
frameworks?

Thanks,

Mark McCormick

2

When DTM first came out I looked at it from this perspective and decided
that it really was too much of a struggle to just get it up and running the
logo tests, let alone figure out how to extend it, to bother with. The logo
tests are generally not sufficient for what you are trying to do.

Mark Roddy

On Fri, Jul 9, 2010 at 8:38 AM, Mark McCormick wrote:

>
> Does anyone have thoughts about the ease
> and effectiveness of using the WLK (Windows
> Logo Kit), WDTF (Windows Device Test
> Framework), DTM (Driver Test Manager)
> for testing of legacy logical/software drivers?
>
> I’d like to be able to test such drivers on real
> hardware and not just VMs. So I’m looking
> for a large-scale test framework in which I can
> provision hardware systems with an OS and
> potentially various applications, run tests on
> my software drivers, and trap any bugchecks.
> Are there any good commercial testing
> frameworks?
>
> Thanks,
>
> Mark McCormick
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

3

Mark,

I would avoid these for what you are proposing. The WLK which is
really DTM with packaged tests is a pain in the butt to use and even
Microsoft folks have now admitted it is a poor choice for testing
drivers at least for a driver developer.

WDTF concentrates on testing of PnP and Power which in your case
you do not care about. Setting up WDTF standalone (it is included in
the WLK) is a pain, since there is a file missing and the setup script
is missing steps. See my paper on Windows Driver Development
http://www.microsoft.com/whdc/driver/foundation/drvdev_intro.mspx if you
want instructions on how to set it up. The other problem with WDTF is
that while you are supposed to be able to extend it with your own tests,
the documentation is lacking and the wizard for Visual Studio produces
something that without help from the docs, or a good sample (not
available anywhere) is usesless.

I would not look at the Microsoft frameworks and managers to do
driver testing. The WLK was supposed to address this, but unfortunately
as it dragged on Microsoft forgot its commitments.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

From: Mark McCormick [mailto:xxxxx@yahoo.com]
Posted At: Friday, July 09, 2010 8:38 AM
Posted To: ntdev
Conversation: WLK or WDTF or DTM for legacy drivers?
Subject: WLK or WDTF or DTM for legacy drivers?

Does anyone have thoughts about the ease
and effectiveness of using the WLK (Windows
Logo Kit), WDTF (Windows Device Test
Framework), DTM (Driver Test Manager)
for testing of legacy logical/software drivers?

I’d like to be able to test such drivers on real
hardware and not just VMs. So I’m looking
for a large-scale test framework in which I can
provision hardware systems with an OS and
potentially various applications, run tests on
my software drivers, and trap any bugchecks.
Are there any good commercial testing
frameworks?

Thanks,

Mark McCormick

__________ Information from ESET Smart Security, version of virus
signature database 5265 (20100709) __________

The message was checked by ESET Smart Security.

http://www.eset.com

4

> I would avoid these for what you are proposing. The WLK which
is

really DTM with packaged tests is a pain in the butt to use and even
Microsoft folks have now admitted it is a poor choice for testing
drivers at least for a driver developer.

FWIW, I set up WLK and tested my xen network and scsiport disk drivers
with no trouble at all. I found it easy to install, configure, and use,
and even more importantly, it rattled out quite a few obscure bugs in my
network drivers (mostly to do with TCP offload and some corner case DMA
bugs).

James

5

James,

I stand by my statement that the WLK pain in the butt to use, if you
are not going virtual you need 3 systems, and if you do not use a domain
controller you need a lot of luck since the doc’s assume a domain
controller. For the OP’s query the results he will get will be the
same as running his driver under verifier after developing and running
PreFast and ChkInf on it.

WLK may be ok for a known class of driver (such as net or SCSI) but
Microsoft does not make it easy to extend for general use. Worse yet
when there is a bug, it can be a royal pain to find the bug since they
do not document how to run the underlying tests for most classes of
devices. Finally to emphasize my contention of the WLK being a pain in
the butt, the last HCT could test a SCSI controller in roughly 50
minutes on a test machine, running a very similar test under the WLK
took 75 hours!!!

Between moving to the WLK and the fact that most of the information
on what is required for a windows compliant device has moved from the
public domain to behind the WinQual login, WHQL has done more to lower
the quality and reliability of Windows drivers and devices than any
other factor in the last 5 years.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: James Harper [mailto:xxxxx@bendigoit.com.au]
Posted At: Friday, July 09, 2010 9:11 AM
Posted To: ntdev
Conversation: WLK or WDTF or DTM for legacy drivers?
Subject: RE: WLK or WDTF or DTM for legacy drivers?

> I would avoid these for what you are proposing. The WLK
which
is
> really DTM with packaged tests is a pain in the butt to use and even
> Microsoft folks have now admitted it is a poor choice for testing
> drivers at least for a driver developer.

FWIW, I set up WLK and tested my xen network and scsiport disk drivers
with no
trouble at all. I found it easy to install, configure, and use, and
even more
importantly, it rattled out quite a few obscure bugs in my network
drivers
(mostly to do with TCP offload and some corner case DMA bugs).

James

__________ Information from ESET Smart Security, version of virus
signature
database 5265 (20100709) __________

The message was checked by ESET Smart Security.

http://www.eset.com

6

Mark McCormick wrote:

Does anyone have thoughts about the ease
and effectiveness of using the WLK (Windows
Logo Kit), WDTF (Windows Device Test
Framework), DTM (Driver Test Manager)
for testing of legacy logical/software drivers?

Well, I can tell you that DTM is derived from the tool that Microsoft
uses to test Windows itself, where they run millions of tests on tens of
thousands of computers at a time. Now, I do not know whether DTM
actually contains all of the hooks you need to add your own tests to the
mix, but given its price, I suspect it’s worth your time to investigate
it. There are commercial bulk testing packages, but they tend to be spendy.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

7

That is true, but they also have a large staff dedicated to testing. Most
developers do not have the resources to do this, and the small scale testing
we tend to do doesn’t justify the setup cost while large scale testing as at
Redmond does.

Mark Roddy

On Fri, Jul 9, 2010 at 12:55 PM, Tim Roberts wrote:

> Mark McCormick wrote:
> >
> > Does anyone have thoughts about the ease
> > and effectiveness of using the WLK (Windows
> > Logo Kit), WDTF (Windows Device Test
> > Framework), DTM (Driver Test Manager)
> > for testing of legacy logical/software drivers?
>
> Well, I can tell you that DTM is derived from the tool that Microsoft
> uses to test Windows itself, where they run millions of tests on tens of
> thousands of computers at a time. Now, I do not know whether DTM
> actually contains all of the hooks you need to add your own tests to the
> mix, but given its price, I suspect it’s worth your time to investigate
> it. There are commercial bulk testing packages, but they tend to be
> spendy.
>
> –
> Tim Roberts, xxxxx@probo.com
> Providenza & Boekelheide, Inc.
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

8

Actually, it is funny. Microsoft went out before WLK and asked large
companies such as their partners how they test. Then they went and
designed a new test harness. Unfortunately Redmond forgot to ask these
partners whether they would use a new test harness from Microsoft or
whether it would interfere with what they were doing in house. I know
of one large firm that had close to 1000 systems that when they did
testing they would push out various versions of Windows, a new driver
and tests including the HCT. Once the new improved WLK came along they
found it would not work with their test harness, so in the end they
removed 100 machines and made that the WLK test bed. Enabling things
so that large firms do 10% of the testing they had in the past, is
another reason why I consider the WLK to be the worst thing for Windows
driver quality that Microsoft has done.

Don Burn (MVP, Windows DKD)

Windows Filesystem and Driver Consulting

Website: http://www.windrvr.com

Blog: http://msmvps.com/blogs/WinDrvr

From: xxxxx@gmail.com [mailto:xxxxx@gmail.com] On Behalf
Of Mark Roddy
Posted At: Friday, July 09, 2010 4:03 PM
Posted To: ntdev
Conversation: WLK or WDTF or DTM for legacy drivers?
Subject: Re: WLK or WDTF or DTM for legacy drivers?

That is true, but they also have a large staff dedicated to testing.
Most developers do not have the resources to do this, and the small
scale testing we tend to do doesn’t justify the setup cost while large
scale testing as at Redmond does.

Mark Roddy

On Fri, Jul 9, 2010 at 12:55 PM, Tim Roberts wrote:

Mark McCormick wrote:
>
> Does anyone have thoughts about the ease
> and effectiveness of using the WLK (Windows
> Logo Kit), WDTF (Windows Device Test
> Framework), DTM (Driver Test Manager)
> for testing of legacy logical/software drivers?

Well, I can tell you that DTM is derived from the tool that Microsoft
uses to test Windows itself, where they run millions of tests on tens of
thousands of computers at a time. Now, I do not know whether DTM
actually contains all of the hooks you need to add your own tests to the
mix, but given its price, I suspect it’s worth your time to investigate
it. There are commercial bulk testing packages, but they tend to be
spendy.

–

Tim Roberts, xxxxx@probo.com

Providenza & Boekelheide, Inc.

—

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Information from ESET Smart Security, version of virus
signature database 5266 (20100709)

The message was checked by ESET Smart Security.

http://www.eset.com

9

Hi Mark

I would not discount WDTF as a possible solution to your testing requirements.
Yes it does provide pre-defined interfaces to create PnP and power related tests, but it is not limited to those.

Can you tell us more about what kind of tests you would like to execute on your drivers, which scenarios are you trying to cover?

If you like to experiment, my suggestion is to give it a try as follow:

  1. Install WDTF (As Don mentioned there is an installer bug which requires you to manually copy and register a dll. Annoying, but not a big deal)
  2. Create a SimpleIO plugin using the provided VS wizard that targets your devices. (Samples are lacking but I can help through this forum).
  3. Create one or more VB scripts that uses the WDTF interfaces and your SimpleIO plugin to implement your testing scenarios.

The power of WDTF is that it allows you to be creative with your driver testing scenarios using a scripted language (vb/java script).
WDTF is used internally by the WLK. It is more of a test development framework than the WLK, thus more scalable and lightweight. If necessary you can execute your custom WDTF scripts using the WLK (not for Logo purposes of course…).

Let me know if you need more info.

Best
Max

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Friday, July 09, 2010 6:04 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] WLK or WDTF or DTM for legacy drivers?

Mark,

I would avoid these for what you are proposing. The WLK which is
really DTM with packaged tests is a pain in the butt to use and even Microsoft folks have now admitted it is a poor choice for testing drivers at least for a driver developer.

WDTF concentrates on testing of PnP and Power which in your case
you do not care about. Setting up WDTF standalone (it is included in
the WLK) is a pain, since there is a file missing and the setup script
is missing steps. See my paper on Windows Driver Development
http://www.microsoft.com/whdc/driver/foundation/drvdev_intro.mspx if you want instructions on how to set it up. The other problem with WDTF is that while you are supposed to be able to extend it with your own tests, the documentation is lacking and the wizard for Visual Studio produces something that without help from the docs, or a good sample (not available anywhere) is usesless.

I would not look at the Microsoft frameworks and managers to do driver testing. The WLK was supposed to address this, but unfortunately
as it dragged on Microsoft forgot its commitments.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

From: Mark McCormick [mailto:xxxxx@yahoo.com] Posted At: Friday, July 09, 2010 8:38 AM Posted To: ntdev
Conversation: WLK or WDTF or DTM for legacy drivers?
Subject: WLK or WDTF or DTM for legacy drivers?

Does anyone have thoughts about the ease
and effectiveness of using the WLK (Windows
Logo Kit), WDTF (Windows Device Test
Framework), DTM (Driver Test Manager)
for testing of legacy logical/software drivers?

I’d like to be able to test such drivers on real
hardware and not just VMs. So I’m looking
for a large-scale test framework in which I can
provision hardware systems with an OS and
potentially various applications, run tests on
my software drivers, and trap any bugchecks.
Are there any good commercial testing
frameworks?

Thanks,

Mark McCormick

__________ Information from ESET Smart Security, version of virus
signature database 5265 (20100709) __________

The message was checked by ESET Smart Security.

http://www.eset.com

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

10

Actually, with the last two WDK’s you have to get the DLL you need to
manually install from either the Vista WDK or from the WLK. The last
two WDK’s have mysteriously dropped the file so there is not a complete
set of files to install WDTF.

Also, as I mentioned in a second post the problem is that the
documentation on developing the plugin is inadequate, and the wizard
does not provide enough data to develop one. I’ve been on and off
trying to create a plugin for the last 3 to 4 months and when I have to
use Russinovich’s Process Explorer to get a clue on the registry values
the plugin needs, this is beyond bad.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: Max Bardetti [mailto:xxxxx@microsoft.com]
Posted At: Friday, July 09, 2010 5:48 PM
Posted To: ntdev
Conversation: RE:WLK or WDTF or DTM for legacy drivers?
Subject: RE: RE:WLK or WDTF or DTM for legacy drivers?

Hi Mark

I would not discount WDTF as a possible solution to your testing
requirements.
Yes it does provide pre-defined interfaces to create PnP and power
related
tests, but it is not limited to those.

Can you tell us more about what kind of tests you would like to
execute on
your drivers, which scenarios are you trying to cover?

If you like to experiment, my suggestion is to give it a try as
follow:

  1. Install WDTF (As Don mentioned there is an installer bug which
    requires you
    to manually copy and register a dll. Annoying, but not a big deal)
  2. Create a SimpleIO plugin using the provided VS wizard that targets
    your
    devices. (Samples are lacking but I can help through this forum).
  3. Create one or more VB scripts that uses the WDTF interfaces and
    your
    SimpleIO plugin to implement your testing scenarios.

The power of WDTF is that it allows you to be creative with your
driver
testing scenarios using a scripted language (vb/java script).
WDTF is used internally by the WLK. It is more of a test development
framework
than the WLK, thus more scalable and lightweight. If necessary you can
execute
your custom WDTF scripts using the WLK (not for Logo purposes of
course…).

Let me know if you need more info.

Best
Max

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:bounce-417267-
xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Friday, July 09, 2010 6:04 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] WLK or WDTF or DTM for legacy drivers?

Mark,

I would avoid these for what you are proposing. The WLK which
is
really DTM with packaged tests is a pain in the butt to use and even
Microsoft
folks have now admitted it is a poor choice for testing drivers at
least for a
driver developer.

WDTF concentrates on testing of PnP and Power which in your case
you do not care about. Setting up WDTF standalone (it is included in
the WLK) is a pain, since there is a file missing and the setup
script
is missing steps. See my paper on Windows Driver Development
http://www.microsoft.com/whdc/driver/foundation/drvdev_intro.mspx if
you want
instructions on how to set it up. The other problem with WDTF is that
while
you are supposed to be able to extend it with your own tests, the
documentation is lacking and the wizard for Visual Studio produces
something
that without help from the docs, or a good sample (not available
anywhere) is
usesless.

I would not look at the Microsoft frameworks and managers to do
driver
testing. The WLK was supposed to address this, but unfortunately
as it dragged on Microsoft forgot its commitments.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

From: Mark McCormick [mailto:xxxxx@yahoo.com] Posted At: Friday,
July 09,
2010 8:38 AM Posted To: ntdev
Conversation: WLK or WDTF or DTM for legacy drivers?
Subject: WLK or WDTF or DTM for legacy drivers?

Does anyone have thoughts about the ease and effectiveness of using
the WLK
(Windows Logo Kit), WDTF (Windows Device Test Framework), DTM (Driver
Test
Manager) for testing of legacy logical/software drivers?

I’d like to be able to test such drivers on real hardware and not just
VMs. So
I’m looking for a large-scale test framework in which I can provision
hardware
systems with an OS and potentially various applications, run tests on
my
software drivers, and trap any bugchecks.
Are there any good commercial testing
frameworks?

Thanks,

Mark McCormick

__________ Information from ESET Smart Security, version of virus
signature
database 5265 (20100709) __________

The message was checked by ESET Smart Security.

http://www.eset.com

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

__________ Information from ESET Smart Security, version of virus
signature
database 5266 (20100709) __________

The message was checked by ESET Smart Security.

http://www.eset.com

11

Don

We are looking at ways to make it easy to get the required dll.

Would you like to go through the specific issues you are having with your plugin? The visual studio plug-in wizard creates a couple of template registry files (.rgs) that should not require any major edits (the critical registry key is the one that allows WDTF to match the plugin to the device under test).

Best
Max

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Friday, July 09, 2010 3:10 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] RE:WLK or WDTF or DTM for legacy drivers?

Actually, with the last two WDK’s you have to get the DLL you need to manually install from either the Vista WDK or from the WLK. The last two WDK’s have mysteriously dropped the file so there is not a complete set of files to install WDTF.

Also, as I mentioned in a second post the problem is that the documentation on developing the plugin is inadequate, and the wizard does not provide enough data to develop one. I’ve been on and off trying to create a plugin for the last 3 to 4 months and when I have to use Russinovich’s Process Explorer to get a clue on the registry values the plugin needs, this is beyond bad.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: Max Bardetti [mailto:xxxxx@microsoft.com] Posted At: Friday,
July 09, 2010 5:48 PM Posted To: ntdev
Conversation: RE:WLK or WDTF or DTM for legacy drivers?
Subject: RE: RE:WLK or WDTF or DTM for legacy drivers?

Hi Mark

I would not discount WDTF as a possible solution to your testing
requirements.
Yes it does provide pre-defined interfaces to create PnP and power
related
tests, but it is not limited to those.

Can you tell us more about what kind of tests you would like to
execute on
your drivers, which scenarios are you trying to cover?

If you like to experiment, my suggestion is to give it a try as
follow:

  1. Install WDTF (As Don mentioned there is an installer bug which
    requires you
    to manually copy and register a dll. Annoying, but not a big deal)
  2. Create a SimpleIO plugin using the provided VS wizard that targets
    your
    devices. (Samples are lacking but I can help through this forum).
  3. Create one or more VB scripts that uses the WDTF interfaces and
    your
    SimpleIO plugin to implement your testing scenarios.

The power of WDTF is that it allows you to be creative with your
driver
testing scenarios using a scripted language (vb/java script).
WDTF is used internally by the WLK. It is more of a test development
framework
than the WLK, thus more scalable and lightweight. If necessary you can
execute
your custom WDTF scripts using the WLK (not for Logo purposes of
course…).

Let me know if you need more info.

Best
Max

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:bounce-417267-
xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Friday, July 09, 2010 6:04 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] WLK or WDTF or DTM for legacy drivers?

Mark,

I would avoid these for what you are proposing. The WLK which
is
really DTM with packaged tests is a pain in the butt to use and even
Microsoft
folks have now admitted it is a poor choice for testing drivers at
least for a
driver developer.

WDTF concentrates on testing of PnP and Power which in your case
you do not care about. Setting up WDTF standalone (it is included in
the WLK) is a pain, since there is a file missing and the setup
script
is missing steps. See my paper on Windows Driver Development
http://www.microsoft.com/whdc/driver/foundation/drvdev_intro.mspx if
you want
instructions on how to set it up. The other problem with WDTF is that
while
you are supposed to be able to extend it with your own tests, the
documentation is lacking and the wizard for Visual Studio produces
something
that without help from the docs, or a good sample (not available
anywhere) is
usesless.

I would not look at the Microsoft frameworks and managers to do
driver
testing. The WLK was supposed to address this, but unfortunately as
it dragged on Microsoft forgot its commitments.

Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

From: Mark McCormick [mailto:xxxxx@yahoo.com] Posted At: Friday,
July 09,
2010 8:38 AM Posted To: ntdev
Conversation: WLK or WDTF or DTM for legacy drivers?
Subject: WLK or WDTF or DTM for legacy drivers?

Does anyone have thoughts about the ease and effectiveness of using
the WLK
(Windows Logo Kit), WDTF (Windows Device Test Framework), DTM (Driver
Test
Manager) for testing of legacy logical/software drivers?

I’d like to be able to test such drivers on real hardware and not just
VMs. So
I’m looking for a large-scale test framework in which I can provision
hardware
systems with an OS and potentially various applications, run tests on
my
software drivers, and trap any bugchecks.
Are there any good commercial testing
frameworks?

Thanks,

Mark McCormick

__________ Information from ESET Smart Security, version of virus
signature
database 5265 (20100709) __________

The message was checked by ESET Smart Security.

http://www.eset.com

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

__________ Information from ESET Smart Security, version of virus
signature
database 5266 (20100709) __________

The message was checked by ESET Smart Security.

http://www.eset.com

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer