HCK/HLK Testing and Submissions

As I write this, HLK is out for Windows 10 build 1803. Supposing I need a driver that runs on platforms that include Windows 8.1 Server 2016, and all Windows 10 flavors, does this mean I need to test with all of HCK, HLK 1607, HLK 1703, HLK 1709, and HLK 1803? Is there any end to this besides our eventual dropping of the oldest operating systems?

Also, suppose I submitted a driver to Microsoft’s portal for signing, but omitted (for example) the HLK 1607 tests. Does this mean the driver would not load on Server 2016? Is there actually anything in the signature that tells Server 2016 the driver isn’t qualified on it?

I get that more testing is better; Microsoft and I are on the same side, here. But I’d like to understand what the practical constraints are, and how others are tackling the accumulation of testing platforms.

And I’m sorry if I horrify anyone by asking, but am I botching my submissions or is it normal for a submission with 5 sets of results to come back as a ZIP file containing 5 folders of the same driver? I’m attaching the driver binary and symbols to each results package, then when I create the package on the latest HLK, I merge all the results together. The signed drivers I get back seem to work, but am I misunderstanding the merge process?

Thanks,
Dave

Did you ever make any progress with this on your own? I’m also struggling to figure out exactly what I need to set up in order to have a properly signed driver.

Even just the process of setting up and locking down updates on a bunch of test targets is pretty daunting.

I still find it daunting as well, though I’ve been through a few successful submissions.

I have a minimal setup that works, though it’s pretty ugly and high maintenance, and if you sneeze anywhere near it the tenuous connection between the controller and target breaks and the client software needs to be reinstalled. Microsoft’s SDV and HCK/HLK tools are absolutely genius in what they can catch, but are also fragile as butterflies.

The machine I use for the HCK/HLK controller dual boots. The HCK controller supposedly doesn’t work correctly in a VM, so one partition boots into Server 2012 to host it. The other partition boots into Windows 10, where I have VMWare images set up to host the HLK 1607, 1703, 1709, and 1803 controllers.

The target machine boots into one of about 8 partitions, including targets for Windows 8.1, Server 2016 Core, and Windows 10 builds 1703, 1709, and 1803. More than once I’ve accidentally left this hooked up to an internet connection and had auto-update helpfully upgrade an OS out of usefulness. It’d be great if all my customers had their machines auto-updated so that those old platforms would simply fade away, but they aren’t typical consumers and their machines are often isolated for the best of reasons.

As to the questions I posed above, they remain unanswered. It’d be wonderful to find out that just the HCK and perhaps the latest HLK were all that were really required, because frankly it’s sheer drudgery to have to alternately fight and nurse all these test platforms, but I won’t hold my breath.

I sometimes get the idea Microsoft expects customers to have a full staff maintaining shining cleanrooms packed with the latest hardware, and tools that work so well that one can launch a build, have automated tests across platforms all complete in parallel, and have the submission automatically submitted and ready by morning. It’s a pretty dream.

Me, I just glare balefully at the two decrepit machines I have to test with and hold my breath each time I try to flip a test client from NotReady to Ready state. Maybe it’s the collected fiction of Lovecraft I’ve been reading, but I swear they’re conspiring against me with multiple cores of malice…

Dave

Thanks Dave. Nice to know I’m not the only one who finds this process incredibly painful.

For Windows 10 they have only one flag that you can check in catalog file
properties so if you skip any Windows 10 version it should be fine I guess,

On Thu, Jul 26, 2018, 8:00 PM xxxxx@pleora.com <
xxxxx@lists.osr.com> wrote:

Thanks Dave. Nice to know I’m not the only one who finds this process
incredibly painful.


NTDEV is sponsored by OSR

Visit the list online at: <
http://www.osronline.com/showlists.cfm?list=ntdev\>

MONTHLY seminars on crash dump analysis, WDF, Windows internals and
software drivers!
Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

Thank you, Rabish, this is a useful bit of information. I may put it to the test on a driver that is hung up on what appears to be an issue with the HLK rather than the driver itself.

Dave

For Windows 10 they have only one flag that you can check in catalog file
properties so if you skip any Windows 10 version it should be fine I guess,

Hey Dave,

So I just did my first successful HLK run from start to finish and I thought I’d share an initial finding.

I ran everything against 1803, got a signed driver back and then tried to install it on a VM that I had rolled back to 1507 and it seemed to work just fine.

I don’t have any other data points at the moment but is it possible that simply running HLK against the latest build will allow one to install on older versions without needing to keep an array of machines locked down to various builds? It seems like a very unnatural thing and Windows REALLY doesn’t like being held back from updates.

I don’t understand why one would need that much testing just to get a driver signature that works on multiple builds of the same OS.

cheers,

Chris

Seemed to install okay on a 1709 as well.

Are we just assuming that we need to run HLK against all these builds or am I not seeing what I think I am?

It always seemed that the most logical signing policy would be that HLK against the latest would let you install against previous builds but perhaps I’m hopelessly naive.

Good to know, thanks Chris. I wouldn’t mind the testing as much if the tools weren’t so capricious. I understand why the experienced developers on this list are wary of upgrades.

Dave

Exactly! The support model seems to be that if anything goes wrong, just burn it to the ground, uninstall and start from scratch.

We have developed an automation system that does a full test cycle for all
HCK and HLK versions: It requires working HCK and HLK servers.

* Connects to the all DUTs (however many) and runs all test (and reruns
upon failure), one by one
* Installs fresh copies of all relevent Windows OSes on all DUTs in order
to cover all required tests
* Controls external AC power switch for cold resets
* Installs relevent drivers and HCK/HLK client
* Packages all results

Using this automation, we manage to pass all HCK and HLK tests automaticly,
for all OSes using a single set of DUTs (we use 4 DUTs currenlty)

On Fri, Jul 27, 2018 at 4:41 PM xxxxx@pleora.com <
xxxxx@lists.osr.com> wrote:

Exactly! The support model seems to be that if anything goes wrong, just
burn it to the ground, uninstall and start from scratch.


NTDEV is sponsored by OSR

Visit the list online at: <
http://www.osronline.com/showlists.cfm?list=ntdev\>

MONTHLY seminars on crash dump analysis, WDF, Windows internals and
software drivers!
Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

Sounds like a really great setup, Menachem! I’d be interested in reading much more about it, either on this list or on a blog somewhere, if your company wouldn’t mind the details being shared. I take it since you say a “single set of DUTs” that the controllers execute against them sequentially? Are the controllers in VMs on a single system, or on separate physical machines?

Dave

We also developed automation. Although the machines that are used during testing are VM on top of KVM\QEMU, guest side scripts and installation instructions can be useful to others:
https://github.com/hck-CI/

Thanks for sharing!

Peter