HLK Questions and Problems

Hello there. I’ve been tasked to get into the HLK Process to test and ultimately sign a driver for virtual network interfaces. I have a few questions though and this seems like a good place to ask without having to wait for 30 days for my account to unlock…

So far I have set up a HLK 1903 and two clients 1909 that could install the client software. I was unable to connect a HLK 2004 to a client with 20H2. Said these versions are not compatible and I don’t know why. All the documentation says it should work but it doesn’t. Oh well.

To celebrate being able to connect the clients with the server, I ran evevery test there was available. A lot of them passed, a lot of them failed. And now I have the simple questions… which tests do I actually need to get the driver signed? I know that i have to create a package taht I can send in to get it signed, but on documentation pages I only found “After the device passes all of the necessary tests…”
Well, what are the necessary tests?

Sorry if this might be a very basic questions, it’s just that there is so much documentation and I can’t seem to find the right search words to get the right answers.

When I look into the Test-Status it tells me “Assessment Failed: 2 Tests” (I test on two machines). Are these the tests that need to pass?

One test is being logged as “Running” and I have not found a way to abort or reset it in any way. Don’t know what to do about that one.

I would greatly appreciate any help you guys can send my way. Even some helpful links would be awesome. Thank you for your time!

You need to choose the device you want to test. When you do that, HLK will show you the set of tests for that device. You have to pass ALL of them.

Hello Tim, thanks for the answer. That helped me, I didn’t see this stated anywhere else. Maybe I’m blind.
So far I have also found a very helpful tutorial that seems to have improved my results.

There is also HLK playlist which you need to download from the MS. It may filter down the list of tests you need to run.

1 Like

@CaptainFlint said:
There is also HLK playlist which you need to download from the MS. It may filter down the list of tests you need to run.

I typically start with setting a check mark on top of column.

  1. It selects all tests which HLK server found applicable to my driver.
  2. Then, I exclude (deselect) those tests which are not compatible with the platform capabilities (say S3 is not supported so HLK won’t try those tests)
  3. Next, I exclude all the tests which I know will never complete - again because of platform limitation (say CHAOS tests)
  4. The remaining tests can be saved as user play list - for particular driver on a particular platform
  5. when a new version of that driver comes up to test, the saved play list is loaded

Hello, I’ve got a few more followup questions:

There are many different combinations of HLK to Windows Version. We are testing because with 21H1, Microsoft will apparently no longer accept cross-signed drivers, only drivers that come directly out of windows update.
Will I have to do the tests with every available Windows HLK/Client Combination?
Do I need only the latest? Or does it even matter? Could I get the signable package and get that signed by only testing on a 1909?

I found an official hlk-testing guide for the driver im testing. Some of the hints for tests say something like “When this test runs, you need to restart the service at the end, else it will fail.”
That sound weird to me. I thought these tests are something you set up correctly and when a new version needs to be signed, you install that version of the thing you want to test, run the tests and are done.
Is it a normal workflow to manually tinker around on the client machine while the tests run?

@rusakov2 in steps 2 and 3 you say you deselect some tests. How would I know if these tests are relevant or not for my test? Is there a list? Are there any critera I just can’t find on what tests are needed and which are not? I loaded the HLK playlist and that did indeed cut the tests in half. But how do I know if some of those tests cant be excluded?

To be quite honest, this has to the be the wonkiest system i have come across. And I’ve come across a lot of wonky systems :slight_smile:

Microsoft will apparently no longer accept cross-signed drivers, only drivers that come directly out of windows update.

That’s not true, Attestation signed drivers are still accepted by Windows 10 and higher. You can run HLK and get the signature, of course, but it’s not necessary for just loading the drivers. The real problem is Windows 8.1 and below, for them WHQL is the only official way. But to test for these OS versions you need to use the older version of the toolkit (HCK), and the clients with the specific Windows version you are testing for.

Will I have to do the tests with every available Windows HLK/Client Combination?
HLK version is fixed for the target OS version, they are listed here, so you don’t have to test every combination, only the versions you need.

As for which OS versions to test for, that’s the tricky part. I have not found any guides about it (maybe others will be able to help). The tests make sure your driver runs smoothly on the specific OS. Generally, it’s expected to run backward-compatible, but there is always a possibility of unexpected quirks that would break it in a specific Windows version. So if you have time and resources, it won’t hurt to test all the applicable ones. Otherwise you could limit yourself to the most significant “series” of the OS versions (like, test for 2012R2 server, and it should be safe to consider Windows 8.1 x64 also tested, since they have the same kernel).

Is it a normal workflow to manually tinker around on the client machine while the tests run?
It is definitely not a normal workflow, but HLK tests do have issues sometimes. Besides, the tests cannot know about all possible real world scenarios, and sometimes they fail just because they expect driver to behave one way, but it behaves in another (but still perfectly valid) way. So there are situations when tinkering with the test progress is inevitable. The alternatives are either changing the driver to accomodate the test (which is not always possible), or letting the test to fail and including errata, explaining why the test cannot be passed.

To be quite honest, this has to the be the wonkiest system i have come across.
It definitely is. And it’s very hard to debug, too. :frowning:

1 Like

@David_Scheele said:
Hello, I’ve got a few more followup questions:

@rusakov2 in steps 2 and 3 you say you deselect some tests. How would I know if these tests are relevant or not for my test? Is there a list? Are there any critera I just can’t find on what tests are needed and which are not? I loaded the HLK playlist and that did indeed cut the tests in half. But how do I know if some of those tests cant be excluded?

I am not aware of any such list. The simple approach was used: knowing, for example, that on a platform under test low power state S3 is supported, but specific device the driver for which is tested cannot reliably come back from low power state. I.e. it will hang the system. It is not a defect in driver. It is limitation of specific hardware. Knowing that such tests need to be excluded from the list for this device.

I highly doubt that explanation will work for passing the HLKs. The entire point of the HLKs is to only logo devices that work predictably and the way Windows expects. Not supporting S3 would mean your device doesn’t comply with the hardware requirements list.

This is the problem with the HLKs. They’re design for the behaviors Windows expects in a desktop or sever… (your device works when the system goes into and comes out of S3)…. They don’t not allow for cases when the system is used in a very different environment, and the system will never enter S3.

Peter

@“Peter_Viscarola_(OSR)” said:
I highly doubt that explanation will work for passing the HLKs. The entire point of the HLKs is to only logo devices that work predictably and the way Windows expects. Not supporting S3 would mean your device doesn’t comply with the hardware requirements list.

The platform mentioned, itself supports S3, but the custom PCIe card installed into it doesn’t support coming out of D3 reliably. So, for the system with this card the HLK tests, for a driver under test (not the driver for that card!) a set of low power state transition related tests are excluded on that basis. Since if HLK would attempt to exercise S0-S3-S0 power state transitions then likely system will hang, and test results will show false negative for that driver which is not at fault.
Yes, system is purpose built and as a whole does not comply with Windows hardware requirements in full. We do what we can do on the given system, and if Microsoft asks for HLK tests we submit results which are passing, and explanation why other tests are not used.

This is the problem with the HLKs. They’re design for the behaviors Windows expects in a desktop or sever…
yep, a limitation. It was also fun, for instance, HLK applicability for an IoT platform running Windows OneCore nicely but didn’t even have a restart feature :slight_smile: at that time.

Since if HLK would attempt to exercise S0-S3-S0 power state transitions then likely system will hang, and test results will show false negative for that driver which is not at fault.

The HLK is not just testing your driver. It’s also testing your hardware. Indeed, that was its original purpose: to allow hardware manufacturers to put the “Designed for Windows” logo on their physical packaging. So, it’s NOT a “false negative”. Your product has failed.

@Tim_Roberts said:

The HLK is not just testing your driver. It’s also testing your hardware. Indeed, that was its original purpose: to allow hardware manufacturers to put the “Designed for Windows” logo on their physical packaging. So, it’s NOT a “false negative”. Your product has failed.

I would NOT agree with you Tim. Nor will our company, which makes successful product derived from system with Windows 10 OS.
For high cost low volume purpose built systems you can view the situation either as “Windows 10 somewhat failed on this hw” as a drawback of Windows 10 OS, or the opposite way, depending which one you like.

In my opinion it is valid to exclude certain low power state tests which HLK wants to exercise, since this purpose built system will never be allowed to enter such power state in real use, i.e. it will be always on from Windows point of view.

If this is a captive system, then why are you wasting time worrying about HLK at all? Just use attestation signing. Microsoft doesn’t really care about your opinion.

1 Like