In the MSDN page, you have:
"Note SHA256 file/page hashes must be used, though certificates may continue to be SHA1"
Error 577 is returned by InstallELAMCertificateInfo when the driver is signed with a file hash algorithm that is not strong enough (sha1).
In the driver signing property page, set the 'File Hash Algorithm' to 'sha256' (at least). Use the same algorithm for the user binary.
it should work with two self-signed certificates for test-signing (2 EKU for the driver, namely ELAM and Code Signing, and only Code Signing EKU for the user certificate).
The certificates are created with the following commands (no admin privilèges needed):
makecert.exe -a SHA256 -r -pe -ss my -n "CN=TestSrv" -eku 184.108.40.206.220.127.116.11.3 TestSrv.cer
makecert.exe -a SHA256 -r -pe -ss my -n "CN=TestElam" -eku 18.104.22.168.4.1.322.214.171.124,126.96.36.199.188.8.131.52.3 TestElam.cer
Certificates are stored in the user personal store (dev machine). They are listed by signtool along with their SHA1 fingerprint when signtool is launched with the /v (verbose) option.
Don’t forget to add both certificates to the target machine’s approved editors and approved root CA stores.
Here is the resource added to the driver's binary:
1, // count of entries, number of maximum entries allowed is 3
L"F84AD274007D589BA3416CC800F3632D9FC1CEF115E81F51C2841098E1E90D5C\0", // certmgr.exe /v path\to\user\binary.exe
0x800C, // the user binary is signed with the sha256 file hash algorith (like the driver binary).
L"\0", //No EKU other then code signing EKU is present in the certificate used to sign the user binary.
The user service is a basic service written with a C# VS template documented there:https://docs.microsoft.com/en-us/dotnet/framework/windows-services/walkthrough-creating-a-windows-service-application-in-the-component-designer
The user binary is signed with the following command (Post-build):
"C:\Program Files (x86)\Windows Kits\10\bin\x64\signtool.exe" sign /fd SHA256 /a /v /ph /sha1 6EAE19B1271E74E937D21DB886308F9F09816114 "$(TargetDir)$(TargetFileName)"
The user certificate is identified with its SHA1 fingerprint but the binary is signed with the sha256 file hash algorithm (/fd sha256).
It is working perfectly:
SessionId: 0 Cid: 095c Peb: e4e6f65000 ParentCid: 0240
DirBase: 3d23e000 ObjectTable: ffff8b80d96df540 HandleCount: 183.
kd> dt nt!_EPROCESS Protection ffffb58de616f600
+0x6ca Protection : _PS_PROTECTION
kd> dt nt!_PS_PROTECTION ffffb58de616f600+0x6ca
+0x000 Level : 0x31 '1'
+0x000 Type : 0y001
+0x000 Audit : 0y0
+0x000 Signer : 0y0011
Type is 1, indicating that the service is a protected process.