Asking for hints to get some kernel output

Hi, I a product which uses a minifilter driver to do some tasks but in a few Win 10 machines, I get error 87 (invalid parameter) when I try to start the driver. It is signed both by my EV SHA2 certificate and Microsoft Hardware Portal.

We could figure out kernel is refusing to load the driver but when trying to determine why…

  • I can’t get any debug log from the driver because it is not loading at all.
  • There is no event record stored in the event logger.
  • Tried to get some logs from kernel, mainly from loader snaps using DbgView but cannot enable it.
  • One weird thing is I couldn’t enable the Show Loader Snaps with gflags utility. The kernel & system tabs’ checkboxes get unchecked after pressing the save button so I have to modify the registry directly by adding GlobalFlag=2 in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session and enabling, just in case, legacy debug output by adding the DEFAULT value at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Debug Print Filter but still loader does not output anything!

Any hint on how to get any debug output? May be running LiveKD because I cannot attach a kernel debugger to the computer.

Regards,
Mauro.

Any hint on how to get any debug output?
There is a regiustry setting to enable DbgOutput - I usually bellavista to do that

Are you 100% positive that you’re not getting called at DriverEntry?

Also, is it really not an option to run a debugger on these systems? That would be my ideal solution to the problem…We’ve previously had luck running VMware’s P2V utility to virtualize a failing machine that we couldn’t/weren’t allowed to physically debug.

@rod_widdowson said:

Any hint on how to get any debug output?
There is a regiustry setting to enable DbgOutput - I usually bellavista to do that

I’ll give a try, thanks!

@“Scott_Noone_(OSR)” said:
Are you 100% positive that you’re not getting called at DriverEntry?

I would say yes. The first thing I do is to read a registry value and nothing appears on procmon (modifying the altitude)

Also, based on procmon log, I can say MmLoadSystemImageEx function in IopLoadDriver is called and returns the error because PnpPrepareDriverLoading which follows MmLoadSystemImageEx on success, reads a registry value (PnpFlags) and does not appear on procmon and then next call I see is IopCheckIfNotNativeDriver.

Also, is it really not an option to run a debugger on these systems? That would be my ideal solution to the problem…We’ve previously had luck running VMware’s P2V utility to virtualize a failing machine that we couldn’t/weren’t allowed to physically debug.

The driver is installed in near 1000 computers of the same company and the problem appears in less than 10.

Kind regards,
Mauro.

The driver is installed in near 1000 computers of the same company and the problem appears in less than 10

Is it happening consistently on these ten machines? If so, that’s really good. Have them box one of those up and send it to you? Get a P2V image of one of them? Go on site and debug it?

Instrument the DriverEntry with DbgPrint or writes to the event log to be 100% sure it’s not being called?

(I’m reminded of a problem that occurred once a week on a gas station in a gas station control computer… the vendor’s team apparently camped out in some road-side gas station for weeks at a time to try to collect event data)

Something odd is going on, that’s for sure. And I’d personally want to see it myself.

I’m sorry… that’s probably not helpful.

Peter

Hi Peter I promise I shall give you the feedback when I can get something.

Today I was unable to get any feedback. Some customers are not friendly. Although I think I couldn’t get kernel feedback because forget to run BCDEDIT /debug on and BCDEDIT dbgsettings local

Yes it happens always on the same machines. And old version of the driver does load and the only major thing I did is to join the compilation. Previously I used WDK 8.1 to create a driver for XP-Win7 and latest WDK 10 for Win8+. Now I create a single driver and modify the subsystem to require Windows 5.1 (or 5.2 on 64-bits). Changed back to 6.2 with no success.

Because I also do dual signing (sha1+sha256), I thought the OS is not recognizing the third signature, the one added by Microsoft portal. (you know, one starts to try with anything that can sound strange). It wasn’t the case neither.

I would expect an invalid signature/hash error, but not an invalid parameter.

But I’m sure my DriverEntry is not called because the first thing I do is to send a debug message that appears on DbgView in the previous versions.

Stay tuned :slight_smile:

I would expect an invalid signature/hash error, but not an invalid parameter.

The translation from NTSTATUS to Win32 error code is many to one. So, just because you see “invalid parameter” in user mode it doesn’t mean that the driver actualy returned STATUS_INVALID_PARAMETER. You can use winerror.exe from the WDK to see all the translations (winerror.exe 87) and, sadly, there’s a ton of them.

Did you check the event log on the failing system? Tracing MmLoadSystemImageEx it looks like it writes to the event log on failure. If you’re lucky it will also contain the REAL error code, which might help narrow the focus.

Ok I could get something more useful from an event log:

00000000 00300002 00000000 4000001A 
C0000045 C000026C 00000000 00000000 
00000000 00000000 

0xC0000045 maps to STATUS_INVALID_PAGE_PROTECTION (0x4000001A seems to be a section protection flags?) which is translated to 87
0xC000026C maps to STATUS_DRIVER_UNABLE_TO_LOAD

Trying to see what is going on.

Thanks to all.

Device Guard maybe? Do you pass the Driver Verifier Code Integrity test? Details here:

https://blogs.msdn.microsoft.com/windows_hardware_certification/2015/05/22/driver-compatibility-with-device-guard-in-windows-10/

I> @“Scott_Noone_(OSR)” said:

Device Guard maybe? Do you pass the Driver Verifier Code Integrity test? Details here:

https://blogs.msdn.microsoft.com/windows_hardware_certification/2015/05/22/driver-compatibility-with-device-guard-in-windows-10/

Well, I was able to reproduce the issue by enabling Virtualization Based Protection. The problem is VMWARE enforces secure boot and makes the OS to remove /debug flag :frowning: so I’m unable to attach WinDbg.

I’ll also verify if there is something missing or not configured properly on HLK tests. Still in doubt why a previous driver works.

Regards,
Mauro.

Hi @“Scott_Noone_(OSR)” @“Peter_Viscarola_(OSR)”

Finally I found the issue: /INTEGRITYCHECK linker flag.

Before running HLK found OS drivers does not have the IMAGE_DLLCHARACTERISTICS_FORCE_INTEGRITY flag. I was thinking if this was the issue or the linker auto merging some sections.

I was lucky. The new driver now loads… and BSODs because another reason I have to fix hahaha but this is a different story.

Still my doubt is why previous versions of the driver didn’t suffer this issue.

Kind regards,
Mauro.

Argggghhhh, I need /INTEGRITYCHECKelse ObRegisterCallbacks does not work!!

Hi, I revive this because I could advance something.

First I discovered something funny. I put a break on my installer before rollback takes place and, at this point if I do:

  1. FLTMC LOAD MyDriver.sys → same error 87
  2. COPY C:\Windows\System32\Drivers\MyDriver.sys C:\AnotherLocation
  3. COPY C:\AnotherLocation\MyDriver.sys C:\Windows\System32\Drivers
  4. FLTMC LOAD MyDriver.sys driver LOADS!!!

So, by lowering procmon’s minifilter I could see, at MiCreateImageOrDataSection function, MiCreateNewSection is called when load succeedes and MiShareExistingControlArea when fails. (Based on what I saw, these are the paths depending if a CONTROL_AREA is being created or shared)

I started suspecting about shared sections and my first shot was to disable Windows Defender completely without success.

Then I saw a prefetch code in kernel also was creating a section so I disabled it too by setting EnablePrefetcher and SfTracingState to 0 under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PrefetchParameters

Now my driver loads!!

Any hint on what can be going on?

Regards,
Mauro.

Is this with a version of your driver built without /INTEGRITYCHECK? Or is this with the new driver? It’s weird in any case.

Hi @“Scott_Noone_(OSR)” ,

The driver causing the issue has /INTEGRITYCHECK active. I was able to get it working by doing this:

  1. Install driver
  2. Start it
  3. On error 87, try to open sys file, read first 1024 bytes, write again the 1024 bytes, close file and go to step 2.

After reading a document explaning Control Flow Guard, I decided to recompile the driver without it and that seems to be a solution (next monday I’ll get a confirmation)

So, as a resume of why I’m getting ERROR_INVALID_PARAMETER/STATUS_INVALID_PAGE_PROTECTION:

  1. Only happens when UEFI Secure Boot is enabled, prefetcher is active and driver compiled with /INTEGRITYCHECK and Control Flow Guard (VS 2017 + WDK 1809 although I think this is not related)
  2. Kernel prefetch component is causing the issue because it holds a section of the file with an incompatible protection flags required by the driver loader code.
  3. The above hack hack suits as a workaround.
  4. If Integrity Check cannot be removed, disabling CFG seems to be another workaround (to be confirmed).

I think this is a bug in Windows because two OS kernel components are involved, how can I report it? Windows Feedback?

Kind regards,
Mauro.

Which version of Windows 10 do you see this on? I’d like to try to repro.

1809 with all updates. I’ll try to compile a sample and see. VS 2017 also has the latest patches.

It is a minifilter driver and I install it using a copy and then, with a Wix’s custom action, I create the services (driver and app service) with CreateService api.

Before installer ends, I start my service which in turn loads the driver with FltLoad. I put a dummy messagebox before rollback starts and I get the error still for a long time.

I have the issue happening on a VM too if you cannot reproduce.

Cheers.

Hi @“Scott_Noone_(OSR)” and @“Peter_Viscarola_(OSR)”

Could contact a guy (https://twitter.com/TheRealHariP) from kernel team and he told me it is a pending issue to resolve. It is related to this one: https://support.microsoft.com/en-us/help/3194715/bugcheck-0x7e-occurs-in-windows-10-when-device-guard-is-active (you hit Scott)

Thanks both for the support.
Mauro.