Problems running SDV in VS 2017 and WDK 1803 - "slamcl: error: at phase 2: Out of memory"

I’m in the process of updating our KMDF driver build from Visual Studio
2015 and WDK 1703 to VS 2017 and WDK 1803 (running on Windows 10 1803),
and everything has gone well except that SDV keeps failing.

When I run SDV I get the following in smvbuild.log:

C:\Program Files (x86)\Microsoft Visual
Studio\2017\Professional\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targets(1100,5):
error MSB6006: “Lib.exe” exited with code 1.
[C:\Users\username\workspace\project\src\windows\subproject\subproject.vcxproj]
Done executing task “LIB” – FAILED.
Done building target “Lib” in project “subproject.vcxproj” – FAILED.

In sdv\smvlink1.log there’s more information:

slamcl Warning: Important: File “c:\program files (x86)\windows
kits\10\include\10.0.17134.0\km\wdm.h”, line 4557, function:
“ReadULongNoFence”: SDV encountered an internal compiler error around
this location.
slamcl: error: at phase 2: Out of memory
slamcl error at translate’ }

On each build, the filename and/or function mentioned on the warning
line changes. I’ve configured SDV-default.xml to use 1 thread and 2500MB
memory, and am out of ideas.

I read that having other versions of Visual Studio or the WDK installed
can cause problems, so I uninstalled VS 2015, WDK 1703 and the older
SDK. I also tried updating the target KMDF version in all the projects
from 1.11 to 1.19 and disabled LTCG but nothing changed. The machine
itself has 32GB RAM and 32 CPU threads (2 CPUs with 8 hyper-threaded
cores each).


Rebecca Cran

I can say that SDV is NOT hopelessly broken on 1803 (i.e. “works for me”).

Have you tried building a WDK sample or a template driver? There might be
something weird about your project that causes the failures.

That being said, it’s very possible that your install is borked. Having
these things side by side *should* work but it’s also known to cause
problems sometimes.

-scott
OSR
@OSRDrivers

Another thing to at least TRY: There are settings for maximum threads and memory for SDV (on the SDV “Configure” tab. Be sure these are set to the maximum allowed values (which should also be the defaults). I suggest this, not because I know this to make any difference, but rather because that’s the way we always run it.

You can also try selecting just one or two rules and see if that makes any difference.

It should go without saying that your driver must compile with no warnings or errors. If you have any of these, be sure to fix them. Yes, I *do* realize that the error message you showed us doesn’t relate to anything in your driver… but still.

SDV can be an odd beast. Temperamental, but often very useful.

Peter
OSR
@OSRDrivers

> That being said, it’s very possible that your install is borked. Having
> these things side by side *should* work but it’s also known to cause
> problems sometimes.

The WDK isn’t fully “side by side” – you only get versioned headers and
libraries. Each new WDK overwrites the prior version’s tools (…\Windows
Kits\10\Tools) and MSBuild files (…\Windows Kits\10\build). The SDK tools
(…\Windows Kits\10\bin) *are* versioned, but not the WDK. This means you
can only have one version of SDV installed.

This can easily become a problem if you install a pre-release/preview WDK.
Now you’ve got “beta quality” WDK tools, with no quick and easy way to get
back to a “release” state.

In my experience, uninstalling some SDK/WDK versions while leaving others
installed is unreliable. At a minimum you should do a “repair” on the
remaining SDK/WDK afterwards. Or just nuke the entire machine from orbit and
reinstall everything – it’s the only way to be sure :-).

–John

On 6/14/2018 12:10 PM, xxxxx@microwiz.com wrote:

In my experience, uninstalling some SDK/WDK versions while leaving
others installed is unreliable.  At a minimum you should do a “repair”
on the remaining SDK/WDK afterwards.  Or just nuke the entire machine
from orbit and reinstall everything – it’s the only way to be sure :-).

I tried that (uninstalled Visual Studio 2017 and WDK 1803, and
reinstalled) and still get the internal compiler error “Out of memory”
with our driver. However the PCIdrv sample /does/ complete, which
suggests it’s something specific to ours. Though I did notice that if I
enable optimization and LTCG in the PCIdrv project, when SDV finishes
the window disappears and AngelicVerifierNull.exe, FastAVN.exe and
z3.exe run and start using all the CPU, memory and disk bandwidth they
can. So I’ll try backing off the settings in our driver and see if that
helps.


Rebecca Cran