Containerized WDK Build Environments?

We’re keenly interested in creating containerized environments containing specific versions of Visual Studio and the WDK for use in our build processes (in other words: “We have different projects that build with different tool chains, and we always will… and we’re frustrated with having new versions of VS and the WDK break down-level installs of VS and the WDK and we are looking to find a solution”)

Yes, I know the EWDK “solves this”… but, of course, it doesn’t because you can’t build any meaningful user-mode applications using the EWDK.

So… Has anybody tried this? Done it successfully, maybe? If so, can you share any keys, experiences, stories, hints or tips? Caveats? Tales of woe?

I *do* promise that when/if we get this working, we’ll write-up instructions (in an article in The NT Insider, for example) that provides a guide to how to get this working for anyone who follows in our footsteps.

Peter
OSR
@OSRDrivers

On Wed, Jun 20, 2018 at 3:41 PM, xxxxx@osr.com wrote:
> We’re keenly interested in creating containerized environments containing specific versions of Visual Studio and the WDK for use in our build processes (in other words: “We have different projects that build with different tool chains, and we always will… and we’re frustrated with having new versions of VS and the WDK break down-level installs of VS and the WDK and we are looking to find a solution”)
>

Container is probably the wrong word. Containers typically run under a
single kernel. That said, most people will know what you mean.

> Yes, I know the EWDK “solves this”… but, of course, it doesn’t because you can’t build any meaningful user-mode applications using the EWDK.
>
> So… Has anybody tried this? Done it successfully, maybe? If so, can you share any keys, experiences, stories, hints or tips? Caveats? Tales of woe?
>

I tried to use a virtual machine for development. I could build
everything properly and have no reason to believe deployment was at
issue on the host side, but from a VM I was unable to deploy to
another VM, even though deploying to a VM is supposed to be supported.

Note that HLK tests are not supported on virtual machines (though they
may run). It looks like no thought was given to virtual environments.
At the same time I see no obvious reason why a virtual environment
shouldn’t work.

The downside is a VM is fairly expensive.

> I do promise that when/if we get this working, we’ll write-up instructions (in an article in The NT Insider, for example) that provides a guide to how to get this working for anyone who follows in our footsteps.
>

For the purpose of maintaining build and testing machines I was
looking into getting automated install machinery set up for Windows.
However that is a huge time sink and it is still not really automated,
so maintaining a netboot image or disk images sent to devices seems to
be the better option.

The removal of the need for discrete machines would make driver
development a lot more accessible. A fairly modest testbed appears to
currently be needed (2 to 5 machines).

Seeing as OSR is employed in doing this I look forward to seeing what
they come up with. You will get it done before I do.

Cheers,
R0b0t1

NO. It’s not the wrong word.

I’m asking, specifically, about containers. Windows Containers, in fact. With Docker. And Packer.
With Visual Studio installed. This is already a “done thing” for apps, and that’s why I’m asking about it for drivers.

See https:

Peter
OSR
@OSRDrivers</https:>

I’ve been wanting exactly this, but haven’t had time to explore it myself. Only commenting to share that there definitely is interest from “the field”.

Peter, Shane. Good point. Good conversation.

From WDK team’s side we absolutely need the type of “isolation” that Peter looks for.

WDKs SxS: It is painful if you need to build against old WDKs and new WDKs on the same machine. There are some fundamental limitations that makes it hard to achieve what Peter needs with just installing a bunch of WDKs on top of each other. Essentially some of the files in the SDK and WDK are ‘shared’ and there can at any given time only be one version of those files on a machine at a time. To add to this fundamental limitation we have also over the last few years been burned by a number of subtle MSI authoring rules that we got wrong. Sorry.

WDKs in containers: Peter’s suggestion is another way to get isolation. Very exciting. I believe the concept works fine, but there will be problems that the container OS (a server core variant I believe) does not provide capabilities a few of the WDK tools needed. For example I remember API Validator does not work because it is missing a runtime component. It is possible I have my wires crossed and it is possible the issues are easy to resolved. We should investigate. We probably will take a look soon and if we learn anything meaningful we will report back.

EWDK: Internally we use the EWDK for a light weight and fairly solid isolation. We absolutely love it. We have discussed with SDK team to extend the app build capabilities in EWDK, so Peter if you can let us know more about what you would need from the EWDK, then we’ll take a look. There will be limits to what can be incorporated in the EWDK, but perhaps we can get enough in to help you.

Long term: We are discussing more fundamental approaches to provide isolation that does not require the EWDK style and integrate well with Visual Studio. We don’t have more to share yet, but we consider the type of problem Peter is experiencing absolutely important to address.

Best,

Jakob

On Wed, Jun 20, 2018 at 4:14 PM, xxxxx@osr.com wrote:
>


>
> NO. It’s not the wrong word.
>
> I’m asking, specifically, about containers. Windows Containers, in fact. With Docker. And Packer.
> With Visual Studio installed. This is already a “done thing” for apps, and that’s why I’m asking about it for drivers.
>
> See https:
>

The first step in the link you provided is “Enable Hyper-V.” From here
it really looks like Microsoft branded a Docker based interface to a
Windows version of libvirt as “containers” when technologically they
are not containers at all. There is a process isolation version I will
read more about.

It looks like they support automated deployment better, so that is good.

I am glad to know that I am wrong in my assumption about the eye
towards compartmentalization. I suppose it will still be a while.

Cheers,
R0b0t1</https:>

(For those who may not be familiar with Mr. Lichtenberg… he’s the dev lead for the WDK and related tools, and a LONG time friend of the driver development community).

Thank you, Mr. Lichtenberg, for jumping into the conversation. Your help and insight is always appreciated.

Honestly… the problem is that it’s *very* hard to narrow down. “Back in the day” we might have had a driver and a (native) C/C++ test app in one solution and that’s it. THESE days, that test app MIGHT be C/C++, or it might equally be written in C#, or there could be a service in the Solution (in C/C++ or C#), or it might be some UWP management app (OK, not really the last of those… but someday, maybe).

We *could* be convinced to build the user-mode parts separately (as a separate Solution)… but we still have to build both parts (the driver Solution and the UM Solution) as part of our CI process. So, bottom line, we need to have a full-blown VS (to accommodate the various user-mode options) and the driver part.

That’s good news.

One approach might be to (a) ship along with the WDK the pieces/parts that are needed to ensure that tools work on Windows Server, (b) provide a container with VS Community and the WDK pre-installed. Extra points for having such a container on Docker Hub. How cool would THAT be??

ANYhow… We still plan to give this a try, and see how it works.

AND I hope there are others in the community who have tried this, and can share the details of their experiences.

Regardless, as I said, we’ll write-up whatever we come up with to help those who come after us.

Peter
OSR
@OSRDrivers

On Thu, Jun 21, 2018 at 8:33 AM, xxxxx@osr.com wrote:
>


>

Ah, sorry. To clarify I meant automated deployment of the development
environment, not automated deployment of the driver under test.

In the latter part of your second post this is kind of what you are
getting at, though implemented via Docker. I think it important to
suggest further the addition of some kind of programmable interface to
the EWDK and its invocation. (Or, perhaps, better linking to
documentation for what currently exists.)

I don’t want to go off into the weeds too far but a closer to native
solution I think would be desirable but may not be possible. Remote
command execution on Windows is generally seen as hard to configure
and consequently insecure. There are also serious problems running
remote GUI processes related to the implementation of terminal
services and session, station, and desktop isolation.

The question may be more generally stated as “what problems are there
creating an interactive build server?”

Cheers,
R0b0t1

Whew! Thank you for that clarification. I am much relieved. I would hate to think of you toiling away with that OTHER kind of deployment.

I’m pretty sure these are solved problems, R0b0t1. People in the real world DO this kind of shit all the time. Us, here in OS development and driver land, well… we tend to be behind the hipsters. But, you know… Now even Microsoft uses git.

I’m not saying I’ve got this problem solved… I’m just saying I’m betting that I can get Jenkins to fire-up a docker image with VS and the WDK in it and build… well… something. Now, Mr. Lichtenberg was kind enough to warn us in advance that this might not be simple. But… let’s see!

I must say… I am *not* encouraged by the astounding SILENCE I’ve heard from the community in response to this question. I *really* expected SOMEbody to tell me that had done it and it worked, and that they TRIED to do it and it failed miserably.

Peter
OSR
@OSRDrivers

The closest thing I’ve done to address this problem is to use Vagrant and create separate VM’s that have just the build utilities installed.

Each project fires up the appropriate VM, pulls down the code, builds what it needs, and saves off the build output, before forgetting about everything that changed in that VM.

It *works*, but it’s kinda heavy weight. It’s kinda slow. But it does what we want which is to integrate with our CI system to build/test/release all variants for a given code change.

It’s not exactly the WDK or EWDK, but this looks to be a decent foundation: https://docs.microsoft.com/en-us/visualstudio/install/build-tools-container

Caveat: I haven’t tried this myself, but I did a quick look because I’m going to need to do something similar in the not-too-distant future.

Phil

Not speaking for LogRhythm

Arghh. I should have looked back over the thread, I would have noticed that the URL I posted is a repeat from earlier in the thread. I hope the forum package includes edit/delete, so we can undo dopiness on our part. :slight_smile:

An update. I had a lot of fun looking at containers for WDK and I am very impressed with Docker. I was able to put together a first version of a “windowsdriverkit1803” container.

How:
* I followed the instructions posted on https://docs.microsoft.com/en-us/visualstudio/install/build-tools-container
* However I used an updated Dockerfile (see bottom of this post)
* The update Dockerfile does following:
* Installs the WDK (including a rather hacky installation of the WDK.vsix)
* Installs git
* Finally clones the Windows Driver Samples git repo.

You build the container all as follows:

docker build -t windowsdriverkit1803:latest -m 2GB .


First time the build took hours. Later builds are due to the Docker caching running in a few seconds.

Once built I could launch the container and build one of our sample drivers from inside the container:

docker run -it windowsdriverkit1803
cd “C:\Windows-driver-samples\tools\sdv\samples\SDV-FailDriver-WDM”
msbuild


Next steps: This is largely untested and there are still lots of thing to look and refine/fix. For example (and as mentioned in my first post) API Validator does not work. We’ll look at that later. But in the interest of sharing I thought it useful to post our progress. Thanks lots to Adonais from our team helping me put this together.

Best,

Jakob

Here follows the Dockerfile:

escape=`

Use the latest Windows Server Core image with .NET Framework 4.7.1.

FROM microsoft/dotnet-framework:4.7.1

Download the Build Tools bootstrapper.

ADD https://aka.ms/vs/15/release/vs_buildtools.exe C:\TEMP\vs_buildtools.exe

Install Build Tools excluding workloads and components with known issues.

RUN C:\TEMP\vs_buildtools.exe --quiet --wait --norestart --nocache --installPath C:\BuildTools
–all --remove Microsoft.VisualStudio.Component.Windows10SDK.10240
–remove Microsoft.VisualStudio.Component.Windows10SDK.10586 --remove Microsoft.VisualStudio.Component.Windows10SDK.14393
–remove Microsoft.VisualStudio.Component.Windows81SDK `
|| IF “%ERRORLEVEL%”==“3010” EXIT 0

Download the WDK bootstrapper.

ADD https://go.microsoft.com/fwlink/?linkid=873060 C:\TEMP\wdksetup.exe

Install WDK excluding WDK.vsix.

RUN C:\TEMP\wdksetup.exe /q

Install WDK.vsix in manual manner.

RUN copy “C:\Program Files (x86)\Windows Kits\10\Vsix\WDK.vsix” C:\TEMP\wdkvsix.zip
RUN powershell Expand-Archive C:\TEMP\wdkvsix.zip -DestinationPath C:\TEMP\wdkvsix
RUN robocopy /e “C:\TEMP\wdkvsix$VCTargets\Platforms” “C:\BuildTools\Common7\IDE\VC\VCTargets\Platforms” || EXIT 0

Download GIT.

ADD https://github.com/git-for-windows/git/releases/download/v2.18.0.windows.1/MinGit-2.18.0-64-bit.zip C:\TEMP\MinGit.zip
RUN powershell Expand-Archive C:\TEMP\MinGit.zip -DestinationPath C:\MinGit

Clone Windows-driver-samples git repo.

RUN C:\MinGit\cmd\git.exe clone https://github.com/Microsoft/Windows-driver-samples.git

Start developer command prompt with any other commands specified.

ENTRYPOINT C:\BuildTools\Common7\Tools\VsDevCmd.bat &&

Default to PowerShell if no other command specified.

CMD [“powershell.exe”, “-NoLogo”, “-ExecutionPolicy”, “Bypass”]

The messaging board ate the last four lines of Dockerfile. Here it is:

?

Start developer command prompt with any other commands specified.

ENTRYPOINT C:\BuildTools\Common7\Tools\VsDevCmd.bat &&

Default to PowerShell if no other command specified.

CMD [“powershell.exe”, “-NoLogo”, “-ExecutionPolicy”, “Bypass”]

Wow! I love this. Thank you, Mr. Lichtenberg! Both for taking your time to work on this (you *and* Adonais) and for sharing your experience.

Love it!

Peter
OSR
@OSRDrivers

I looked at the EWDK as a starting point for a containerized build solution
and gave up because it was just too huge to be a viable container image and
there appears to be no reasonable way to get docker containers to use
network share mount points. A msft supported docker image for each
released wdk with reasonable size would be a huge win from my perspective.

Mark Roddy

On Thu, Jun 21, 2018 at 10:15 AM xxxxx@osr.com
wrote:

> (For those who may not be familiar with Mr. Lichtenberg… he’s the dev
> lead for the WDK and related tools, and a LONG time friend of the driver
> development community).
>
> Thank you, Mr. Lichtenberg, for jumping into the conversation. Your help
> and insight is always appreciated.
>
>


>
> Honestly… the problem is that it’s very hard to narrow down. “Back in
> the day” we might have had a driver and a (native) C/C++ test app in one
> solution and that’s it. THESE days, that test app MIGHT be C/C++, or it
> might equally be written in C#, or there could be a service in the Solution
> (in C/C++ or C#), or it might be some UWP management app (OK, not really
> the last of those… but someday, maybe).
>
> We could be convinced to build the user-mode parts separately (as a
> separate Solution)… but we still have to build both parts (the driver
> Solution and the UM Solution) as part of our CI process. So, bottom line,
> we need to have a full-blown VS (to accommodate the various user-mode
> options) and the driver part.
>
>


>
> That’s good news.
>
> One approach might be to (a) ship along with the WDK the pieces/parts that
> are needed to ensure that tools work on Windows Server, (b) provide a
> container with VS Community and the WDK pre-installed. Extra points for
> having such a container on Docker Hub. How cool would THAT be??
>
> ANYhow… We still plan to give this a try, and see how it works.
>
> AND I hope there are others in the community who have tried this, and can
> share the details of their experiences.
>
> Regardless, as I said, we’ll write-up whatever we come up with to help
> those who come after us.
>
> Peter
> OSR
> @OSRDrivers
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: <
> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

And I should read the thread to the bottom :-). That docker script looks
great. Will hack away at it.
Mark Roddy

On Wed, Jun 27, 2018 at 2:20 PM Mark Roddy wrote:

> I looked at the EWDK as a starting point for a containerized build
> solution and gave up because it was just too huge to be a viable container
> image and there appears to be no reasonable way to get docker containers to
> use network share mount points. A msft supported docker image for each
> released wdk with reasonable size would be a huge win from my perspective.
>
> Mark Roddy
>
>
> On Thu, Jun 21, 2018 at 10:15 AM xxxxx@osr.com
> wrote:
>
>> (For those who may not be familiar with Mr. Lichtenberg… he’s the dev
>> lead for the WDK and related tools, and a LONG time friend of the driver
>> development community).
>>
>> Thank you, Mr. Lichtenberg, for jumping into the conversation. Your help
>> and insight is always appreciated.
>>
>>


>>
>> Honestly… the problem is that it’s very hard to narrow down. “Back
>> in the day” we might have had a driver and a (native) C/C++ test app in one
>> solution and that’s it. THESE days, that test app MIGHT be C/C++, or it
>> might equally be written in C#, or there could be a service in the Solution
>> (in C/C++ or C#), or it might be some UWP management app (OK, not really
>> the last of those… but someday, maybe).
>>
>> We could be convinced to build the user-mode parts separately (as a
>> separate Solution)… but we still have to build both parts (the driver
>> Solution and the UM Solution) as part of our CI process. So, bottom line,
>> we need to have a full-blown VS (to accommodate the various user-mode
>> options) and the driver part.
>>
>>


>>
>> That’s good news.
>>
>> One approach might be to (a) ship along with the WDK the pieces/parts
>> that are needed to ensure that tools work on Windows Server, (b) provide a
>> container with VS Community and the WDK pre-installed. Extra points for
>> having such a container on Docker Hub. How cool would THAT be??
>>
>> ANYhow… We still plan to give this a try, and see how it works.
>>
>> AND I hope there are others in the community who have tried this, and
>> can share the details of their experiences.
>>
>> Regardless, as I said, we’ll write-up whatever we come up with to help
>> those who come after us.
>>
>> Peter
>> OSR
>> @OSRDrivers
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> Visit the list online at: <
>> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>>
>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
>> software drivers!
>> Details at http:
>>
>> To unsubscribe, visit the List Server section of OSR Online at <
>> http://www.osronline.com/page.cfm?name=ListServer&gt;
>>
></http:>

Oh that is just the ticket. The container works great. Greatest container
ever. Now I have to go convince it to build our stuff.

Thanks Jakob!

Mark Roddy

On Wed, Jun 27, 2018 at 2:26 PM Mark Roddy wrote:

> And I should read the thread to the bottom :-). That docker script looks
> great. Will hack away at it.
> Mark Roddy
>
>
> On Wed, Jun 27, 2018 at 2:20 PM Mark Roddy wrote:
>
>> I looked at the EWDK as a starting point for a containerized build
>> solution and gave up because it was just too huge to be a viable container
>> image and there appears to be no reasonable way to get docker containers to
>> use network share mount points. A msft supported docker image for each
>> released wdk with reasonable size would be a huge win from my perspective.
>>
>> Mark Roddy
>>
>>
>> On Thu, Jun 21, 2018 at 10:15 AM xxxxx@osr.com
>> wrote:
>>
>>> (For those who may not be familiar with Mr. Lichtenberg… he’s the dev
>>> lead for the WDK and related tools, and a LONG time friend of the driver
>>> development community).
>>>
>>> Thank you, Mr. Lichtenberg, for jumping into the conversation. Your
>>> help and insight is always appreciated.
>>>
>>>


>>>
>>> Honestly… the problem is that it’s very hard to narrow down. “Back
>>> in the day” we might have had a driver and a (native) C/C++ test app in one
>>> solution and that’s it. THESE days, that test app MIGHT be C/C++, or it
>>> might equally be written in C#, or there could be a service in the Solution
>>> (in C/C++ or C#), or it might be some UWP management app (OK, not really
>>> the last of those… but someday, maybe).
>>>
>>> We could be convinced to build the user-mode parts separately (as a
>>> separate Solution)… but we still have to build both parts (the driver
>>> Solution and the UM Solution) as part of our CI process. So, bottom line,
>>> we need to have a full-blown VS (to accommodate the various user-mode
>>> options) and the driver part.
>>>
>>>


>>>
>>> That’s good news.
>>>
>>> One approach might be to (a) ship along with the WDK the pieces/parts
>>> that are needed to ensure that tools work on Windows Server, (b) provide a
>>> container with VS Community and the WDK pre-installed. Extra points for
>>> having such a container on Docker Hub. How cool would THAT be??
>>>
>>> ANYhow… We still plan to give this a try, and see how it works.
>>>
>>> AND I hope there are others in the community who have tried this, and
>>> can share the details of their experiences.
>>>
>>> Regardless, as I said, we’ll write-up whatever we come up with to help
>>> those who come after us.
>>>
>>> Peter
>>> OSR
>>> @OSRDrivers
>>>
>>>
>>> —
>>> NTDEV is sponsored by OSR
>>>
>>> Visit the list online at: <
>>> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>>>
>>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
>>> software drivers!
>>> Details at http:
>>>
>>> To unsubscribe, visit the List Server section of OSR Online at <
>>> http://www.osronline.com/page.cfm?name=ListServer&gt;
>>>
>></http:>

So now that I’ve played around with this for a few days, I have to say that
“windows containers” appear to be a moving target on WIn10. Building the
container on a current release works, building on an insider build is a
nightmare of inconsistent results. Once the container is built and
validated I think it is good to go on any supported platform, regardless of
where it was originally generated, but I haven’t verified that.

Of course I could do all of this on S2016 instead, but then I’d have to
deal with nested hyper-visors and that is another nightmare of what works
and what doesn’t…

Mark Roddy

On Fri, Jun 29, 2018 at 3:23 PM Mark Roddy wrote:

> Oh that is just the ticket. The container works great. Greatest container
> ever. Now I have to go convince it to build our stuff.
>
> Thanks Jakob!
>
> Mark Roddy
>
>
> On Wed, Jun 27, 2018 at 2:26 PM Mark Roddy wrote:
>
>> And I should read the thread to the bottom :-). That docker script looks
>> great. Will hack away at it.
>> Mark Roddy
>>
>>
>> On Wed, Jun 27, 2018 at 2:20 PM Mark Roddy wrote:
>>
>>> I looked at the EWDK as a starting point for a containerized build
>>> solution and gave up because it was just too huge to be a viable container
>>> image and there appears to be no reasonable way to get docker containers to
>>> use network share mount points. A msft supported docker image for each
>>> released wdk with reasonable size would be a huge win from my perspective.
>>>
>>> Mark Roddy
>>>
>>>
>>> On Thu, Jun 21, 2018 at 10:15 AM xxxxx@osr.com
>>> wrote:
>>>
>>>> (For those who may not be familiar with Mr. Lichtenberg… he’s the dev
>>>> lead for the WDK and related tools, and a LONG time friend of the driver
>>>> development community).
>>>>
>>>> Thank you, Mr. Lichtenberg, for jumping into the conversation. Your
>>>> help and insight is always appreciated.
>>>>
>>>>


>>>>
>>>> Honestly… the problem is that it’s very hard to narrow down. “Back
>>>> in the day” we might have had a driver and a (native) C/C++ test app in one
>>>> solution and that’s it. THESE days, that test app MIGHT be C/C++, or it
>>>> might equally be written in C#, or there could be a service in the Solution
>>>> (in C/C++ or C#), or it might be some UWP management app (OK, not really
>>>> the last of those… but someday, maybe).
>>>>
>>>> We could be convinced to build the user-mode parts separately (as a
>>>> separate Solution)… but we still have to build both parts (the driver
>>>> Solution and the UM Solution) as part of our CI process. So, bottom line,
>>>> we need to have a full-blown VS (to accommodate the various user-mode
>>>> options) and the driver part.
>>>>
>>>>


>>>>
>>>> That’s good news.
>>>>
>>>> One approach might be to (a) ship along with the WDK the pieces/parts
>>>> that are needed to ensure that tools work on Windows Server, (b) provide a
>>>> container with VS Community and the WDK pre-installed. Extra points for
>>>> having such a container on Docker Hub. How cool would THAT be??
>>>>
>>>> ANYhow… We still plan to give this a try, and see how it works.
>>>>
>>>> AND I hope there are others in the community who have tried this, and
>>>> can share the details of their experiences.
>>>>
>>>> Regardless, as I said, we’ll write-up whatever we come up with to help
>>>> those who come after us.
>>>>
>>>> Peter
>>>> OSR
>>>> @OSRDrivers
>>>>
>>>>
>>>> —
>>>> NTDEV is sponsored by OSR
>>>>
>>>> Visit the list online at: <
>>>> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>>>>
>>>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
>>>> software drivers!
>>>> Details at http:
>>>>
>>>> To unsubscribe, visit the List Server section of OSR Online at <
>>>> http://www.osronline.com/page.cfm?name=ListServer&gt;
>>>>
>>></http:>