Containerized WDK Build Environments?

(For those who may not be familiar with Mr. Lichtenberg… he’s the dev lead for the WDK and related tools, and a LONG time friend of the driver development community).

Thank you, Mr. Lichtenberg, for jumping into the conversation. Your help and insight is always appreciated.

Honestly… the problem is that it’s *very* hard to narrow down. “Back in the day” we might have had a driver and a (native) C/C++ test app in one solution and that’s it. THESE days, that test app MIGHT be C/C++, or it might equally be written in C#, or there could be a service in the Solution (in C/C++ or C#), or it might be some UWP management app (OK, not really the last of those… but someday, maybe).

We *could* be convinced to build the user-mode parts separately (as a separate Solution)… but we still have to build both parts (the driver Solution and the UM Solution) as part of our CI process. So, bottom line, we need to have a full-blown VS (to accommodate the various user-mode options) and the driver part.

That’s good news.

One approach might be to (a) ship along with the WDK the pieces/parts that are needed to ensure that tools work on Windows Server, (b) provide a container with VS Community and the WDK pre-installed. Extra points for having such a container on Docker Hub. How cool would THAT be??

ANYhow… We still plan to give this a try, and see how it works.

AND I hope there are others in the community who have tried this, and can share the details of their experiences.

Regardless, as I said, we’ll write-up whatever we come up with to help those who come after us.

Peter
OSR
@OSRDrivers

On Thu, Jun 21, 2018 at 8:33 AM, xxxxx@osr.com wrote:
>


>

Ah, sorry. To clarify I meant automated deployment of the development
environment, not automated deployment of the driver under test.

In the latter part of your second post this is kind of what you are
getting at, though implemented via Docker. I think it important to
suggest further the addition of some kind of programmable interface to
the EWDK and its invocation. (Or, perhaps, better linking to
documentation for what currently exists.)

I don’t want to go off into the weeds too far but a closer to native
solution I think would be desirable but may not be possible. Remote
command execution on Windows is generally seen as hard to configure
and consequently insecure. There are also serious problems running
remote GUI processes related to the implementation of terminal
services and session, station, and desktop isolation.

The question may be more generally stated as “what problems are there
creating an interactive build server?”

Cheers,
R0b0t1

Whew! Thank you for that clarification. I am much relieved. I would hate to think of you toiling away with that OTHER kind of deployment.

I’m pretty sure these are solved problems, R0b0t1. People in the real world DO this kind of shit all the time. Us, here in OS development and driver land, well… we tend to be behind the hipsters. But, you know… Now even Microsoft uses git.

I’m not saying I’ve got this problem solved… I’m just saying I’m betting that I can get Jenkins to fire-up a docker image with VS and the WDK in it and build… well… something. Now, Mr. Lichtenberg was kind enough to warn us in advance that this might not be simple. But… let’s see!

I must say… I am *not* encouraged by the astounding SILENCE I’ve heard from the community in response to this question. I *really* expected SOMEbody to tell me that had done it and it worked, and that they TRIED to do it and it failed miserably.

Peter
OSR
@OSRDrivers

The closest thing I’ve done to address this problem is to use Vagrant and create separate VM’s that have just the build utilities installed.

Each project fires up the appropriate VM, pulls down the code, builds what it needs, and saves off the build output, before forgetting about everything that changed in that VM.

It *works*, but it’s kinda heavy weight. It’s kinda slow. But it does what we want which is to integrate with our CI system to build/test/release all variants for a given code change.

It’s not exactly the WDK or EWDK, but this looks to be a decent foundation: https://docs.microsoft.com/en-us/visualstudio/install/build-tools-container

Caveat: I haven’t tried this myself, but I did a quick look because I’m going to need to do something similar in the not-too-distant future.

Phil

Not speaking for LogRhythm

Arghh. I should have looked back over the thread, I would have noticed that the URL I posted is a repeat from earlier in the thread. I hope the forum package includes edit/delete, so we can undo dopiness on our part. :slight_smile:

An update. I had a lot of fun looking at containers for WDK and I am very impressed with Docker. I was able to put together a first version of a “windowsdriverkit1803” container.

How:
* I followed the instructions posted on https://docs.microsoft.com/en-us/visualstudio/install/build-tools-container
* However I used an updated Dockerfile (see bottom of this post)
* The update Dockerfile does following:
* Installs the WDK (including a rather hacky installation of the WDK.vsix)
* Installs git
* Finally clones the Windows Driver Samples git repo.

You build the container all as follows:

docker build -t windowsdriverkit1803:latest -m 2GB .


First time the build took hours. Later builds are due to the Docker caching running in a few seconds.

Once built I could launch the container and build one of our sample drivers from inside the container:

docker run -it windowsdriverkit1803
cd “C:\Windows-driver-samples\tools\sdv\samples\SDV-FailDriver-WDM”
msbuild


Next steps: This is largely untested and there are still lots of thing to look and refine/fix. For example (and as mentioned in my first post) API Validator does not work. We’ll look at that later. But in the interest of sharing I thought it useful to post our progress. Thanks lots to Adonais from our team helping me put this together.

Best,

Jakob

Here follows the Dockerfile:

escape=`

Use the latest Windows Server Core image with .NET Framework 4.7.1.

FROM microsoft/dotnet-framework:4.7.1

Download the Build Tools bootstrapper.

ADD https://aka.ms/vs/15/release/vs_buildtools.exe C:\TEMP\vs_buildtools.exe

Install Build Tools excluding workloads and components with known issues.

RUN C:\TEMP\vs_buildtools.exe --quiet --wait --norestart --nocache --installPath C:\BuildTools
–all --remove Microsoft.VisualStudio.Component.Windows10SDK.10240
–remove Microsoft.VisualStudio.Component.Windows10SDK.10586 --remove Microsoft.VisualStudio.Component.Windows10SDK.14393
–remove Microsoft.VisualStudio.Component.Windows81SDK `
|| IF “%ERRORLEVEL%”==“3010” EXIT 0

Download the WDK bootstrapper.

ADD https://go.microsoft.com/fwlink/?linkid=873060 C:\TEMP\wdksetup.exe

Install WDK excluding WDK.vsix.

RUN C:\TEMP\wdksetup.exe /q

Install WDK.vsix in manual manner.

RUN copy “C:\Program Files (x86)\Windows Kits\10\Vsix\WDK.vsix” C:\TEMP\wdkvsix.zip
RUN powershell Expand-Archive C:\TEMP\wdkvsix.zip -DestinationPath C:\TEMP\wdkvsix
RUN robocopy /e “C:\TEMP\wdkvsix$VCTargets\Platforms” “C:\BuildTools\Common7\IDE\VC\VCTargets\Platforms” || EXIT 0

Download GIT.

ADD https://github.com/git-for-windows/git/releases/download/v2.18.0.windows.1/MinGit-2.18.0-64-bit.zip C:\TEMP\MinGit.zip
RUN powershell Expand-Archive C:\TEMP\MinGit.zip -DestinationPath C:\MinGit

Clone Windows-driver-samples git repo.

RUN C:\MinGit\cmd\git.exe clone https://github.com/Microsoft/Windows-driver-samples.git

Start developer command prompt with any other commands specified.

ENTRYPOINT C:\BuildTools\Common7\Tools\VsDevCmd.bat &&

Default to PowerShell if no other command specified.

CMD [“powershell.exe”, “-NoLogo”, “-ExecutionPolicy”, “Bypass”]

The messaging board ate the last four lines of Dockerfile. Here it is:

?

Start developer command prompt with any other commands specified.

ENTRYPOINT C:\BuildTools\Common7\Tools\VsDevCmd.bat &&

Default to PowerShell if no other command specified.

CMD [“powershell.exe”, “-NoLogo”, “-ExecutionPolicy”, “Bypass”]

Wow! I love this. Thank you, Mr. Lichtenberg! Both for taking your time to work on this (you *and* Adonais) and for sharing your experience.

Love it!

Peter
OSR
@OSRDrivers

I looked at the EWDK as a starting point for a containerized build solution
and gave up because it was just too huge to be a viable container image and
there appears to be no reasonable way to get docker containers to use
network share mount points. A msft supported docker image for each
released wdk with reasonable size would be a huge win from my perspective.

Mark Roddy

On Thu, Jun 21, 2018 at 10:15 AM xxxxx@osr.com
wrote:

> (For those who may not be familiar with Mr. Lichtenberg… he’s the dev
> lead for the WDK and related tools, and a LONG time friend of the driver
> development community).
>
> Thank you, Mr. Lichtenberg, for jumping into the conversation. Your help
> and insight is always appreciated.
>
>


>
> Honestly… the problem is that it’s very hard to narrow down. “Back in
> the day” we might have had a driver and a (native) C/C++ test app in one
> solution and that’s it. THESE days, that test app MIGHT be C/C++, or it
> might equally be written in C#, or there could be a service in the Solution
> (in C/C++ or C#), or it might be some UWP management app (OK, not really
> the last of those… but someday, maybe).
>
> We could be convinced to build the user-mode parts separately (as a
> separate Solution)… but we still have to build both parts (the driver
> Solution and the UM Solution) as part of our CI process. So, bottom line,
> we need to have a full-blown VS (to accommodate the various user-mode
> options) and the driver part.
>
>


>
> That’s good news.
>
> One approach might be to (a) ship along with the WDK the pieces/parts that
> are needed to ensure that tools work on Windows Server, (b) provide a
> container with VS Community and the WDK pre-installed. Extra points for
> having such a container on Docker Hub. How cool would THAT be??
>
> ANYhow… We still plan to give this a try, and see how it works.
>
> AND I hope there are others in the community who have tried this, and can
> share the details of their experiences.
>
> Regardless, as I said, we’ll write-up whatever we come up with to help
> those who come after us.
>
> Peter
> OSR
> @OSRDrivers
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: <
> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

And I should read the thread to the bottom :-). That docker script looks
great. Will hack away at it.
Mark Roddy

On Wed, Jun 27, 2018 at 2:20 PM Mark Roddy wrote:

> I looked at the EWDK as a starting point for a containerized build
> solution and gave up because it was just too huge to be a viable container
> image and there appears to be no reasonable way to get docker containers to
> use network share mount points. A msft supported docker image for each
> released wdk with reasonable size would be a huge win from my perspective.
>
> Mark Roddy
>
>
> On Thu, Jun 21, 2018 at 10:15 AM xxxxx@osr.com
> wrote:
>
>> (For those who may not be familiar with Mr. Lichtenberg… he’s the dev
>> lead for the WDK and related tools, and a LONG time friend of the driver
>> development community).
>>
>> Thank you, Mr. Lichtenberg, for jumping into the conversation. Your help
>> and insight is always appreciated.
>>
>>


>>
>> Honestly… the problem is that it’s very hard to narrow down. “Back
>> in the day” we might have had a driver and a (native) C/C++ test app in one
>> solution and that’s it. THESE days, that test app MIGHT be C/C++, or it
>> might equally be written in C#, or there could be a service in the Solution
>> (in C/C++ or C#), or it might be some UWP management app (OK, not really
>> the last of those… but someday, maybe).
>>
>> We could be convinced to build the user-mode parts separately (as a
>> separate Solution)… but we still have to build both parts (the driver
>> Solution and the UM Solution) as part of our CI process. So, bottom line,
>> we need to have a full-blown VS (to accommodate the various user-mode
>> options) and the driver part.
>>
>>


>>
>> That’s good news.
>>
>> One approach might be to (a) ship along with the WDK the pieces/parts
>> that are needed to ensure that tools work on Windows Server, (b) provide a
>> container with VS Community and the WDK pre-installed. Extra points for
>> having such a container on Docker Hub. How cool would THAT be??
>>
>> ANYhow… We still plan to give this a try, and see how it works.
>>
>> AND I hope there are others in the community who have tried this, and
>> can share the details of their experiences.
>>
>> Regardless, as I said, we’ll write-up whatever we come up with to help
>> those who come after us.
>>
>> Peter
>> OSR
>> @OSRDrivers
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> Visit the list online at: <
>> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>>
>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
>> software drivers!
>> Details at http:
>>
>> To unsubscribe, visit the List Server section of OSR Online at <
>> http://www.osronline.com/page.cfm?name=ListServer&gt;
>>
></http:>

Oh that is just the ticket. The container works great. Greatest container
ever. Now I have to go convince it to build our stuff.

Thanks Jakob!

Mark Roddy

On Wed, Jun 27, 2018 at 2:26 PM Mark Roddy wrote:

> And I should read the thread to the bottom :-). That docker script looks
> great. Will hack away at it.
> Mark Roddy
>
>
> On Wed, Jun 27, 2018 at 2:20 PM Mark Roddy wrote:
>
>> I looked at the EWDK as a starting point for a containerized build
>> solution and gave up because it was just too huge to be a viable container
>> image and there appears to be no reasonable way to get docker containers to
>> use network share mount points. A msft supported docker image for each
>> released wdk with reasonable size would be a huge win from my perspective.
>>
>> Mark Roddy
>>
>>
>> On Thu, Jun 21, 2018 at 10:15 AM xxxxx@osr.com
>> wrote:
>>
>>> (For those who may not be familiar with Mr. Lichtenberg… he’s the dev
>>> lead for the WDK and related tools, and a LONG time friend of the driver
>>> development community).
>>>
>>> Thank you, Mr. Lichtenberg, for jumping into the conversation. Your
>>> help and insight is always appreciated.
>>>
>>>


>>>
>>> Honestly… the problem is that it’s very hard to narrow down. “Back
>>> in the day” we might have had a driver and a (native) C/C++ test app in one
>>> solution and that’s it. THESE days, that test app MIGHT be C/C++, or it
>>> might equally be written in C#, or there could be a service in the Solution
>>> (in C/C++ or C#), or it might be some UWP management app (OK, not really
>>> the last of those… but someday, maybe).
>>>
>>> We could be convinced to build the user-mode parts separately (as a
>>> separate Solution)… but we still have to build both parts (the driver
>>> Solution and the UM Solution) as part of our CI process. So, bottom line,
>>> we need to have a full-blown VS (to accommodate the various user-mode
>>> options) and the driver part.
>>>
>>>


>>>
>>> That’s good news.
>>>
>>> One approach might be to (a) ship along with the WDK the pieces/parts
>>> that are needed to ensure that tools work on Windows Server, (b) provide a
>>> container with VS Community and the WDK pre-installed. Extra points for
>>> having such a container on Docker Hub. How cool would THAT be??
>>>
>>> ANYhow… We still plan to give this a try, and see how it works.
>>>
>>> AND I hope there are others in the community who have tried this, and
>>> can share the details of their experiences.
>>>
>>> Regardless, as I said, we’ll write-up whatever we come up with to help
>>> those who come after us.
>>>
>>> Peter
>>> OSR
>>> @OSRDrivers
>>>
>>>
>>> —
>>> NTDEV is sponsored by OSR
>>>
>>> Visit the list online at: <
>>> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>>>
>>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
>>> software drivers!
>>> Details at http:
>>>
>>> To unsubscribe, visit the List Server section of OSR Online at <
>>> http://www.osronline.com/page.cfm?name=ListServer&gt;
>>>
>></http:>

So now that I’ve played around with this for a few days, I have to say that
“windows containers” appear to be a moving target on WIn10. Building the
container on a current release works, building on an insider build is a
nightmare of inconsistent results. Once the container is built and
validated I think it is good to go on any supported platform, regardless of
where it was originally generated, but I haven’t verified that.

Of course I could do all of this on S2016 instead, but then I’d have to
deal with nested hyper-visors and that is another nightmare of what works
and what doesn’t…

Mark Roddy

On Fri, Jun 29, 2018 at 3:23 PM Mark Roddy wrote:

> Oh that is just the ticket. The container works great. Greatest container
> ever. Now I have to go convince it to build our stuff.
>
> Thanks Jakob!
>
> Mark Roddy
>
>
> On Wed, Jun 27, 2018 at 2:26 PM Mark Roddy wrote:
>
>> And I should read the thread to the bottom :-). That docker script looks
>> great. Will hack away at it.
>> Mark Roddy
>>
>>
>> On Wed, Jun 27, 2018 at 2:20 PM Mark Roddy wrote:
>>
>>> I looked at the EWDK as a starting point for a containerized build
>>> solution and gave up because it was just too huge to be a viable container
>>> image and there appears to be no reasonable way to get docker containers to
>>> use network share mount points. A msft supported docker image for each
>>> released wdk with reasonable size would be a huge win from my perspective.
>>>
>>> Mark Roddy
>>>
>>>
>>> On Thu, Jun 21, 2018 at 10:15 AM xxxxx@osr.com
>>> wrote:
>>>
>>>> (For those who may not be familiar with Mr. Lichtenberg… he’s the dev
>>>> lead for the WDK and related tools, and a LONG time friend of the driver
>>>> development community).
>>>>
>>>> Thank you, Mr. Lichtenberg, for jumping into the conversation. Your
>>>> help and insight is always appreciated.
>>>>
>>>>


>>>>
>>>> Honestly… the problem is that it’s very hard to narrow down. “Back
>>>> in the day” we might have had a driver and a (native) C/C++ test app in one
>>>> solution and that’s it. THESE days, that test app MIGHT be C/C++, or it
>>>> might equally be written in C#, or there could be a service in the Solution
>>>> (in C/C++ or C#), or it might be some UWP management app (OK, not really
>>>> the last of those… but someday, maybe).
>>>>
>>>> We could be convinced to build the user-mode parts separately (as a
>>>> separate Solution)… but we still have to build both parts (the driver
>>>> Solution and the UM Solution) as part of our CI process. So, bottom line,
>>>> we need to have a full-blown VS (to accommodate the various user-mode
>>>> options) and the driver part.
>>>>
>>>>


>>>>
>>>> That’s good news.
>>>>
>>>> One approach might be to (a) ship along with the WDK the pieces/parts
>>>> that are needed to ensure that tools work on Windows Server, (b) provide a
>>>> container with VS Community and the WDK pre-installed. Extra points for
>>>> having such a container on Docker Hub. How cool would THAT be??
>>>>
>>>> ANYhow… We still plan to give this a try, and see how it works.
>>>>
>>>> AND I hope there are others in the community who have tried this, and
>>>> can share the details of their experiences.
>>>>
>>>> Regardless, as I said, we’ll write-up whatever we come up with to help
>>>> those who come after us.
>>>>
>>>> Peter
>>>> OSR
>>>> @OSRDrivers
>>>>
>>>>
>>>> —
>>>> NTDEV is sponsored by OSR
>>>>
>>>> Visit the list online at: <
>>>> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>>>>
>>>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
>>>> software drivers!
>>>> Details at http:
>>>>
>>>> To unsubscribe, visit the List Server section of OSR Online at <
>>>> http://www.osronline.com/page.cfm?name=ListServer&gt;
>>>>
>>></http:>

Thanks for the follow-up. Mr. Roddy. That’s very cool.

Did you see that there’s a new Windows Server Container image that’s supposed to have “more stuff” in it to support more apps?
https:</https:>

Also… My Friend Google tells me that VS versions PRIOR to VS 2017 can’t be installed in a container (because too much stuff they need is missing, or something)… which would, for us, completely defeat the need for using Containers. If anybody has info on this, please go public.

Having pre-built containerized WDK build environments would be a Very Good Thing.

Peter
OSR
@OSRDrivers

" Please note that for compatibility reasons we recommend running the same
build version for the container host and the container itself. ’

Arghhhhhhh!!!

No. I want my containers isolated from OS dependencies as much as possible.

Mark Roddy

On Tue, Jul 3, 2018 at 1:17 PM xxxxx@osr.com wrote:

> Thanks for the follow-up. Mr. Roddy. That’s very cool.
>
> Did you see that there’s a new Windows Server Container image that’s
> supposed to have “more stuff” in it to support more apps?
> <
> https://blogs.technet.microsoft.com/virtualization/2018/06/27/insider-preview-windows-container-image/
> >
>
> Also… My Friend Google tells me that VS versions PRIOR to VS 2017 can’t
> be installed in a container (because too much stuff they need is missing,
> or something)… which would, for us, completely defeat the need for using
> Containers. If anybody has info on this, please go public.
>
> Having pre-built containerized WDK build environments would be a Very Good
> Thing.
>
> Peter
> OSR
> @OSRDrivers
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: <
> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

Silly me. Here, all along, I thought that was the point of containers.

Check this out:

https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility

So… really… how would this work? I need to have a host system that?s the newest rev of windows im order to host a container that?s on the newest rev of windows. Even if I?m using Hyper-V isolation.

That feels… impractical.

Or, you know… perhaps I?m missing something fundamental. I?m not ?Mr. Container? by any means.

Peter
OSR
@OSRDrivers

No it is a huge wtf. The point of my endeavors was to have this all tied
into a git server that enables distributed CI/CD (gitlab bitbucket etc)
such that there is no need for some giant server farm someplace with some
huge queue of build jobs on it, instead the build system can be distributed
to any available system that can host the desired docker image, like you
know your own dev system. And if by chance you have to build stuff for both
linux and windows wow, it’s all good, you can do that too.

I’ll continue down this path for a while 'cause it really does have to
happen, but …

Mark Roddy

On Tue, Jul 3, 2018 at 5:36 PM xxxxx@osr.com wrote:

>


>
> Silly me. Here, all along, I thought that was the point of containers.
>
> Check this out:
>
>
> https://docs.microsoft.com/en-us/virtualization/windowscontainers/deploy-containers/version-compatibility
>
> So… really… how would this work? I need to have a host system that?s
> the newest rev of windows im order to host a container that?s on the newest
> rev of windows. Even if I?m using Hyper-V isolation.
>
> That feels… impractical.
>
> Or, you know… perhaps I?m missing something fundamental. I?m not ?Mr.
> Container? by any means.
>
> Peter
> OSR
> @OSRDrivers
>
>
>
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: <
> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

Hello @OSRDrivers /Peter, I have managed to install both VS2013 and VS2015 on a windows server docker container. I have also installed a couple of SDKs and WDKs on the container. However, my learning has been that, container is a different beast altogether. If something can be installed on a VM or a physical server, it necessarily does not mean that the same can be installed on a container. For example: just today, i hit an error installing a certain version of WDK that otherwise would install flawlessly on a VM or a physical system.
If you need further details on how i could get these things installed, please let me know and i will be happy to share information about the same.

FYI: i was even able to build our driver projects on this container without issues.

Regards,
Kiran Hegde

Just for pedantic’s sake, “necessarily does not mean” is a very different statement than “does not necessarily mean”.

Just for the sake of pedantry you mean :wink:

Or for the sake of being pedantic, but pedantic cannot be a possessive in English

RE the actual problem, I have been going the other way. Not even trying to maintain isolated environments, but to try to ensure that the projects all build with the latest tools. Of course I have the luxury of needing only to support versions I designate since our software is not for general distribution but only within our firm. That means Windows 10 & server 2016 x64 except for that annoying VS 2019 add-in that still has to be x86 when even our Excel 2010 compatible one can be x64 but I digress