Windows Kernel Writeable and Executable pages

NTDEV
21

I the long distant past, the display driver dynamically generated code for a highly performance optimized BitBlt. It basically looked at the raster operation and pixel format, and generated code that was optimal for one scan line, and then ran this scan line code as many times as needed. Actually, on modern graphics systems, things like shaders I believe dynamically generate code, and if you’re running the software 3D renderer, that code may run on the main CPU instead of being loaded into the GPU.

There ARE some cases where dynamically generated code can significantly improve performance. A hypothetical example. Let’s say you have software RAID code, which needs to xor across 3 to 30 disk drives. It MIGHT be a useful performance optimization to dynamically generate (like at volume mount time) highly optimized code to do the xor on the correct number of drives, although you also could just burn memory and have a different function that was at compile time created for every combination of drives supported. This would trade no dynamic code generation for larger code, which with today’s memory sizes and security objectives, is perhaps a better tradeoff. In the past, the contract between the OS and driver developers has been you could dynamically generate code if needed, and this contract has been slowing changing, such that in Server 2016, dynamic code generation may no longer be allowed.

Unlike Linux, Windows has an expectation of BINARY driver compatibility in a new OS release. This compatibility is not always perfect, sometimes by design, and sometimes due to compatibility bugs. Linux on the other had has a stated policy of not even driver source compatibility between OS kernel releases. Drivers often work with no or minimal changes, but the Linux kernel team makes no major effort to maintain backward compatibility. This is a huge problem for companies that do not want to make public their driver sources. The Linux kernel team can basically declare no writable executable kernel memory, fixup some set of drivers in the kernel tree, and call it a done deal, without having to consider the impact on out of tree drivers. The Linux compatibility policy has the advantage of allowing the kernel to evolve faster, but has the disadvantage of causing third party driver developers arbitrary unscheduled work.

You should also keep in mind, THIS list is made of mostly third party driver developers, who have no control over what Microsoft puts in the kernel. There are some Microsoft folks here, who on occasion give third part developers useful insights, and perhaps take back some of what’s discussed here to discuss with the Windows kernel team. There is absolutely no expectation that anything discussed here will change decisions by the Microsoft kernel team.

It sounds like you, as a third part developer, would like to change some global behavior of the OS, making all executable kernel code read-only. You might technically find a way to do this, but the business question is: if you do this, is your company prepared to absorb the expense of possibility thousands of irate customer support calls, for the small percentage of systems this trick did not work on. Over the years, I’ve done my share of “bending the rules” to make something work, and the size of the risk you take needs to be offset by the size of the potential sales upside. Just by comparison, companies like Tesla would be taking a huge risk to sell a self-driving car, but that risk might be offset by the potential massive sales a self-driving car would have. Let’s say you wrote code that made all executable pages read-only, I’m not optimistic that would give you some unique product that had drastically better sales.

Hopefully this at least partially answers your question about why kernel pages aren’t all just executable and read-only.

Jan

From: on behalf of jc scaly
Reply-To: Windows List
Date: Monday, November 21, 2016 at 1:24 PM
To: Windows List
Subject: RE:[ntdev] Windows Kernel Writeable and Executable pages

I would very much like to leave the “are RWX pages a security flaw” talk out of this post and get back to understand: why are Windows kernel modules allocate and execute code dynamically.

22

Thank you Jan and Ami for your thorough and helpful comments.

Ami - I would attempt to do as you said and report my findings back here.

Jan - for our discussion lets treat this as simply a research I am
conducting in order to understand my previous findings.
I do not wish to force RWX pages to be either RW or RX if they are
legitimately used within the kernel; this is too dangerous for the outcome.
I would like to here more about the Server 2016 changes you’ve stated. Is
there any official demand by Microsoft that states dynamically allocated
code is forbidden? Or are you only taking a theoretical point here?

Thanks!

On Nov 22, 2016 2:43 AM, “Jan Bottorff” wrote:

I the long distant past, the display driver dynamically generated code for
a highly performance optimized BitBlt. It basically looked at the raster
operation and pixel format, and generated code that was optimal for one
scan line, and then ran this scan line code as many times as needed.
Actually, on modern graphics systems, things like shaders I believe
dynamically generate code, and if you’re running the software 3D renderer,
that code may run on the main CPU instead of being loaded into the GPU.

There ARE some cases where dynamically generated code can significantly
improve performance. A hypothetical example. Let’s say you have software
RAID code, which needs to xor across 3 to 30 disk drives. It MIGHT be a
useful performance optimization to dynamically generate (like at volume
mount time) highly optimized code to do the xor on the correct number of
drives, although you also could just burn memory and have a different
function that was at compile time created for every combination of drives
supported. This would trade no dynamic code generation for larger code,
which with today’s memory sizes and security objectives, is perhaps a
better tradeoff. In the past, the contract between the OS and driver
developers has been you could dynamically generate code if needed, and this
contract has been slowing changing, such that in Server 2016, dynamic code
generation may no longer be allowed.

Unlike Linux, Windows has an expectation of BINARY driver compatibility in
a new OS release. This compatibility is not always perfect, sometimes by
design, and sometimes due to compatibility bugs. Linux on the other had has
a stated policy of not even driver source compatibility between OS kernel
releases. Drivers often work with no or minimal changes, but the Linux
kernel team makes no major effort to maintain backward compatibility. This
is a huge problem for companies that do not want to make public their
driver sources. The Linux kernel team can basically declare no writable
executable kernel memory, fixup some set of drivers in the kernel tree, and
call it a done deal, without having to consider the impact on out of tree
drivers. The Linux compatibility policy has the advantage of allowing the
kernel to evolve faster, but has the disadvantage of causing third party
driver developers arbitrary unscheduled work.

You should also keep in mind, THIS list is made of mostly third party
driver developers, who have no control over what Microsoft puts in the
kernel. There are some Microsoft folks here, who on occasion give third
part developers useful insights, and perhaps take back some of what’s
discussed here to discuss with the Windows kernel team. There is absolutely
no expectation that anything discussed here will change decisions by the
Microsoft kernel team.

It sounds like you, as a third part developer, would like to change some
global behavior of the OS, making all executable kernel code read-only. You
might technically find a way to do this, but the business question is: if
you do this, is your company prepared to absorb the expense of possibility
thousands of irate customer support calls, for the small percentage of
systems this trick did not work on. Over the years, I’ve done my share of
“bending the rules” to make something work, and the size of the risk you
take needs to be offset by the size of the potential sales upside. Just by
comparison, companies like Tesla would be taking a huge risk to sell a
self-driving car, but that risk might be offset by the potential massive
sales a self-driving car would have. Let’s say you wrote code that made all
executable pages read-only, I’m not optimistic that would give you some
unique product that had drastically better sales.

Hopefully this at least partially answers your question about why kernel
pages aren’t all just executable and read-only.

Jan

*From: * on behalf of jc scaly <
xxxxx@gmail.com>
*Reply-To: *Windows List
*Date: *Monday, November 21, 2016 at 1:24 PM
*To: *Windows List
*Subject: *RE:[ntdev] Windows Kernel Writeable and Executable pages

I would very much like to leave the “are RWX pages a security flaw” talk
out of this post and get back to understand: why are Windows kernel modules
allocate and execute code dynamically.

—
NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and
software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at <
http://www.osronline.com/page.cfm?name=ListServer&gt;</http:></http:>

23

The driver certification tests for Server 2016 seem to require a driver try hard to not have any write executable memory. Microsoft has stated, but it’s not clear under what conditions they inflexibly enforce (maybe secure boot enabled only? Or maybe only if device guard is enabled?), the requirement that ONLY drivers passing Server 2016 certifications will be allowed to run. Effectively, to have a production release driver for Server 2016 that runs under all configurations, almost all RWE memory mappings need to be gone. I say almost, as they don’t seem to force you to use the newer API MmMapIoSpaceEx to map a device BAR, and the older MmMapIoSpace API has no option to specify noexecute protection (although they might have decided it’s low risk to just force anything mapped with MmMapIoSpace non-executable). The problem basically is MmMapIoSpaceEx is only defined in Win 10 or later, so if you use that API, you driver binary will no longer load on previous OS versions. Lots of us prefer a single binary that still loads on previous versions to some point.

So no, this is not a hypothetical situation, it’s a hard requirement to pass certification tests. This is for drivers being shipped NOW. Some months ago, when I was working on getting a driver to pass the Server 2016 certification, I had to fix all the reasons it had RWX memory, which required adding some flags to allocations, and to things like MDL mapping calls. A few weeks ago, another developer on this list was going through the same process. I expect as drivers get updated for Server 2016, the removal of RWX data will often get applied to older OS version builds too

Microsoft is clearly applying gentle pressure to make developers remove all RWX memory. They didn’t change the OS, potentially breaking your existing RWX allocating driver. I could image at some point in the future (like 5 years from now), Microsoft may decide the risk of RWX is more important than backward compatibility, and will just change the default, and any drivers that didn’t get fixed, and require RWX memory will break. They could also potentially send updates to some sample population of systems, to probe how many would break with RWX disabled, and based on that impact, go forward or back out the change. Like the memory manager could be temporarily be taught to fixup, but log cases where forcing no RWX caused a fault. A lot of people complain that Windows is sending data back to Microsoft, but this is an excellent example where the data feedback can help Microsoft find a balance between needed change and needed backward compatibility.

You mentioned that Linux had no RWX pages. Was this always the case, or was it changed at some point in the OS? Or was it changed over time in drivers?

Jan

From: on behalf of jc scaly
Reply-To: Windows List
Date: Monday, November 21, 2016 at 10:20 PM
To: Windows List
Subject: Re: [ntdev] Windows Kernel Writeable and Executable pages

I would like to here more about the Server 2016 changes you’ve stated. Is there any official demand by Microsoft that states dynamically allocated code is forbidden? Or are you only taking a theoretical point here?

24

Hey Jan,
Thanks again, your information is very helpful.

First of all, I understand why third party drivers introduce RWX to the
system, but why is a clean installation of Windows contain them? ClipSp is
a signed Microsoft driver - and as you said, Microsoft attempts at removing
them.

Maybe a Windows Server 2016 has none of them in a clean install?

About Linux - I have only checked Ubuntu 12.04, but I have an OS
independent script, that takes a dump and outputs all RWX pages. It is here:
https://github.com/scalys7/Virtual-Address-Space-Research

I have read a bit about previous versions had RWX pages, I think it was
either on netBSD or Linux, and that when a patch to transform the system to
W^X occurred later on, all but some space needed for BIOS was successfully
ported.

You can take a dump out of some VM loaded with older kernel version and use
my script to find out.

On Nov 22, 2016 11:40 AM, “Jan Bottorff” wrote:

> The driver certification tests for Server 2016 seem to require a driver
> try hard to not have any write executable memory. Microsoft has stated, but
> it’s not clear under what conditions they inflexibly enforce (maybe secure
> boot enabled only? Or maybe only if device guard is enabled?), the
> requirement that ONLY drivers passing Server 2016 certifications will be
> allowed to run. Effectively, to have a production release driver for Server
> 2016 that runs under all configurations, almost all RWE memory mappings
> need to be gone. I say almost, as they don’t seem to force you to use the
> newer API MmMapIoSpaceEx to map a device BAR, and the older MmMapIoSpace
> API has no option to specify noexecute protection (although they might have
> decided it’s low risk to just force anything mapped with MmMapIoSpace
> non-executable). The problem basically is MmMapIoSpaceEx is only defined
> in Win 10 or later, so if you use that API, you driver binary will no
> longer load on previous OS versions. Lots of us prefer a single binary that
> still loads on previous versions to some point.
>
>
>
> So no, this is not a hypothetical situation, it’s a hard requirement to
> pass certification tests. This is for drivers being shipped NOW. Some
> months ago, when I was working on getting a driver to pass the Server 2016
> certification, I had to fix all the reasons it had RWX memory, which
> required adding some flags to allocations, and to things like MDL mapping
> calls. A few weeks ago, another developer on this list was going through
> the same process. I expect as drivers get updated for Server 2016, the
> removal of RWX data will often get applied to older OS version builds too
>
>
>
> Microsoft is clearly applying gentle pressure to make developers remove
> all RWX memory. They didn’t change the OS, potentially breaking your
> existing RWX allocating driver. I could image at some point in the future
> (like 5 years from now), Microsoft may decide the risk of RWX is more
> important than backward compatibility, and will just change the default,
> and any drivers that didn’t get fixed, and require RWX memory will break.
> They could also potentially send updates to some sample population of
> systems, to probe how many would break with RWX disabled, and based on that
> impact, go forward or back out the change. Like the memory manager could be
> temporarily be taught to fixup, but log cases where forcing no RWX caused a
> fault. A lot of people complain that Windows is sending data back to
> Microsoft, but this is an excellent example where the data feedback can
> help Microsoft find a balance between needed change and needed backward
> compatibility.
>
>
>
> You mentioned that Linux had no RWX pages. Was this always the case, or
> was it changed at some point in the OS? Or was it changed over time in
> drivers?
>
>
>
> Jan
>
>
>
> *From: * on behalf of jc scaly <
> xxxxx@gmail.com>
> *Reply-To: *Windows List
> *Date: *Monday, November 21, 2016 at 10:20 PM
> *To: *Windows List
> *Subject: *Re: [ntdev] Windows Kernel Writeable and Executable pages
>
>
>
> I would like to here more about the Server 2016 changes you’ve stated. Is
> there any official demand by Microsoft that states dynamically allocated
> code is forbidden? Or are you only taking a theoretical point here?
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:> showlists.cfm?list=ntdev>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:></http:>

25

Jonathan,

You wrote “It is truly remarkable to me that I’ve posted a question in the most
relevant forum filled with best Windows kernel researchers out there and
the post has gone so off-topic that my question was left unanswered.”

This is a forum of very skilled windows driver developers. They know and understand IRPs and PnP and MDLs better than anyone. This is not, however, a forum of kernel “researchers”. There are such forums out there (which I don’t think I can link to due to rules), but this is not one of them. So you are talking to the wrong audience (which in many cases is the audience that causes security flaws in drivers which then ends up at security conferences!).

I haven’t had the time to scroll through this entire thread as it’s devolved into offtopic/flame-war, but your initial findings are consistent with Windows behavior, which is why Device Guard was put into place to try to mitigate against.

Another thing you should keep in mind are Large Pages. Because of their 2MB granularity, it’s perfectly normal to find RWX pages all over the place. If you’d like to discuss more, you can send me an email/tweet.

26

>There are such forums out there (which I don’t think I can link to due to rules)…

Just sad, very sad!

27

Seriously?

Maybe it’s because in a purely speculative, academic, question borne out of idle curiosity, other people find other semi-related topics more diverting. So, you know… they discuss those. I mean, if you’re question EVER gets answered, that’s a good thing. But given that you’re not paying our salaries, I’m not sure you should be so entitled as to think the forum owes you an answer on your terms.

AFAIC, Tim Roberts answered your question fully within 20 minutes of its posting. There’s legacy stuff. There are trade-offs. It’s not viewed as a big deal.

For years, non-paged pool was *never* NX, because it all came from the same large page.

It is what it is. You have a specific agenda. Good for you. I’m sure the world will be a better place when all pages are NX that can be NX. Until then, I’ve got code to write.

Peter
OSR
@OSRDrivers

28

D. T. wrote:

It is just a matter of consistency.
The “least privilege” rule is simple: if a privilege is not required than it is not granted.
So, if a region of memory does not contain code, no page in this region should be executable.

Yeah!!! +1 for you D. T.
Good point. The earth has a round shape and it is orbiting the sun. Yeah, that is what it is.

29

> The earth has a round shape and it is orbiting the sun. Yeah, that is what it is.

I am pretty sure the if it were up to MSFT they would try their best to maintain sort of “backwards compatibility” with the belief that the Earth is flat and is rested on top of 3 tortoises. Furthermore, I am pretty sure Mr.Bottorff would be telling us about the “tremendous advantages” of such “compatibility”…

Anton Bassov

30

You do understand that the only reason that anyone says that the earth goes around the sun instead of vise versa is to simplify one?s perceptions by altering one?s frame of reference?

Both of these propositions are equally invalid in Newtonian physics ? and it gets worse the more ?modern? a view one takes. The sun is part a spiral galaxy, and so is itself in relative motion; and the galaxy is part of a cluster etc. etc.

It is entirely possible to accurately calculate the motions of the sun, moon and other celestial bodies using the earth as a fixed point of reference, but for a given level of precision, taking another body as the datum may simplify the calculations considerably.

It is in fact one of the key skills of good programmers to recapitulate problems in terms that are easier to understand and solve, but it does not necessarily make one set of terms more true than another in the same way that a Bessel function is far easier to express in polar coordinates than in Cartesian coordinates, but both expressions are equally true

I have no idea how to help the OP in this thread, but entirely agree that it is nonsense to suggest that the existence of RWX pages in the kernel is an inherent vulnerability.

Sent from Mailhttps: for Windows 10

From: xxxxx@hotmail.commailto:xxxxx
Sent: November 24, 2016 8:33 AM
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: RE:[ntdev] Windows Kernel Writeable and Executable pages

> The earth has a round shape and it is orbiting the sun. Yeah, that is what it is.

I am pretty sure the if it were up to MSFT they would try their best to maintain sort of “backwards compatibility” with the belief that the Earth is flat and is rested on top of 3 tortoises. Furthermore, I am pretty sure Mr.Bottorff would be telling us about the “tremendous advantages” of such “compatibility”…

Anton Bassov

—
NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></mailto:xxxxx></mailto:xxxxx></https:>

31

> There ARE some cases where dynamically generated code can significantly improve performance.

Is this still the case with modern Windows drivers? It seems that a lot of this dynamic code generation can take place at init time when the driver first queries the hardware rather than having chunks of RWX memory floating around during the entire uptime of the driver