Detecting NonPaged Memory

MmBuildMdlForNonPagedPool() only works for locked pages

No. MmBuildMdlForNonPagedPool() only works for non-paged pool. You should not be calling it for an address you received from an usermode appliication, even if you know that area was locked.

ProbeAndLockPages doesn’t protect the usermode address range from being unmapped and remapped. It only locks the physical pages. The usermode app can call VirtualFree for this address, which will unmap the pages.

In short, if you don’t know what an address is, you build an MDL for it, do ProbeAndLockPages, then MmGetMdlSystemAddressSafe.

That doesn’t feel very satisfying
Don’t try to micro-optimize it.

With DV enabled, it bugchecks.

Because you didn’t have a kernel debugger attached. And DV is telling you “You did something wrong.”

Without, it runs without a hint that some of the pfns aren’t worth the memory they were stored in.

Right. The OS doesn’t try to “protect itself” against other kernel-mode callers doing the wrong thing. You’re a big boy, you’re part of t he OS, “do the right thing the right way”

As Mr. Roddy so insightfully wrote:

What real world problem are you trying to solve

I can invent all sorts of potential things I might care about… but in the real-world, these just aren’t problems.

So it is in driver writing.

Peter

MmProbeAndLockPages() can be called at DISPATCH_LEVEL if the memory is nonpaged, else it has to be called at <= APC_LEVEL.
Knowing if a pointer is pinned or not can therefore make the difference between having to pend until safe or not.

Well, the very fact that code running at elevated IRQL may be left wondering whether the memory at the address it accesses is pageable immediately reveals that a driver in question has some serious design flaws. In fact, it indicates that the entire architecture of your driver has to be reworked and redesigned. Don’t forget that pointers that your code receives are not random. Therefore, if a code running at elevated IRQL is left with the possibility of accessing the paged memory, you can be 100% sure that there is something wrong with your driver’s entire architecture…

Anton Bassov

“In device drivers, all pool is non-paged and all locks are spin”

Although I wholeheartedly agree with the former part of the above statement, I would give a serious consideration to the latter one…

Unless critical sections that are guarded by the spinlocks happen to be really short, indiscriminate use of spinlocks may have a very negative impact on your code’s horizontal scalability (which becomes more and more of a concern with the increase of the CPU cores). I don’t know if it is going to tell you something, but Solaris designers went through the whole trouble of introducing interrupt threads and support for priority inheritance exactly for the reason of reducing the number of scenarios that don’t allow any synch options other than spinlocks.
They were the first ones (due to the specifics of SPARC) to notice that mindless use of spinlocks starts biting you more and more and more with the growth of the number of processing cores/threads.

Therefore, unless we are speaking about something really short and trivial ( like, for example, updating just a couple of variables in a critical section) , I would rather go for fast mutexes whenever such an option is available. Otherwise, I would,probably, consider re-designing the code in a way that allows the use of dispatcher-level constructs ( like, for example, deferring the actual processing to the workitem/dedicated thread, rather that doing it right in a DPC routine) whenever it applies…

Anton Bassov

Heed the words of Anton

Well, the very fact that code running at elevated IRQL may be left wondering whether the memory at the address it accesses is pageable immediately reveals that a driver in question has some serious design flaws. In fact, it indicates that the entire architecture of your driver has to be reworked and redesigned. Don’t forget that pointers that your code receives are not random. Therefore, if a code running at elevated IRQL is left with the possibility of accessing the paged memory, you can be 100% sure that there is something wrong with your driver’s entire architecture…

There you go making assumptions again. It turns out that post op callbacks may be called at DPC. Most drivers perform actions, call API functions, access data, that requires <= APC. If that is a sign that all such drivers need to be completely redesigned, then woe is the Windows KM development community. My only sin here was to propose that knowing whether a piece of memory is locked or not can allow us to make better choices that improve performance.

I’m not sure why you feel that you always know where memory comes from. We process IRPs constantly that have memory that may be locked KM memory, or have an MDL that has been locked and mapped to a KM Va, or may be mapped but not locked, or may have an MDL that has not been mapped into the KM virtual address space, or may just provide a user mode buffer. KM drivers have to make a lot of decisions, and provide various methods of handling these IRPs, especially on the post op side. We can check the IRQL, which allows us to make our code more efficient. There are occasions where knowing if the memory described by an MDL and the SystemVa are locked or pageable would allow for better choices to be made.

That said, I did some debugging and found that physical memory appears to be locked by reference count, and since there is no way to know if the reference count is permanent or about to be decremented, so there is no way to know if it is pageable or not.

I also traced the MmIsAddressValid() code and found that it is not the pariah that I’ve heard it described as. It simply traces the page tables checking each level’s Valid bit (summarizing). If that is called at DPC, then you know it cannot be paged out, so you are guaranteed that the memory can be accessed without causing a page fault.

_Ron

We process IRPs constantly that have memory that may be locked KM memory, or have an MDL that has been locked and mapped to a KM Va, or may be mapped but not locked, or may have an MDL that has not been mapped into the KM virtual address space, or may just provide a user mode buffer

Really? In actual, practical, Windows drivers? This hasn’t been my experience. Or, to be more precise, some of these details are hidden from a properly written driver. Consider, for example, MmGetSystemAdressForMdlSafe:

FORCEINLINE
PVOID
MmGetSystemAddressForMdlSafe (
    _Inout_ PMDL Mdl,
    _In_    ULONG Priority  // MM_PAGE_PRIORITY logically OR'd with MdlMapping*
    )
{
    if (Mdl->MdlFlags & (MDL_MAPPED_TO_SYSTEM_VA | MDL_SOURCE_IS_NONPAGED_POOL)) {
        return Mdl->MappedSystemVa;
    } else {
        return MmMapLockedPagesSpecifyCache(Mdl, KernelMode, MmCached,   
                                            NULL, FALSE, Priority);
    }
}

This is an example of how Windows “smooths over” differences such as “already mapped vs not yet mapped” for you, so you don’t have to worry about such things.

physical memory appears to be locked by reference count

Yes, the count is in the PFN (the Page Frame Number Database).

If that is called at DPC, then you know it cannot be paged out

Hmmmm… are you SURE? How about on a Multiprocessor system?

Mr. Roddy asked, some days ago:

What real world problem are you trying to solve?

And so, I will ask again, because your questions seem to reflect an abstract view of what might occur in a driver, as opposed to the day-to-day reality of writing Windows drivers. Please, tell us… What real world problem are you trying to solve?

Peter

rstruempf wrote:

I’m not sure why you feel that you always know where memory comes from.

Because, in almost every driver class, you do.

We process IRPs constantly that have memory that may be locked KM memory, or have an MDL that has been locked and mapped to a KM Va, or may be mapped but not locked, or may have an MDL that has not been mapped into the KM virtual address space, or may just provide a user mode buffer.

I’ve been writing Windows drivers for 28 years.  Like Peter, I’ve never
encountered a driver situation where I did not know exactly what the
memory was, and it’s hard for me to imagine one that wouldn’t.

Most drivers perform actions, call API functions, access data, that requires <= APC.

…which does not necessarily imply that these drivers actually pass pointers to pageable memory to functions that may possibly be called at elevated IRQL, does it…

If that is a sign that all such drivers need to be completely redesigned, then woe is the Windows KM development community.

However, whenever they do something like that, this can be, indeed, taken as a clear indication that these drivers need a complete rewrite/redesign…

We process IRPs constantly that have memory that may be locked KM memory, or have an MDL that has been locked
and mapped to a KM Va, or may be mapped but not locked, or may have an MDL that has not been mapped into
the KM virtual address space, or may just provide a user mode buffer.

Well, “mapped-but-not-locked” and “locked-but-not-mapped” MDLs are very clear indicators of a buggy code, which may or may not be
accidental. However, if a drivers processes a UM buffer (i.e METHOD_NEITHER IRP) in context of anything,apart from the originating thread, it can be taken as a clear indication that a driver writer has absolutely no clue about KM programming…

Anton Bassov

@Peter_Viscarola said:

We process IRPs constantly that have memory that may be locked KM memory, or have an MDL that has been locked and mapped to a KM Va, or may be mapped but not locked, or may have an MDL that has not been mapped into the KM virtual address space, or may just provide a user mode buffer

Really? In actual, practical, Windows drivers? This hasn’t been my experience. Or, to be more precise, some of these details are hidden from a properly written driver. Consider, for example, MmGetSystemAdressForMdlSafe:

That does appear to take care of some of the possible paths; thank you for the look at the source.

My statement, however, was in response to your assertion that in well written drivers, we know where the memory we work with comes from. A quick look at the SwapBuffers driver shows that it must handle 3 or 4 different paths due to the many possible variations in the IO buffers being worked with.

If that is called at DPC, then you know it cannot be paged out

Hmmmm… are you SURE? How about on a Multiprocessor system?

Thanks for straightening me out on that. Bad mistake on my part.

Mr. Roddy asked, some days ago:

What real world problem are you trying to solve?

And so, I will ask again, because your questions seem to reflect an abstract view of what might occur in a driver, as opposed to the day-to-day reality of writing Windows drivers. Please, tell us… What real world problem are you trying to solve?

My problem was solved on day 1 when you told me that there is no way to tell whether a block of memory is pageable. That meant that I had to probe and lock it myself.

The continued discussion is not intended to solve a problem, but to help me better understand how the kernel operates. In particular, there were a number of assertions that in well written drivers, there is never a question about where the memory we work with comes from, so no one would ever want to know if the memory was pageable.

_Ron

I’ve been writing Windows drivers for 28 years. Like Peter, I’ve never
encountered a driver situation where I did not know exactly what the
memory was, and it’s hard for me to imagine one that wouldn’t.

I don’t understand that statement. Every IRP processed by my drivers has one or more buffers that have been handed to it by some other code, and I certainly do not know the source of it, particularly whether it is pageable or not. Am I missing something?

_Ron

Well, “mapped-but-not-locked” and “locked-but-not-mapped” MDLs are very clear indicators of a buggy code, which may or may not be
accidental. However, if a drivers processes a UM buffer (i.e METHOD_NEITHER IRP) in context of anything,apart from the originating thread, it can be taken as a clear indication that a driver writer has absolutely no clue about KM programming…

To be clear, you are saying that in a well designed driver, that you ALWAYS know whether the IO buffers of the IRP are paged or nonpaged?

_Ron

On Sep 29, 2018, at 3:42 PM, rstruempf wrote:
>
>> I’ve been writing Windows drivers for 28 years. Like Peter, I’ve never
>> encountered a driver situation where I did not know exactly what the
>> memory was, and it’s hard for me to imagine one that wouldn’t.
>
> I don’t understand that statement. Every IRP processed by my drivers has one or more buffers that have been handed to it by some other code, and I certainly do not know the source of it, particularly whether it is pageable or not. Am I missing something?

Well, I can’t say “never”, of course. I’m just saying it’s hard to imagine. If you’re a top-level driver, you’re getting IRPs from user mode that are either buffered or direct, and the rules on those are quite explicit. Or, you’re getting IRPs from a kernel driver that is impersonating user-mode, and again that impersonation means they are following the rules.

In an intermediate driver, getting driver-to-driver communication, they’re either impersonating direct I/O, or there is a documented contract.

Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

I don’t understand that statement. Every IRP processed by my drivers has one or more buffers that have been handed to it
by some other code, and I certainly do not know the source of it, particularly whether it is pageable or not. Am I missing something?

When it comes to IRPs, the IO Manager has a set of well-defined rules…

There are 3 possible types of IO:

1.METHOD_BUFFERED. In this case you access the system buffer, which resides in non-pageable memory
2.METHOD_DIRECT. In this case you deal with MDL that has been mapped and locked in RAM

Therefore, you always receive pointers to non-pageable memory in your IRPs when dealing with above IO methods, which is ensured by the IO Manager. Taking into consideration that you may receive an IRP at DISPATCH_LEVEL, it makes a perfect sense,don’t you think. Therefore, you don’t have to worry about it in your driver.

3.METHOD_NEITHER. In this case you are dealing with a user buffer that you receive directly from the caller. Therefore, it may be pageable
( and even reside in the userland). In this case it is your responsibility to probe/ lock and map pages if you are planning to process this IRP in context of anything, apart from the originating thread. In practice, you would normally use this method only when dealing with a tightly -coupled app/driver pair (i.e.when your driver is designed for the needs of some particular app)

To be clear, you are saying that in a well designed driver, that you ALWAYS know whether the IO buffers of the IRP are paged or nonpaged?

As you can see it yourself, this is, indeed, the case…

Anton Bassov

thank you for the look at the source

It’s from WDM.H, you know…

A quick look at the SwapBuffers driver shows that it must handle 3 or 4 different paths…

Dude… the SwapBuffers example is a File System Minifilter and was written (many years ago) specifically to demonstrate several instances of buffer manipulation. It is in no way, shape, manner, or form, representative of what drivers that are NOT File System Minifilters need to deal with. Heck, I’d argue that it’s not even representative of File System Minifilters. But that’s a discussion for another day, and another Forum.

To be clear, you are saying that in a well designed driver, that you ALWAYS know whether the IO buffers of the IRP are paged or nonpaged

Yes, assuming we read “ALWAYS know” to mean “can typically make assumptions about” – then, yes.

Am I missing something?

Yes, I think so. Something very important, in fact.

Carefully read what @Tim_Roberts said above, which is 100% correct and really cuts to the heart of the matter. That was good insight from Mr. Roberts, as usual.

There are established conventions – which are effectively contracts – in every branch of the device tree. If you’re a filter over the Volume driver, you’re not all of a sudden going to get a buffer that’s not described by Direct I/O, and if it’s using Direct I/O then the buffer has been pinned. Full stop. CAN some driver allocate a buffer from paged pool and then send it to you, without an MDL already having been built? Sure… But that’s a fatal bug in that other driver and not a situation about which you have to worry.

And see what @anton_bassov wrote (helpfully), too. If you’re the first driver entered from the I/O Manager, and you’re processing requests from user-mode, the I/O Manager will ensure your contract for Direct, Buffered, or Neither I/O is met. And, just so we’re clear, let’s take Neither I/O away from consideration… because unless you’re a file system or filter or doing something exceedingly special, this is not something a well-written driver will tangle with. If some OTHER driver in the system sends you requests that do not comply with your contract, _that’s a fatal bug in that other driver _and not a situation about which you have to worry.

So… CAN buffers arrive at your driver in all states and all manner of ways? Yes. Is this a situation with which we typically have to deal in writing Windows drivers? No. It would be mighty tiresome, and super prone to error (not to mention, demanding of several hundred lines of repeated boilerplate code) if we could make no assumptions about the buffers arriving at our drivers entry points. It would take, oh, something like ten minutes, for the owner of the I/O Subsystem to say to themselves “Gee, this is kinda chaotic… Perhaps I should write some centralized code to smooth this shit out?”

Peter

@anton_bassov said:
When it comes to IRPs, the IO Manager has a set of well-defined rules…

There are 3 possible types of IO:

1.METHOD_BUFFERED. In this case you access the system buffer, which resides in non-pageable memory
2.METHOD_DIRECT. In this case you deal with MDL that has been mapped and locked in RAM

Therefore, you always receive pointers to non-pageable memory in your IRPs when dealing with above IO methods, which is ensured by the IO Manager. Taking into consideration that you may receive an IRP at DISPATCH_LEVEL, it makes a perfect sense,don’t you think. Therefore, you don’t have to worry about it in your driver.

3.METHOD_NEITHER. In this case you are dealing with a user buffer that you receive directly from the caller. Therefore, it may be pageable
( and even reside in the userland). In this case it is your responsibility to probe/ lock and map pages if you are planning to process this IRP in context of anything, apart from the originating thread. In practice, you would normally use this method only when dealing with a tightly -coupled app/driver pair (i.e.when your driver is designed for the needs of some particular app)

Thank you, Anton! That was exactly the information I was missing.

I have read all of the documentation I can find, and other than the comments in the example drivers, such as SwapBuffer, I have never found a description of these types of IO. I’m going to scan the Windows Internal books now to see if I can find that in there. Maybe I skimmed a chapter too quickly.

_Ron

@Peter_Viscarola said:
Dude… the SwapBuffers example is a File System Minifilter and was written (many years ago) specifically to demonstrate several instances of buffer manipulation. It is in no way, shape, manner, or form, representative of what drivers that are NOT File System Minifilters need to deal with. Heck, I’d argue that it’s not even representative of File System Minifilters. But that’s a discussion for another day, and another Forum.

I’m actually working on minifilters, but I posted here because I figured memory concerns was a general topic, rather than minifilter specific. I need to find where the information Anton provided is documented, and do a thorough job of reading that source. The MS online documentation are spotty and not at all thorough, and I have not found that finely laid out to the point kind of detail in Windows Internals. Back in the old days, I used to read Richter’s books cover to cover and then hold onto them as a reference, but thus far I have not found that kind of quality documentation for kernel drivers.

_Ron

Yes you are missing something.

The point is that you have to know something about that other code and how it works. It does not work by magic and there is prescriptive set of rules. This means that the odds that you don

I have read all of the documentation I can find, and other than the comments in the example drivers, such as SwapBuffer,
I have never found a description of these types of IO.

Well, I don’t really know where you are looking for info, but MSDN documentation does not seem to be on the list of your sources…

Look - this is, apparently, one of the most basic things that a driver writer has to learn, and, hence, it is thoroughly described on MSDN.

Start from

https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/methods-for-accessing-data-buffers

This doc presents a general overview of IO and provides links to the docs that describe each IO type in more detail.

When it comes to IOCTLs, you may want to look at the following one

https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/defining-i-o-control-codes.

In fact, the best way to go is to start right at https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel , and to go all the way down…

Anton Bassov