Just don’t specify that anything is pageable, it’ll all be non-pageable by default, and you’ll be good to go.
Well, I am afraid I may experience “the full power of the new platform’s troll-management functionality” ( BTW, you gave me a free pass for one “inflammatory” post, did not you), but I think that certain NTDEV members are going to be VERY upset by such a suggestion, taking into consideration its author. If it was suggested by someone else we would, probably, have a chance to see some “exciting” reaction (probably, even in capitals), but in this particular case it is made by " the ultimate authority figure", and, hence, cannot be questioned…
Anton Bassov
The best thing it can do is to call MmIsAddressValid() , but even this function is not really reliable, because, as it has been already mentioned on this thread, the situation may change by the time you make an actual use of its return value. According to Doron, this function should have never been exposed to anyone outside MM, in the first place.
That’s the conclusion I have been coming to. I was just hoping. Thanks for your help.
Do you know, does MmIsAddressValid() just tell you whether the page is currently mapped to a physical page (i.e. the PTE Valid bit)?
_Ron
Just don’t specify that anything is pageable, it’ll all be non-pageable by default, and you’ll be good to go.
Problem solved?
Yes, I’m serious.
That is my plan B for this case. I have other cases where I would like to know if a piece of memory is pageable or not though.
For example, and I guess I should have led with this since it drives me nuts that I can’t do it, annotations could be much more descriptive if we could read this simple bit. Possibly a majority of my functions, as well as a great many of those in the Windows kernel API, have to allow DISPATCH_LEVEL in the annotations, but resort to warning the user in the documentation that if this or this or this, then it is <= APC_LEVEL. If there were a low cost check for pageable memory, we could use:
When(KeIsMemoryPageable(Buffer), IRQL_requires_max(APC_LEVEL))
_Ron
Do you know, does MmIsAddressValid() just tell you whether the page is currently mapped to a physical page (i.e. the PTE Valid bit)?
IIRC, it just returns the state of the ‘Present’ bit of the PTE…
BTW, if you think carefully about MmIsAddressValid() you are (hopefully) going to realise that your original question about the “locked” attribute of a page cannot have any reliable positive answer for exactly the same reason that MmIsAddressValid() cannot be reliably used by anyone outside MM. Even if you find a way to read this attribute, the situation may change by the time you try to make any actual use of it, which is totally out of your control…
Anton Bassov
@anton_bassov said:
If it was suggested by someone else we would, probably, have a chance to see some “exciting” reaction (probably, even in capitals), but in this particular case it is made by " the ultimate authority figure", and, hence, cannot be questioned…
Just like King Edward in “A Knight’s Tale”. That’s quite a compliment, Peter. 
_Ron
The point about annotations is that they’re entirely static, not dynamic. So we could never have an annotation that says “if the buffer is pinned, the the IRQL can be DISPATCH_LEVEL.”
I would like to know if a piece of memory is pageable or not
There is no such API available to Windows driver writers. Full stop. We’ve been writing drivers for Windows for almost 25 years now, and we haven’t felt the need for this API yet. If you’re feeling like you need it, I’d suggest perhaps you’re thinking about things in a “non-Windows way” architecturally.
Peter
but I think that certain NTDEV members are going to be VERY upset by such a >suggestion
I am a firm believer in the saying “In device drivers, all pool is non-paged and all locks are spin” — I need an actual reason to make my device driver code or data pageable. And “cuz it could run on a physical system in 100 VMs” is not a sufficient reason to me.
Peter
The point about annotations is that they’re entirely static, not dynamic. So we could never have an annotation that says “if the buffer is pinned, the the IRQL can be DISPATCH_LEVEL.”
So annotations are used only by the static analyzer, not by DV to check values at runtime? I suppose that makes sense, I just never realized it.
_Ron
annotations are used only by the static analyze
By Static Driver Verifier (SDV) and by VS Code Analysis (CA), yes.
Peter
annotations are used only by the static analyze
By Static Driver Verifier (SDV) and by VS Code Analysis (CA), yes.
I suppose that deep down I had to have known that. The annotation macros all expand to nothingness, so it would be unreasonable to think that they would modify the code to check things at runtime.
Can you think of no advantages to knowing if a pointer handed to you is nonpaged? What if you have a pointer and you want an mdl to describe it, but you don’t know if it is pageable? If it is nonpaged pool or locked, you can call MmBuildMdlForNonPagedPool(), if not you have to call MmProbeAndLockPages().
I don’t know what the code looks like, but “build mdl knowing it is non-paged” sounds more efficient than “probing and locking + adding a reference count that needs to be unlocked + building an mdl”. What would you do in this situation? Just call MmProbeAndLockPages() to be safe? Both functions are void, so it’s not like you can call MmBuildMdlForNonPagedPool() and check for an error code.
MmProbeAndLockPages() can be called at DISPATCH_LEVEL if the memory is nonpaged, else it has to be called at <= APC_LEVEL. Knowing if a pointer is pinned or not can therefore make the difference between having to pend until safe or not.
MmBuildMdlForNonPagedPool() only works for locked pages. The documentation doesn’t say what happens if you call it for a range that includes a non-pinned page, but I imagine it is a bugcheck. So knowing if the pointer includes paged memory can avoid a bugcheck.
If I’m way off base, just say so. I’ve got less than a year of kernel development under my belt, and most of what I know comes from you all, and the MS documentation that perpetually falls short. You are the definitive source for information, I’m just looking for something that feels satisfying.
_Ron
On Sep 20, 2018, at 9:20 PM, rstruempf wrote:
>
>
>>> annotations are used only by the static analyze
>>
>> By Static Driver Verifier (SDV) and by VS Code Analysis (CA), yes.
>
> Can you think of no advantages to knowing if a pointer handed to you is nonpaged? What if you have a pointer and you want an mdl to describe it, but you don’t know if it is pageable? If it is nonpaged pool or locked, you can call MmBuildMdlForNonPagedPool(), if not you have to call MmProbeAndLockPages().
>
> I don’t know what the code looks like, but “build mdl knowing it is non-paged” sounds more efficient than “probing and locking + adding a reference count that needs to be unlocked + building an mdl”. What would you do in this situation? Just call MmProbeAndLockPages() to be safe? Both functions are void, so it’s not like you can call MmBuildMdlForNonPagedPool() and check for an error code.
Yes, if you don’t know, then you assume the pointer is paged and call MmProbeAndLockPages.
> MmProbeAndLockPages() can be called at DISPATCH_LEVEL if the memory is nonpaged, else it has to be called at <= APC_LEVEL. Knowing if a pointer is pinned or not can therefore make the difference between having to pend until safe or not.
Yes. If you have the potential to work with paged data, then you cannot run at DISPATCH_LEVEL. It’s just that simple.
In real life, this is not a problem. There are almost never any occasions where a single piece of code will sometimes received paged and sometimes received non-paged address. You virtually always get one or the other, and you had better know which one it is.
—
Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.
In real life, this is not a problem. There are almost never any occasions where a single piece of code will sometimes received paged and sometimes received non-paged address. You virtually always get one or the other, and you had better know which one it is.
That doesn’t feel very satisfying, more like if KeGetCurrentIrql() didn’t exist and being told to always queue to a worker thread just to be safe, but I’ll run with it and see if I ever run across an actual need for this.
Out of curiosity, I just called MmBuildMdlForNonPagedPool() on an address that had some paged sections to see what it does. With DV enabled, it bugchecks. Without, it runs without a hint that some of the pfns aren’t worth the memory they were stored in. I suppose it might throw an exception if it tried to handle a page that was swapped out, but I’m not that curious.
_Ron
What real world problem are you trying to solve? In almost all cases your driver is not dealing with random pointers, it is either passed in an mdl already built by the os for a user buffer, or is building an mdl for nonpaged or paged memory it has either itself allocated or has been delivered from a trusted and documented source that notes what kind of buffer it is.
MmBuildMdlForNonPagedPool() only works for locked pages
No. MmBuildMdlForNonPagedPool() only works for non-paged pool. You should not be calling it for an address you received from an usermode appliication, even if you know that area was locked.
ProbeAndLockPages doesn’t protect the usermode address range from being unmapped and remapped. It only locks the physical pages. The usermode app can call VirtualFree for this address, which will unmap the pages.
In short, if you don’t know what an address is, you build an MDL for it, do ProbeAndLockPages, then MmGetMdlSystemAddressSafe.
With DV enabled, it bugchecks.
Because you didn’t have a kernel debugger attached. And DV is telling you “You did something wrong.”
Without, it runs without a hint that some of the pfns aren’t worth the memory they were stored in.
Right. The OS doesn’t try to “protect itself” against other kernel-mode callers doing the wrong thing. You’re a big boy, you’re part of t he OS, “do the right thing the right way”
As Mr. Roddy so insightfully wrote:
What real world problem are you trying to solve
I can invent all sorts of potential things I might care about… but in the real-world, these just aren’t problems.
So it is in driver writing.
Peter
MmProbeAndLockPages() can be called at DISPATCH_LEVEL if the memory is nonpaged, else it has to be called at <= APC_LEVEL.
Knowing if a pointer is pinned or not can therefore make the difference between having to pend until safe or not.
Well, the very fact that code running at elevated IRQL may be left wondering whether the memory at the address it accesses is pageable immediately reveals that a driver in question has some serious design flaws. In fact, it indicates that the entire architecture of your driver has to be reworked and redesigned. Don’t forget that pointers that your code receives are not random. Therefore, if a code running at elevated IRQL is left with the possibility of accessing the paged memory, you can be 100% sure that there is something wrong with your driver’s entire architecture…
Anton Bassov
“In device drivers, all pool is non-paged and all locks are spin”
Although I wholeheartedly agree with the former part of the above statement, I would give a serious consideration to the latter one…
Unless critical sections that are guarded by the spinlocks happen to be really short, indiscriminate use of spinlocks may have a very negative impact on your code’s horizontal scalability (which becomes more and more of a concern with the increase of the CPU cores). I don’t know if it is going to tell you something, but Solaris designers went through the whole trouble of introducing interrupt threads and support for priority inheritance exactly for the reason of reducing the number of scenarios that don’t allow any synch options other than spinlocks.
They were the first ones (due to the specifics of SPARC) to notice that mindless use of spinlocks starts biting you more and more and more with the growth of the number of processing cores/threads.
Therefore, unless we are speaking about something really short and trivial ( like, for example, updating just a couple of variables in a critical section) , I would rather go for fast mutexes whenever such an option is available. Otherwise, I would,probably, consider re-designing the code in a way that allows the use of dispatcher-level constructs ( like, for example, deferring the actual processing to the workitem/dedicated thread, rather that doing it right in a DPC routine) whenever it applies…
Anton Bassov
Well, the very fact that code running at elevated IRQL may be left wondering whether the memory at the address it accesses is pageable immediately reveals that a driver in question has some serious design flaws. In fact, it indicates that the entire architecture of your driver has to be reworked and redesigned. Don’t forget that pointers that your code receives are not random. Therefore, if a code running at elevated IRQL is left with the possibility of accessing the paged memory, you can be 100% sure that there is something wrong with your driver’s entire architecture…
There you go making assumptions again. It turns out that post op callbacks may be called at DPC. Most drivers perform actions, call API functions, access data, that requires <= APC. If that is a sign that all such drivers need to be completely redesigned, then woe is the Windows KM development community. My only sin here was to propose that knowing whether a piece of memory is locked or not can allow us to make better choices that improve performance.
I’m not sure why you feel that you always know where memory comes from. We process IRPs constantly that have memory that may be locked KM memory, or have an MDL that has been locked and mapped to a KM Va, or may be mapped but not locked, or may have an MDL that has not been mapped into the KM virtual address space, or may just provide a user mode buffer. KM drivers have to make a lot of decisions, and provide various methods of handling these IRPs, especially on the post op side. We can check the IRQL, which allows us to make our code more efficient. There are occasions where knowing if the memory described by an MDL and the SystemVa are locked or pageable would allow for better choices to be made.
That said, I did some debugging and found that physical memory appears to be locked by reference count, and since there is no way to know if the reference count is permanent or about to be decremented, so there is no way to know if it is pageable or not.
I also traced the MmIsAddressValid() code and found that it is not the pariah that I’ve heard it described as. It simply traces the page tables checking each level’s Valid bit (summarizing). If that is called at DPC, then you know it cannot be paged out, so you are guaranteed that the memory can be accessed without causing a page fault.
_Ron