Hi All,
When i got a bug check D1, which is irql_not_less_or_equal, i found out i was using a wdfobjectdelete to delete a DMA object in a function that runs in irql dispatch level.because the wdfobjectdelete should be used in passive level thread. i had a doubt not with respect to above mentioned scenario. when read in some msdn blogs namely The NDIS blog, there he had mention like memory manager runs at irql 0, if the memory referenced is not in RAM, the memory manager will get a exception and it will page-in the memory from disk to RAM and return to cpu. He says that when access from function running at higher irql access a memory which is paged out, when the memory get an exception the memory manager code cannot preempt the function with higher irql so it will give a bug check. say when accessing the page-able memory from a function running in higher IRQL and at the time of access if the memory referenced resides in RAM will it give a bug-check? if the answer is yes then why it is not happening when we access non-paged memory because all address translation happens through memory manager? if this is simple question forgive me am newbie?
IRQL_NOT_LESS_OR_EQUAL also happens when you access an invalid (not just pageable) memory location at DISPATCH_LEVEL.
I think that you’re confusing two parts of the memory manager. One part maintains page tables. Those page tables (at least the ones for non-pageable memory) are not themselves pageable. They are always resident. Thus memory translation can occur even without paging.
Another part of the memory manager swaps pages in and out of memory. That part depends on code that needs the dispatcher. If you trigger a page fault in your code, your thread needs to be unscheduled until the storage stacks can bring that memory back from the page file.
- Jake Oshins
Windows Kernel Team
Microsoft
-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@gmail.com
Sent: Friday, November 8, 2013 3:51 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] IRQL_NOT_LESS_OR_EQUAL
Hi All,
When i got a bug check D1, which is irql_not_less_or_equal, i found out i was using a wdfobjectdelete to delete a DMA object in a function that runs in irql dispatch level.because the wdfobjectdelete should be used in passive level thread. i had a doubt not with respect to above mentioned scenario. when read in some msdn blogs namely The NDIS blog, there he had mention like memory manager runs at irql 0, if the memory referenced is not in RAM, the memory manager will get a exception and it will page-in the memory from disk to RAM and return to cpu. He says that when access from function running at higher irql access a memory which is paged out, when the memory get an exception the memory manager code cannot preempt the function with higher irql so it will give a bug check. say when accessing the page-able memory from a function running in higher IRQL and at the time of access if the memory referenced resides in RAM will it give a bug-check? if the answer is yes then why it is not happening when we access non-paged memory because all address translation happens through memory manager? if this is simple question forgive me am newbie?
NTDEV is sponsored by OSR
Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
xxxxx@gmail.com wrote:
say when accessing the page-able memory from a function running in higher IRQL and at the time of access if the memory referenced resides in RAM will it give a bug-check?
No.
if the answer is yes then why it is not happening when we access non-paged memory because all address translation happens through memory manager?
The issue is not whether the memory is pageable or not. The issue is
whether the memory is PRESENT or not. When pageable memory is paged
out, it is written to the swap file, and is no longer present in
memory. When you try to access such an address, that causes a “page
fault” exception, which the operating system handles by reading the page
back into memory. With non-paged memory, the memory is ALWAYS present.
A page fault cannot occur.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
> Hi All,
When i got a bug check D1, which is irql_not_less_or_equal, i found
out i was using a wdfobjectdelete to delete a DMA object in a
function that runs in irql dispatch level.because the
wdfobjectdelete should be used in passive level thread. i had a
doubt not with respect to above mentioned scenario. when read in
some msdn blogs namely The NDIS blog, there he had mention like
memory manager runs at irql 0, if the memory referenced is not in
RAM, the memory manager will get a exception and it will page-in
the memory from disk to RAM and return to cpu. He says that when
access from function running at higher irql access a memory which
is paged out, when the memory get an exception the memory manager
code cannot preempt the function with higher irql so it will give a
bug check. say when accessing the page-able memory from a function
running in higher IRQL and at the time of access if the memory
referenced resides in RAM will it give a bug-check? if the answer
is yes then why it is not happening when we access non-paged memory
because all address translation happens through memory manager?
I don’t even understand what this question means. There are two concepts
here: the Memory Manager, that allocates pages in memory for various
purposes, and will move pageable pages out to the paging file if it needs
to create space for other pageable pages.
Then there is the address translation hardware. This maps virtual
addresses generated by the CPU to physical addresses. It has nothing to
do with the Memory Manager. But the Memory Manager has everything to do
with it. The address translation hardware is managed by the Memory
Manager. It does this by manipulating CPU registers to point to the
appropriate memory map, and managing the contents of the memory maps. So
when a page is moved out to disk, the Memory Manager changes the entry in
the memory map to indicate this page is no longer present. When a page
fault happens, the Memory Manager changes the entry in the memory map to
indicate that the page is now present, and where it is.
So the BSOD trap is what happens when the translation hardware detects
that the page whose virtual address has been presented is either a
completely non-existent page, or a page which had once been present but
currently is marked as paged out. For nonpaged pool, the pages can never
be paged out, so those entries in the page table are never changed by the
Memory Manager once they are created. If you call ExAllocatePool and it
sees there is not enough nonpaged pool to allocate, it will try to cause
some pageable pages to move out to create available page frames sufficient
to satisfy the allocation; the page tables are updated to point to this
memory. These entries will never change again.
Address translation does not happen through the memory manager. Address
translation happens in the hardware. The role of the memory manager is to
control the maps used by that hardware.
if
this is simple question forgive me am newbie?
Non-paged memory is always in memory. It is never paged. Therefore, you
cannot take an access fault when you try to read or write it (well, you
can get an access violation if you try to write a write-protected page,
but that’s not a crucial issue here).
The problem with touching pageable memory is that it may or may not be
present. If it is present, the access works perfectly. If it has been
paged out, you get a fault. If this fault happens at elevated IRQL, this
produces the BSOD. So you test the driver on your 64GB machine with
absolutely nothing else running, just by accident, the page you touch is
always in memory because there was no reason to page it out. But on a
customer’s server, running SQL Server, IIS, and handling all the email
traffic, on a 16GB machine, there’s a lot of pressure on memory, and
eventually, that page you were successfully touching in your lab has been
paged out. BSOD time.
Now, in Driver Verifier, there is an option which, when the IRQL is
raised, removes all pageable pages from the memory map. So if you touch
any pageable page, you will get a BSOD in your lab.
This is why there are functions like MmProbeAndLockPages (MmPALP). This
function “promotes” pages temporarily to be nonpaged. They are locked
into memory at whatever addresses they were at; any pages, as described by
the MDL, that had been paged out will page fault and be paged in (MmPALP
can only be called at PASSIVE_LEVEL).
So you have several options:
Use buffered I/O. The buffer is allocated in the kernel, from the
nonpaged pool. It will require a copy to/from user space, but this is
managed for you and you don’t care about it. It’s magic.
Use direct I/O. The pages of the user buffer, which were pageable, are
locked in with MmPALP, and they remain nonpageable until the IRP is
completed, at which point they become pageable again.
Use Neither mode I/O. You are responsible for creating the MDLs, using
MmPALP yourself, etc., and therefore it is your responsibility to make
sure the pages you are touching are locked into memory. And unlocked when
you are done with them.
You may not, under any circumstances, touch a pageable page which has not
been locked down, from elevated IRQL. This is going to (eventually)
result in a BSOD. If your driver has an error that allows it to touch
pageable memory, turning on the DV option to make all pageable memory
disappear at elevated IRQL, will make your error show up in a
deterministic way, and early.
joe
NTDEV is sponsored by OSR
Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
Hi Jake,
if the storage stacks needs to be brought back. will it use a file system driver to bring it back? if yes the file system driver runs in passive level and can the scheduler preempt the current thread for the file system driver with low irql to run?
Hi Joseph,
Then there is the address translation hardware. This maps virtual
addresses generated by the CPU to physical addresses. It has nothing to
do with the Memory Manager. But the Memory Manager has everything to do
with it. The address translation hardware is managed by the Memory
Manager. It does this by manipulating CPU registers to point to the
appropriate memory map, and managing the contents of the memory maps. So
when a page is moved out to disk, “the Memory Manager changes the entry in
the memory map to indicate this page is no longer present. When a page
fault happens, the Memory Manager changes the entry in the memory map to
indicate that the page is now present, and where it is”.
for the memory manager to update the mapping it involves a software, so the space for the memory manager software to run will be provided by preempting the dispatch level thread ? is it right, if yes then the memory manager software should run at higher irql than current thread?
And one more thing is the BSOD we are getting when we enable DV is because of the page fault. when enable the driver verifier it will make the page-able memory disappear and it will give bug-check when we access page-able memory address without even checking whether it is present in ram or not?
suggest me some articles for learning about memory manager if possible?
Code that runs on DISPATCH_LEVEL cannot be preempted (other than interrupted by an ISR). It cannot call any wait functions that would cause the thread to be de-scheduled.
Because of that, it’s only allowed to use spin-wait (spinlocks). Any operations that would cause the scheduler to de-schedule it are not allowed.
Reading a page from the pagefile causes the thread to be put in waiting state, and thus is not allowed while at DISPATCH_LEVEL.
Hi Alex,
Now am clear about that, but my doubt is if the code is trying to access a memory which needs a translation.
It cannot make this dispatch level thread to wait and get the page back to RAM.is it right?
> It cannot make this dispatch level thread to wait
No for sure.
No dispatch level code can be “made to wait”.
–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com
>if the code is trying to access a memory which needs a translation
You’re confusing the hardware concept of virtual->Physical address translation (performed by MMU) and software concept of pagefile-supported virtual memory.
> Hi Joseph,
Then there is the address translation hardware. This maps
virtual
addresses generated by the CPU to physical addresses. It has nothing to
do with the Memory Manager. But the Memory Manager has everything to do
with it. The address translation hardware is managed by the Memory
Manager. It does this by manipulating CPU registers to point to the
appropriate memory map, and managing the contents of the memory maps. So
when a page is moved out to disk, “the Memory Manager changes the entry in
the memory map to indicate this page is no longer present. When a page
fault happens, the Memory Manager changes the entry in the memory map to
indicate that the page is now present, and where it is”.for the memory manager to update the mapping it involves a software, so
the space for the memory manager software to run will be provided by
preempting the dispatch level thread ?
This doesn’t even make sense as a question. Last week, one of my 3rd
graders said the length a calculator was 5 ounces. Saying that thread
preemption “creates space” is about as weird and nonsensical.
For the memory manager to update the mapping requires some operations on
the memory map; anything from a simple store to allocating some space to
hold a new submap (the memory map is hierarchical). This has nothing to
do with preemption of a thread.
The memory manager can do this even if called from DISPATCH_LEVEL (such as
might occur for ExAllocatePool…() calls, adding memory to the
NONPAGEDPOOL).
But, when a page fault is taken, the running thread must have its entire
context suspended until the page arrives, after which the scheduler is
poked to restart the thread. And a thread running at DISPATCH_LEVEL
cannot be suspended; it is beyond the scheduler’s reach.
is it right, if yes then the memory
manager software should run at higher irql than current thread?
Irrelevant. The current thread is suspended on a fault, and therefore it
does not matter what its IRQL is (as long as it is < DISPATCH_LEVEL).
And one more thing is the BSOD we are getting when we enable DV is because
of the page fault. when enable the driver verifier it will make the
page-able memory disappear and it will give bug-check when we access
page-able memory address without even checking whether it is present in
ram or not?
That’s right. If you turn that option on, it SIMULATES having all
pageable memory mapped out, just to trap those kinds of errors. The point
is that it is not SUPPOSED to “check”; the memory map entries for all
pageable pages are temporarily marked as “page absent”, and there IS no
way to “check”; the only authority on the matter, the page table, says “it
is not here”.
suggest me some articles for learning about memory manager if possible?
Try the “Inside Windows” book(s). I don’t recall how much they discuss
the memory manager.
joe
NTDEV is sponsored by OSR
Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
OSR is HIRING!! See http://www.osr.com/careers
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
Hi joseph,
if a current thread (i have read that say when a interrupt occurs if a thread is executing the program counter will be updated with the offset of isr blindly storing the currently executing thread context and after acknowledging interrupt the thread continues.is this right?
> if a current thread (
Exception does not suspend a thread, it just transfers control.
–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com
xxxxx@gmail.com wrote:
if a current thread (
As Max said, an exception does not cause a thread to be suspended. It
merely jumps to an exception handler.
> if a thread running at dispatch level (i know it cant be pre-empted) is suspended, due to some exception how it will be re-scheduled as the scheduler can’t schedule a thread with irql at dispatch level?
If the processor is at dispatch level when an exception occurs, you will
get a blue screen and the system will halt.
> say in scenario when running a dispatch level thread and the cpu gets an interrupt after servicing the interrupt how the execution of dispatch level thread will be continued?
Interrupt processing is done by the hardware. That’s not “scheduling”
as the kernel uses the term. The live registers are temporarily saved
while the ISR runs. When the ISR returns, those saved registers are
restored and the thread continues, unaware of any interruption.
> i have read that say when a interrupt occurs if a thread is executing the program counter will be updated with the offset of isr blindly storing the currently executing thread context and after acknowledging interrupt the thread continues.is this right?
More or less. An interrupt causes a jump to the interrupt handler. It
is up to the interrupt handler to save enough of the thread’s state that
it can be restarted when the ISR is complete. Note that this is not
“scheduling”. This is all the hardware interrupt processing.
–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.
If a DISPATCH_LEVEL thread takes an exception, it is not “suspended”.
What you get is a BSOD. So the situation can never arise. You will never
see a thread being suspended as you describe. Note that an “interrupt” is
not the same as “suspend”. The interrupt does not suspend te thread, it
“hijacks” it and runs in the context of that thread. The thread is
“resumed” when the RTI (Return From Interrupt) instruction is executed.
No concept of “suspend” (in the scheduler sense) is involved.
When an interrupt occurs, the Flags register and {R, E}IP is pushed onto
the stack. RTI restores them. For complete details, download the free
Intel manuals from the Intel web site.
joe
Hi joseph,
if a current thread (> exception. it will be re-scheduled by the scheduler. if a
> thread running at dispatch level (i know it cant be
> pre-empted) is suspended, due to some exception how it will
> be re-scheduled as the scheduler can’t schedule a thread with
> irql at dispatch level? say in scenario when running a
> dispatch level thread and the cpu gets an interrupt after
> servicing the interrupt how the execution of dispatch level
> thread will be continued?
> i have read that say when a interrupt occurs if a thread is executing the
> program counter will be updated with the offset of isr blindly storing
> the currently executing thread context and after acknowledging interrupt
> the thread continues.is this right?
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> i have read that say when a interrupt occurs if a thread is executing the program counter will be
updated with the offset of isr
Not only this.
At this event (totally handled by CPU hardware), the thread should also enter (or re-enter) kernel mode. Some values are pushed to the kernel stack during this.
The PC (x86 world uses the wording of Instruction Pointer instead) is updated to the kernel-mode interrupt prolog.
It pushes the rest of the necessary structure to the kernel stack, then raises the IRQL, then calls HalBeginSystemInterrupt (to do the APIC work necessary before your ISR), then calls your ISR, then calls HalEndSystemInterrupt, and then lowers the IRQL. Then it returns to previous CPU mode using IRET.
Reschedule can occur during the “lowers the IRQL” part of the picture.
–
Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com
In reading this answer and your original question, I realize that you have
another massive misconception that needs to be dismissed.
You seem to think that an “exception” will cause the thread to be
descheduled. Nothing could be further from the truth. Divide by zero is
an exception. Access fault (of an illegal address) is an exception. And
page fault (an access fault where the page table marks the frame as
“absent”) is an exception. There’s a ton of other exception cases; read
the Intel manuals. I can’t think of one that would cause a thread to be
suspended, EXCEPT the page fault exception.
Here’s the exception table, from page 172 of my copy of the Intel manual
3A (System programming)
0 #DE Divide by zero
1 #DB RESERVED Fault
2 — NMI Interrupt Interrupt
3 #BP Breakpoint Trap
4 #OF Overflow Trap
5 #BR BOUND Range Exceeded Fault
6 #UD Invalid Opcode
7 #NM Device Not Available (No Math Coprocessor)
8 #DF Double Fault
9 — Coprocessor Segment Overrun (reserved)
10 #TS Invalid TSS
11 #NP Segment Not Present
12 #SS Stack-Segment
13 #GP General Protection
14 #PF Page Fault
15 — (Intel reserved. Do not use.)
16 #MF x87 FPU Floating-Point Error
17 #AC Alignment Check
18 #MC Machine Check
19 #XM SIMD Floating-Point Exception
20 #VE Virtualization Exception Fault No EPT violations6
21-31 — Intel reserved. Do not use.
32-255 — User Defined (Non-reserved)
For most of the exceptions, the exception is simply reflected back to the
app using the built-in exception mechanism, and the app responds. There
isn’t even a change in the timeslice context. It all just is part of the
execution of the thread. Only in the case of a page fault does the thread
need to be suspended; the thread is suspended, the page is brought in
(possibly by writing some modified page out), and the thread is resumed.
A few of the faults are going to be BSOD. But the notion that “exception”
== “suspend” is completely incorrect.
joe
If a DISPATCH_LEVEL thread takes an exception, it is not “suspended”.
What you get is a BSOD. So the situation can never arise. You will
never
see a thread being suspended as you describe. Note that an “interrupt”
is
not the same as “suspend”. The interrupt does not suspend te thread, it
“hijacks” it and runs in the context of that thread. The thread is
“resumed” when the RTI (Return From Interrupt) instruction is executed.
No concept of “suspend” (in the scheduler sense) is involved.When an interrupt occurs, the Flags register and {R, E}IP is pushed onto
the stack. RTI restores them. For complete details, download the free
Intel manuals from the Intel web site.
joe> Hi joseph,
> if a current thread (>> an
>> exception. it will be re-scheduled by the scheduler. if a
>> thread running at dispatch level (i know it cant be
>> pre-empted) is suspended, due to some exception how it will
>> be re-scheduled as the scheduler can’t schedule a thread with
>> irql at dispatch level? say in scenario when running a
>> dispatch level thread and the cpu gets an interrupt after
>> servicing the interrupt how the execution of dispatch level
>> thread will be continued?
>> i have read that say when a interrupt occurs if a thread is executing the
>> program counter will be updated with the offset of isr blindly storing
the currently executing thread context and after acknowledging
interrupt
>> the thread continues.is this right?
>> —
>> NTDEV is sponsored by OSR
>> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
OSR is HIRING!! See http://www.osr.com/careers
>> For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
Hi All,
Thanks for the information you all have shared, the confusion i got was how the processing will be when a page fault occurs in a dispatch level thread(i know it won’t occur as it won’t access page-able memory) because i don’t what part of handling done by hardware and what part is done by software. Now am clear about after reading intel manuals 3a and some memory management articles. Special thanks to joseph.