virtual address of PXE,PPE,PDE,PTE

OSR_Community_User · January 7, 2013, 4:02pm

Hi gurus,

Background:-

I have a thorny issue in a storport miniport, where a system thread tries to access a buffer which is no longer valid. My research has led me to the virtual memory structures, and the magic !pte command in WinDbg. If I do !pte , I get something like (I’m at home so can’t cut-and-paste?)

PXE at fffff6fd’7bdedf88 PPE at fffff6fd’7e123456 PDE at fffff6fd’7ea12345 PTE at fffff6fd’880abcde
contains 11b6cf863 contains 1234863 contains 2345863 contains 431200
PFN 11b6cf PFN 1234 PFN (…etc).

The PTE for my bad address looks corrupt - the values don’t look like valid PFNs, so I’m trying to catch whatever corrupted it, unmapped the memory, etc?

I understand how the CPU reads this structure, starting at known physical address in CR3, and traversing the PFNs in the four-level PTE structure.
In 32-bit, I understand how to calculate the virtual addresses at all points in the chain, from fixed starting point, and knowing the base of the PDEs.

Question:-
---------------
How do I determine the virtual addresses of the PPE/PDE/PTE tables in 64-bit? (Clearly it’s possible, as the !pte extension can do it).
All I’ve found is the !pfn extension, which just seems to backpoint to the PTE that owns it.

Thanks,
Andy
PS If it helps, I have attended an OSR course some years ago, but no longer have access to the email address I was registered with.

Tim_Roberts · January 7, 2013, 4:34pm

Andy Green wrote:

The PTE for my bad address looks corrupt - the values don’t look like valid PFNs, so I’m trying to catch whatever corrupted it, unmapped the memory, etc…

Do all of the page entries show the “valid” bit?

Question:-

How do I determine the virtual addresses of the PPE/PDE/PTE tables in 64-bit? (Clearly it’s possible, as the !pte extension can do it).
All I’ve found is the !pfn extension, which just seems to backpoint to the PTE that owns it.

For completeness, remember that the page tables do not need to exist in
virtual memory at all. Windows happens to map them, but that’s for
convenience, not because the hardware requires it.

Your dump snippet shows you the virtual addresses of the PXE and PPE.
That should be pretty much all you need.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Calvin_Guan-3 · January 7, 2013, 5:06pm

On Mon, Jan 7, 2013 at 1:01 PM, Andy Green wrote:

> How do I determine the virtual addresses of the PPE/PDE/PTE tables in
> 64-bit?

In case you want to do it by hand, 4K page in amd64 long mode goes like
this:

CR3 points to PML4 physical.
VA64_b63-48: Not used as of 2013.
VA64_b47-39: offset from PML4 , PML4 entries point to PDP physical.
VA64_b38-30: offset from PDP, PDP entries point to PD physical.
VA64_b29-21: offset PD, PD entries point to PT physical.
VA64_b20-12: offset PT, PT entries point to PFN of 4K page.

PAE must have already been enabled.
Calvin

OSR_Community_User · January 7, 2013, 5:14pm

Hi Tim, thanks for the rapid response

On 7 Jan 2013, at 21:33, Tim Roberts wrote:

Andy Green wrote:
> The PTE for my bad address looks corrupt - the values don’t look like valid PFNs, so I’m trying to catch whatever corrupted it, unmapped the memory, etc?

Do all of the page entries show the “valid” bit?

No, there is a 1kB hole in the page table which is mostly zeros but containing a few mysterious quadlet values. I have no idea what this data is or where it came from.

> Question:-
> ---------------
> How do I determine the virtual addresses of the PPE/PDE/PTE tables in 64-bit? (Clearly it’s possible, as the !pte extension can do it).
> All I’ve found is the !pfn extension, which just seems to backpoint to the PTE that owns it.

For completeness, remember that the page tables do not need to exist in
virtual memory at all. Windows happens to map them, but that’s for
convenience, not because the hardware requires it.

Your dump snippet shows you the virtual addresses of the PXE and PPE.
That should be pretty much all you need.

It’s all I need to read the dump output, but I’m wondering whether I can code some forensics into the driver to see if anything touched the page table.

In short, it’s one of those really nasty problems where I don’t know it went wrong until some time after the event.
My storport driver happily processes SRBs, and I’ve introduced some code to log the startIo, the completion and the data pointer (virtual address) used.
At some point, the file system/volume driver tries to access a data buffer (I’m not even in the call stack) at DISPATCH level, but the address is no longer valid (it hasn’t been paged out, the PTE is just garbage)
I can see from my logs that about 15 commands ago, my driver processed an SRB pointing to that buffer, so I’ve added code just before I notify that the SRB is complete, to check MmIsAddressValid(), and it doesn’t trigger, so I guess something is spooning the PTEs after I’ve finished with it, possible one of the next commands.

ho hum!!!

OSR_Community_User · January 7, 2013, 5:19pm

Hi Calvin, thanks.

I know how to break down the address into offsets within page table levels as you describe.
In physical addresses, i understand it from start to finish.
The bit I’m missing is how to get the virtual address of PDP,PD,PT.

On 7 Jan 2013, at 22:05, Calvin Guan (news) wrote:

On Mon, Jan 7, 2013 at 1:01 PM, Andy Green wrote:
> How do I determine the virtual addresses of the PPE/PDE/PTE tables in 64-bit?
>
> In case you want to do it by hand, 4K page in amd64 long mode goes like this:
>
> CR3 points to PML4 physical.
> VA64_b63-48: Not used as of 2013.
> VA64_b47-39: offset from PML4 , PML4 entries point to PDP physical.
> VA64_b38-30: offset from PDP, PDP entries point to PD physical.
> VA64_b29-21: offset PD, PD entries point to PT physical.
> VA64_b20-12: offset PT, PT entries point to PFN of 4K page.
>
> PAE must have already been enabled.
> Calvin
>
>
>
> — NTDEV is sponsored by OSR For our schedule of WDF, WDM, debugging and other seminars visit: http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

–
Please note new email address xxxxx@thegreenho.me.
(The old one xxxxx@ntlworld.com will still get to us)

Pavel_Lebedinsky · January 7, 2013, 10:22pm

What’s the actual output of !pte ?

From: Andy Greenmailto:xxxxx
Sent: ?1/?7/?2013 1:02 PM
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: [ntdev] virtual address of PXE,PPE,PDE,PTE

Hi gurus,

Background:-
-------------------
I have a thorny issue in a storport miniport, where a system thread tries to access a buffer which is no longer valid. My research has led me to the virtual memory structures, and the magic !pte command in WinDbg. If I do !pte , I get something like (I’m at home so can’t cut-and-paste?)

PXE at fffff6fd’7bdedf88 PPE at fffff6fd’7e123456 PDE at fffff6fd’7ea12345 PTE at fffff6fd’880abcde
contains 11b6cf863 contains 1234863 contains 2345863 contains 431200
PFN 11b6cf PFN 1234 PFN (…etc).

The PTE for my bad address looks corrupt - the values don’t look like valid PFNs, so I’m trying to catch whatever corrupted it, unmapped the memory, etc?

I understand how the CPU reads this structure, starting at known physical address in CR3, and traversing the PFNs in the four-level PTE structure.
In 32-bit, I understand how to calculate the virtual addresses at all points in the chain, from fixed starting point, and knowing the base of the PDEs.

Question:-
---------------
How do I determine the virtual addresses of the PPE/PDE/PTE tables in 64-bit? (Clearly it’s possible, as the !pte extension can do it).
All I’ve found is the !pfn extension, which just seems to backpoint to the PTE that owns it.

Thanks,
Andy
PS If it helps, I have attended an OSR course some years ago, but no longer have access to the email address I was registered with.

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx></mailto:xxxxx>

OSR_Community_User · January 8, 2013, 3:28am

> What’s the actual output of !pte ?

0: kd> !pte fffff88873840000
VA fffff88873840000
PXE at FFFFF6FB7DBEDF88 PPE at FFFFF6FB7DBF1108 PDE at FFFFF6FB7E221CE0 PTE at FFFFF6FC4439C200
contains 0000000116F84863 contains 00000000B6654863 contains 00000000B5D04863 contains 0087394000000000
pfn 116f84 —DA–KWEV pfn b6654 —DA–KWEV pfn b5d04 —DA–KWEV not valid
PageFile: 0
Offset: 873940
Protect: 0

0: kd> dq 0xFFFFF6FC4439C200
fffff6fc4439c200 0087394000000000 0000000000000000 fffff6fc4439c210 0000000000000000 0000000000000000
fffff6fc4439c220 0000000000000000 0000000000000000 fffff6fc4439c230 0000000000000000 0000000000000000
fffff6fc4439c240 0000000000000000 0000000000000000 fffff6fc4439c250 0000000000000000 0000000000000000
fffff6fc4439c260 0000000000000000 0000000000000000 fffff6fc4439c270 0000000000000000 0000000000000000
0: kd> dq
fffff6fc4439c280 0000000000000000 0000000000000000 fffff6fc4439c290 0000000000000000 0000000000000000
fffff6fc4439c2a0 0000000000000000 0000000000000000 fffff6fc4439c2b0 0000000000000000 0000000000000000
fffff6fc4439c2c0 0000000000000000 0000000000000000 fffff6fc4439c2d0 0000000000000000 0000000000000000
fffff6fc4439c2e0 0000000000000000 0000000000000000 fffff6fc4439c2f0 0000000000000000 0000000000000000
0: kd> dq
fffff6fc4439c300 0000000000000000 0000000000000000 fffff6fc4439c310 0000000000000000 0000000000000000
fffff6fc4439c320 0000000000000000 0000000000000000 fffff6fc4439c330 0000000000000000 0000000000000000
fffff6fc4439c340 0000000000000000 0000000000000000 fffff6fc4439c350 0000000000000000 0000000000000000
fffff6fc4439c360 0000000000000000 0000000000000000 fffff6fc4439c370 0000000000000000 0000000000000000
0: kd> dq
fffff6fc4439c380 0000000000000000 0000000000000000 fffff6fc4439c390 0000000000000000 0000000000000000
fffff6fc4439c3a0 0000000000000000 0000000000000000 fffff6fc4439c3b0 0000000000000000 0000000000000000
fffff6fc4439c3c0 0000000000000000 0000000000000000 fffff6fc4439c3d0 0000000000000000 0000000000000000
fffff6fc4439c3e0 0000000000000000 0000000000000000 fffff6fc4439c3f0 0000000000000000 0000000000000000
0: kd> dq
fffff6fc4439c400 000000010573e963 00000001052bf963 fffff6fc4439c410 00000001052c0963 0000000105141963
fffff6fc4439c420 00000000c4142963 00000000c60c3963 fffff6fc4439c430 00000000c64c4963 00000000c5cc5963
fffff6fc4439c440 00000000c5d46963 00000000c6047963 fffff6fc4439c450 000000010a248963 00000000c3949963
fffff6fc4439c460 00000000c5cca963 00000000c604b963 fffff6fc4439c470 00000000c484c963 00000000c4bcd963
0: kd> dq
fffff6fc4439c480 00000000c4fce963 00000000c40cf963 fffff6fc4439c490 00000000c4b50963 00000000c4151963
fffff6fc4439c4a0 00000000c4552963 00000000c5553963 fffff6fc4439c4b0 0000000105254963 0000000105055963
fffff6fc4439c4c0 0000000104dd6963 00000001049d7963 fffff6fc4439c4d0 00000000c45d8963 00000000c39d9963
fffff6fc4439c4e0 00000000c3dda963 00000000c415b963 fffff6fc4439c4f0 0000000104d5c963 0000000104e5d963

Pavel_Lebedinsky · January 8, 2013, 5:12am

This PTE encoding (pagefile format with protection bits set to zero) means the page has been freed.

On win8/win7 the 0xfffff88x`xxxxxxxx VA range is used for system PTE mappings. One potential scenario that could lead to a crash like this is accessing the system address for an MDL after it has already been unmapped (by calling MmUnmapLockedPages).

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@thegreenho.me
Sent: Tuesday, January 8, 2013 12:28 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] virtual address of PXE,PPE,PDE,PTE

What’s the actual output of !pte ?

0: kd> !pte fffff88873840000
VA fffff88873840000
PXE at FFFFF6FB7DBEDF88 PPE at FFFFF6FB7DBF1108 PDE at FFFFF6FB7E221CE0 PTE at FFFFF6FC4439C200
contains 0000000116F84863 contains 00000000B6654863 contains 00000000B5D04863 contains 0087394000000000
pfn 116f84 —DA–KWEV pfn b6654 —DA–KWEV pfn b5d04 —DA–KWEV not valid
PageFile: 0
Offset: 873940
Protect: 0

Scott_Noone_OSR · January 8, 2013, 9:36am

Have you enabled Driver Verifier yet? It might catch the issue earlier. I’d
recommend enabling it for at least your mini-port and StorPort.sys.

-scott
OSR

“Pavel Lebedynskiy” wrote in message news:xxxxx@ntdev…

This PTE encoding (pagefile format with protection bits set to zero) means
the page has been freed.

On win8/win7 the 0xfffff88x`xxxxxxxx VA range is used for system PTE
mappings. One potential scenario that could lead to a crash like this is
accessing the system address for an MDL after it has already been unmapped
(by calling MmUnmapLockedPages).

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@thegreenho.me
Sent: Tuesday, January 8, 2013 12:28 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] virtual address of PXE,PPE,PDE,PTE

What’s the actual output of !pte ?

0: kd> !pte fffff88873840000
VA fffff88873840000
PXE at FFFFF6FB7DBEDF88 PPE at FFFFF6FB7DBF1108 PDE at
FFFFF6FB7E221CE0 PTE at FFFFF6FC4439C200
contains 0000000116F84863 contains 00000000B6654863 contains
00000000B5D04863 contains 0087394000000000
pfn 116f84 —DA–KWEV pfn b6654 —DA–KWEV pfn
4 —DA–KWEV not valid

PageFile: 0

Offset: 873940

Protect: 0

Carlo_Cianfarani · January 9, 2013, 5:57am

Not sure to understand what Pavel said…

“This PTE encoding (pagefile format with protection bits set to zero) means the page has been freed. On win8/win7 the 0xfffff88x`xxxxxxxx VA range is used for system PTE mappings. One potential scenario that could lead to a crash like this is accessing the system address for an MDL after it has already been unmapped (by calling MmUnmapLockedPages)”

To me it seems the PTE entry (0087394000000000) refers to a page stored in system pagefile (page file number 0) at offset 873940…Just the protection bits set to zero means the page has been freed ?

Thanks

Pavel_Lebedinsky · January 9, 2013, 2:35pm

> To me it seems the PTE entry (0087394000000000) refers to a page stored in system pagefile (page file number 0) at offset 873940…Just the protection bits set to zero means the page has been freed ?

Yes. Pages that are actually paged out have non-zero protection in the PTE. Zero protection means the page has been freed (if it’s a pool VA) or unmapped (if it’s system PTEs). Accessing such a page results in a PAGE_FAULT_IN_NONPAGED_AREA bugcheck.

Carlo_Cianfarani · January 10, 2013, 5:14am

> Yes. Pages that are actually paged out have non-zero protection in the PTE. Zero protection means the page has been freed (if it’s a pool VA) or unmapped (if it’s system PTEs). Accessing such a page results in a PAGE_FAULT_IN_NONPAGED_AREA bugcheck

From its very name “PAGE_FAULT_IN_NONPAGED_AREA” bugcheck seems to refer to a non paged pre-defined system VA range (searching for it I found http://www.osronline.com/article.cfm?article=335)

Now in the first case (it’s a pool VA) I believe the only possible choice is the “non paged pool” while in the second case (system PTE VAs) the range is “non pageable” by design…

Does it make sense ?

Pavel_Lebedinsky · January 10, 2013, 10:28pm

I’m assuming you are referring to this part of the article:

Whenever the Memory Manager detects a page fault (that is, a failure
to translate a kernel virtual address to a physical address) in one
of its pre-assigned address ranges in which the memory is supposed
to be non-pageable, it halts system execution with a
PAGE_FAULT_IN_NONPAGED_AREA bugcheck.

This is not quite correct. Referencing a freed *paged* pool page will also trigger a PAGE_FAULT_IN_NONPAGED_AREA bugcheck, so pageability of the address range doesn’t matter.

The description provided by !analyze and debugger docs is more accurate:

PAGE_FAULT_IN_NONPAGED_AREA

Invalid system memory was referenced. This cannot be protected by try-except,
it must be protected by a Probe. Typically the address is just plain bad or it
is pointing at freed memory.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@alice.it
Sent: Thursday, January 10, 2013 2:15 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] virtual address of PXE,PPE,PDE,PTE

Yes. Pages that are actually paged out have non-zero protection in the PTE. Zero protection means the page has been freed (if it’s a pool VA) or unmapped (if it’s system PTEs). Accessing such a page results in a PAGE_FAULT_IN_NONPAGED_AREA bugcheck

From its very name “PAGE_FAULT_IN_NONPAGED_AREA” bugcheck seems to refer to a non paged pre-defined system VA range (searching for it I found http://www.osronline.com/article.cfm?article=335)

Now in the first case (it’s a pool VA) I believe the only possible choice is the “non paged pool” while in the second case (system PTE VAs) the range is “non pageable” by design…

Does it make sense ?

Carlo_Cianfarani · January 11, 2013, 5:50am

> This is not quite correct. Referencing a freed *paged* pool page will also trigger a PAGE_FAULT_IN_NONPAGED_AREA bugcheck, so pageability of the address range doesn’t matter.

Just to clarify a point…

As far as i understand it is in charge of an OS (executive) component to manage allocation/free of pool memory chunks (e.g. a paged pool page). When drive code calls ExFreePool/ExFreePoolWithTag the OS manager will free the pool chunk zeroing the related pool header Pool_Type field (see for example http://analyze-v.com/?p=734)

What about the page frame ? Is it moved into (system) free list invalidating the PTE pointing to it (pagefile format with protection bits set to zero) and the entry in the system paged pool working set (I know in Win 7 there exist a system working set dedicated to the paged pool…)

Thanks.

Pavel_Lebedinsky · January 12, 2013, 5:20am

When the last allocation on a page is freed, the physical page may be released immediately or it may be cached for a while. It’s an implementation detail.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@alice.it
Sent: Friday, January 11, 2013 2:50 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] virtual address of PXE,PPE,PDE,PTE

This is not quite correct. Referencing a freed *paged* pool page will also trigger a PAGE_FAULT_IN_NONPAGED_AREA bugcheck, so pageability of the address range doesn’t matter.

Just to clarify a point…

As far as i understand it is in charge of an OS (executive) component to manage allocation/free of pool memory chunks (e.g. a paged pool page). When drive code calls ExFreePool/ExFreePoolWithTag the OS manager will free the pool chunk zeroing the related pool header Pool_Type field (see for example http://analyze-v.com/?p=734)

What about the page frame ? Is it moved into (system) free list invalidating the PTE pointing to it (pagefile format with protection bits set to zero) and the entry in the system paged pool working set (I know in Win 7 there exist a system working set dedicated to the paged pool…)

Carlo_Cianfarani · January 12, 2013, 9:17am

> When the last allocation on a page is freed, the physical page may be released immediately or it may be cached for a while. It’s an implementation detail.

My question arises from a Windows Internals statement (6th edition, part 2 p. 565). There a demo driver (myfault.sys) allocates a page of paged pool, free the pool, raise the IRQL to DPC/dispatch level, and then touch the page it has freed. The first reference to the freed page might not cause a crash…

quoting it:
“The reason the first page reference might not cause a crash is that it won?t generate a page fault if
the page that the driver frees remains in the system working set.”

So it seems when the pool page has been freed at the same time might be not trimmed from system working set (system paged pool working set on Win 7 I guess)…in other words the first time the page is referenced might exist a valid PTE pointing to the physical page…

Is it a feasible scenario ?

Pavel_Lebedinsky · January 12, 2013, 9:56pm

Technically speaking, accessing freed pool is undefined behavior so anything is feasible.

In practice, it is possible that if you free a pool page and then access it you will not bugcheck (even assuming that no one has re-allocated that VA in the meantime).

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@alice.it
Sent: Saturday, January 12, 2013 6:17 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] virtual address of PXE,PPE,PDE,PTE

When the last allocation on a page is freed, the physical page may be released immediately or it may be cached for a while. It’s an implementation detail.

My question arises from a Windows Internals statement (6th edition, part 2 p. 565). There a demo driver (myfault.sys) allocates a page of paged pool, free the pool, raise the IRQL to DPC/dispatch level, and then touch the page it has freed. The first reference to the freed page might not cause a crash…

quoting it:
“The reason the first page reference might not cause a crash is that it won?t generate a page fault if the page that the driver frees remains in the system working set.”

So it seems when the pool page has been freed at the same time might be not trimmed from system working set (system paged pool working set on Win 7 I guess)…in other words the first time the page is referenced might exist a valid PTE pointing to the physical page…

Is it a feasible scenario ?

NTDEV is sponsored by OSR

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer