Windows System Software -- Consulting, Training, Development -- Unique Expertise, Guaranteed Results

Home NTDEV

Before Posting...

Please check out the Community Guidelines in the Announcements and Administration Category.

More Info on Driver Writing and Debugging


The free OSR Learning Library has more than 50 articles on a wide variety of topics about writing and debugging device drivers and Minifilters. From introductory level to advanced. All the articles have been recently reviewed and updated, and are written using the clear and definitive style you've come to expect from OSR over the years.


Check out The OSR Learning Library at: https://www.osr.com/osr-learning-library/


Are page tables paged out?

OSR_Community_UserOSR_Community_User Member Posts: 110,217
Hello, Developers!

I have a question that is not directly related to driver development,
but to internal workings on NT memory manager.

Nagar Rejeev in his book (First Edition) states several times
that Page Frame Database is allocated from non-paged pool [1].
(Page 201, last paragraph and page 211, last paragraph)

But on page 212, last paragraph he says
"To avoid consuming this significant amount of memory
for translation information, page tables are also paged in
and out of memory".[2]

This leaves me with 2 questions:
1. If [1] is correct how non-paged memory
can be paged out? Does memory manager have
some internal mechnism to page out non-paged memory?

2. If [2] is correct and some process accesses page that was swapped
to disk, will I/O requests be generated not only to bring accessed page
to RAM, but also to bring other pages to memory (pages containing
page table)?

Regards, Dennis



__________________________________________________
Do You Yahoo!?
Talk to your friends online with Yahoo! Messenger.
http://im.yahoo.com

Comments

  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    Dennis,

    You appear to be a little bit confused.

    The Page Fram Database has NOTHING to do with Page Tables. The PFN
    database is a list of Physical Memory Pages. Therefore, this is
    not a case of non-paged memory being paged out. I assume that the
    PFN database can't be paged (although I suppose theoretically you
    could do a desing that allowed it even though that might not make sense).

    Process Page Table can indeed be paged out if need be. The process
    of translating a memory reference can generate a page fault to fault
    in the needed page or even the needed page for a page table entry to
    continue the translation process. Obviously this means that a page
    reference could either fault in the desired pages and/or the page
    table entries needed to find the page to fault in. Swapping has
    nothing really to do with this (directly).


    Rick Cadruvi...


    From: SMTP%"[email protected]" 1-APR-2000 14:20:31.83
    To: "NT Developers Interest List"
    CC:
    Subj: [ntdev] Are page tables paged out?

    Hello, Developers!

    I have a question that is not directly related to driver development,
    but to internal workings on NT memory manager.

    Nagar Rejeev in his book (First Edition) states several times
    that Page Frame Database is allocated from non-paged pool [1].
    (Page 201, last paragraph and page 211, last paragraph)

    But on page 212, last paragraph he says
    "To avoid consuming this significant amount of memory
    for translation information, page tables are also paged in=20
    and out of memory".[2]

    This leaves me with 2 questions:
    1. If [1] is correct how non-paged memory=20
    can be paged out? Does memory manager have
    some internal mechnism to page out non-paged memory?

    2. If [2] is correct and some process accesses page that was swapped
    to disk, will I/O requests be generated not only to bring accessed page
    to RAM, but also to bring other pages to memory (pages containing
    page table)?

    Regards, Dennis




    __________________________________________________

    Do You Yahoo!?

    Talk to your friends online with Yahoo! Messenger.

    http://im.yahoo.com


    ---
    You are currently subscribed to ntdev as: [email protected]
    To unsubscribe send a blank email to $subst('Email.Unsub')
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    Rick,
    Thanks on your explanation. I was really confused by
    page table and page frame database. I have to admit that
    Nagar's descrition on this topic is not completelly clear.

    I now have another question:
    Does it makes any sence to page out page table and
    causing multiple I/O requests for serving single page fault.
    It seem not to be very cheap.

    Regars.

    > -----Original Message-----
    > From: [email protected]
    > [mailto:[email protected]]On Behalf Of [email protected]
    > Sent: Sunday, April 02, 2000 12:22 AM
    > To: NT Developers Interest List
    > Subject: [ntdev] Re: Are page tables paged out?
    >
    >
    > Dennis,
    >
    > You appear to be a little bit confused.
    >
    > The Page Fram Database has NOTHING to do with Page Tables. The PFN
    > database is a list of Physical Memory Pages. Therefore, this is
    > not a case of non-paged memory being paged out. I assume that the
    > PFN database can't be paged (although I suppose theoretically you
    > could do a desing that allowed it even though that might not make sense).
    >
    > Process Page Table can indeed be paged out if need be. The process
    > of translating a memory reference can generate a page fault to fault
    > in the needed page or even the needed page for a page table entry to
    > continue the translation process. Obviously this means that a page
    > reference could either fault in the desired pages and/or the page
    > table entries needed to find the page to fault in. Swapping has
    > nothing really to do with this (directly).
    >
    >
    > Rick Cadruvi...
    >
    >
    > From: SMTP%"[email protected]" 1-APR-2000 14:20:31.83
    > To: "NT Developers Interest List" <[email protected]>
    > CC:
    > Subj: [ntdev] Are page tables paged out?
    >
    > Hello, Developers!
    >
    > I have a question that is not directly related to driver development,
    > but to internal workings on NT memory manager.
    >
    > Nagar Rejeev in his book (First Edition) states several times
    > that Page Frame Database is allocated from non-paged pool [1].
    > (Page 201, last paragraph and page 211, last paragraph)
    >
    > But on page 212, last paragraph he says
    > "To avoid consuming this significant amount of memory
    > for translation information, page tables are also paged in=20
    > and out of memory".[2]
    >
    > This leaves me with 2 questions:
    > 1. If [1] is correct how non-paged memory=20
    > can be paged out? Does memory manager have
    > some internal mechnism to page out non-paged memory?
    >
    > 2. If [2] is correct and some process accesses page that was swapped
    > to disk, will I/O requests be generated not only to bring accessed page
    > to RAM, but also to bring other pages to memory (pages containing
    > page table)?
    >
    > Regards, Dennis
    >
    >
    >
    >
    > __________________________________________________
    >
    > Do You Yahoo!?
    >
    > Talk to your friends online with Yahoo! Messenger.
    >
    > http://im.yahoo.com
    >
    >
    > ---
    > You are currently subscribed to ntdev as: [email protected]
    > To unsubscribe send a blank email to $subst('Email.Unsub')
    >
    >
    > ---
    > You are currently subscribed to ntdev as: [email protected]
    > To unsubscribe send a blank email to $subst('Email.Unsub')
    >


    __________________________________________________
    Do You Yahoo!?
    Talk to your friends online with Yahoo! Messenger.
    http://im.yahoo.com
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    >Does it makes any sence to page out page table and
    >causing multiple I/O requests for serving single page fault.
    >It seem not to be very cheap.

    The page table page can be paged out only if all PTEs in it are not present.
    So - such events are rather rare.

    Max
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    Really? So in other words, a page in the page table can only be paged out
    if all of the pages that it describes have been paged out themselves? i.e.
    the page tables are the last thing to get paged out?

    At 12:37 PM 4/2/00 +0400, you wrote:
    > >Does it makes any sence to page out page table and
    > >causing multiple I/O requests for serving single page fault.
    > >It seem not to be very cheap.
    >
    >The page table page can be paged out only if all PTEs in it are not present.
    >So - such events are rather rare.
    >
    > Max
    >
    >
    >---
    >You are currently subscribed to ntdev as: [email protected]
    >To unsubscribe send a blank email to $subst('Email.Unsub')
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    > Really? So in other words, a page in the page table can only be paged out
    > if all of the pages that it describes have been paged out themselves?
    i.e.
    > the page tables are the last thing to get paged out?

    Certainly. And can you imagine any other ways of pageable page tables?
    When a PTE is made valid, the reference count for the page table containing
    the PTE is incremented. And vice versa. Page can be swapped out only if
    refcount == 0.

    Max
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    All right, color me stupid, but why is it exactly that a page cannot be
    paged out if its reference count is not zero? Wouldn't the reference count
    be the number of valid references to this particular virtual->physical
    mapping? Wouldn't a zero count indicate that the particular mapping, if it
    is pageable, should be discarded? I thought, and perhaps I am very mistaken
    here, that pageable virtual->physical mappings can always be paged out as
    long as they are not explicitly 'pinned down', as in MmProbeAndLockPages.

    > -----Original Message-----
    > From: Maxim S. Shatskih [mailto:[email protected]]
    > Sent: Monday, April 03, 2000 3:42 AM
    > To: NT Developers Interest List
    > Subject: [ntdev] Re: Are page tables paged out?
    >
    >
    > > Really? So in other words, a page in the page table can
    > only be paged out
    > > if all of the pages that it describes have been paged out
    > themselves?
    > i.e.
    > > the page tables are the last thing to get paged out?
    >
    > Certainly. And can you imagine any other ways of pageable page tables?
    > When a PTE is made valid, the reference count for the page
    > table containing
    > the PTE is incremented. And vice versa. Page can be swapped
    > out only if
    > refcount == 0.
    >
    > Max
    >
    >
    >
    > ---
    > You are currently subscribed to ntdev as: [email protected]
    > To unsubscribe send a blank email to $subst('Email.Unsub')
    >
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    Pages are paged out on an LRU basis. If in fact the page table has not been
    used recently, then neither have all of its associated pages (well this is
    almost true.) Paging out the page table indeed does make sense in this case.


    While the initial translation (in a series of same-page address operations)
    is expensive (up to two page faults, always two translations,) the
    translation lookaside buffer hardware makes the subsequent translations
    optimal, at least for a while.

    Given the size of physical memory there really isn't any other practical way
    to organize virtual memory other than an n-tier directory approach. If you
    don't want to clog up non-paged pool with unused intermediate translation
    directories, then you need also to make your page tables pageable.

    > -----Original Message-----
    > From: Dennis [mailto:[email protected]]
    > Sent: Saturday, April 01, 2000 5:48 PM
    > To: NT Developers Interest List
    > Subject: [ntdev] Re: Are page tables paged out?
    >
    >
    > Rick,
    > Thanks on your explanation. I was really confused by
    > page table and page frame database. I have to admit that
    > Nagar's descrition on this topic is not completelly clear.
    >
    > I now have another question:
    > Does it makes any sence to page out page table and
    > causing multiple I/O requests for serving single page fault.
    > It seem not to be very cheap.
    >
    > Regars.
    >
    > > -----Original Message-----
    > > From: [email protected]
    > > [mailto:[email protected]]On Behalf Of [email protected]
    > > Sent: Sunday, April 02, 2000 12:22 AM
    > > To: NT Developers Interest List
    > > Subject: [ntdev] Re: Are page tables paged out?
    > >
    > >
    > > Dennis,
    > >
    > > You appear to be a little bit confused.
    > >
    > > The Page Fram Database has NOTHING to do with Page Tables. The PFN
    > > database is a list of Physical Memory Pages. Therefore, this is
    > > not a case of non-paged memory being paged out. I assume that the
    > > PFN database can't be paged (although I suppose theoretically you
    > > could do a desing that allowed it even though that might
    > not make sense).
    > >
    > > Process Page Table can indeed be paged out if need be. The process
    > > of translating a memory reference can generate a page fault to fault
    > > in the needed page or even the needed page for a page table entry to
    > > continue the translation process. Obviously this means that a page
    > > reference could either fault in the desired pages and/or the page
    > > table entries needed to find the page to fault in. Swapping has
    > > nothing really to do with this (directly).
    > >
    > >
    > > Rick Cadruvi...
    > >
    > >
    > > From: SMTP%"[email protected]" 1-APR-2000 14:20:31.83
    > > To: "NT Developers Interest List" <[email protected]>
    > > CC:
    > > Subj: [ntdev] Are page tables paged out?
    > >
    > > Hello, Developers!
    > >
    > > I have a question that is not directly related to driver
    > development,
    > > but to internal workings on NT memory manager.
    > >
    > > Nagar Rejeev in his book (First Edition) states several times
    > > that Page Frame Database is allocated from non-paged pool [1].
    > > (Page 201, last paragraph and page 211, last paragraph)
    > >
    > > But on page 212, last paragraph he says
    > > "To avoid consuming this significant amount of memory
    > > for translation information, page tables are also paged in=20
    > > and out of memory".[2]
    > >
    > > This leaves me with 2 questions:
    > > 1. If [1] is correct how non-paged memory=20
    > > can be paged out? Does memory manager have
    > > some internal mechnism to page out non-paged memory?
    > >
    > > 2. If [2] is correct and some process accesses page that was swapped
    > > to disk, will I/O requests be generated not only to bring
    > accessed page
    > > to RAM, but also to bring other pages to memory (pages containing
    > > page table)?
    > >
    > > Regards, Dennis
    > >
    > >
    > >
    > >
    > > __________________________________________________
    > >
    > > Do You Yahoo!?
    > >
    > > Talk to your friends online with Yahoo! Messenger.
    > >
    > > http://im.yahoo.com
    > >
    > >
    > > ---
    > > You are currently subscribed to ntdev as: [email protected]
    > > To unsubscribe send a blank email to $subst('Email.Unsub')
    > >
    > >
    > > ---
    > > You are currently subscribed to ntdev as: [email protected]
    > > To unsubscribe send a blank email to $subst('Email.Unsub')
    > >
    >
    >
    >
    > __________________________________________________
    >
    > Do You Yahoo!?
    >
    > Talk to your friends online with Yahoo! Messenger.
    >
    > http://im.yahoo.com
    >
    >
    > ---
    > You are currently subscribed to ntdev as: [email protected]
    > To unsubscribe send a blank email to $subst('Email.Unsub')
    >
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    Please forgive any errors in this description. It has been a long
    time since I studied this and I am doing this off the top of my head.

    Page table entries are hardware dependent. In fact, it would be possible
    (I think MIPS is this way) to NOT have a hardware PTE. The hardware
    implementations that I have looked at tdo not have "reference counts"
    in the PTEs. They would effectively be meaningless. They would have a "valid"
    bit as well as some kind of "dirty" bit to indicate that there was a recent
    write to the page so that page cleanup can make sure that section files
    get written out to the disk before pages are removed.

    The reference count referred to is probably the one in the Page Frame
    Database. This reference count is used to "lock" down pages. The PFN
    database is an OS construct and has NOTHING to do with the hardware
    except that it helps the OS track physical memory pages. By OS definition,
    a page with a non-zero reference count in it's PFN database entry is NOT
    pageable. This is essentially how MmProbeAndLockPages locks pages.
    It's a different mechanism, but similar that is used to lock pages within
    a working set. A process has a working set list that works in a similar
    manner to the PFN database to provide for working set trimming.

    I would imagine that what Max was saying is that when a page mapped by a
    PTE is paged in, the reference count for the page containing the PTE gets
    incremented to prevent it's being paged. I am not certain that this happens,
    but it certainly makes sense. In theory, however, there would be no reason
    why an OS couldn't choose to page PTE pages without paging the pages that
    they map. It wouldn't make sense to do this since a page fault would
    need to be rendered the next time a reference happened. Therefore, it makes
    sense to only page out a PTE page when all the pages it maps are paged out.
    I suppose that if you had some pages mapped by a PTE that were locked
    (probably for a DMA transfer), you might decide to page the PTE page
    since ALL the pages it maps cannot be paged. I wouldn't even be surprised
    if that happens in the case of a severe memory resource crisis. It certainly
    would be okay in theory to do.


    just my opinion based aon a potentially faulty recall mechanism. The old
    brain is not quite what it used to be.


    Rick Cadruvi...


    >All right, color me stupid, but why is it exactly that a page cannot be
    >paged out if its reference count is not zero? Wouldn't the reference count
    >be the number of valid references to this particular virtual->physical
    >mapping? Wouldn't a zero count indicate that the particular mapping, if it
    >is pageable, should be discarded? I thought, and perhaps I am very mistaken
    >here, that pageable virtual->physical mappings can always be paged out as
    >long as they are not explicitly 'pinned down', as in MmProbeAndLockPages.
    >
    >> -----Original Message-----
    >> From: Maxim S. Shatskih [mailto:[email protected]]
    >> Sent: Monday, April 03, 2000 3:42 AM
    >> To: NT Developers Interest List
    >> Subject: [ntdev] Re: Are page tables paged out?
    >>
    >>
    >> > Really? So in other words, a page in the page table can
    >> only be paged out
    >> > if all of the pages that it describes have been paged out
    >> themselves?
    >> i.e.
    >> > the page tables are the last thing to get paged out?
    >>
    >> Certainly. And can you imagine any other ways of pageable page tables?
    >> When a PTE is made valid, the reference count for the page
    >> table containing
    >> the PTE is incremented. And vice versa. Page can be swapped
    >> out only if
    >> refcount == 0.
    >>
    >> Max
    >>
    >>
    >>
    >> ---
    >> You are currently subscribed to ntdev as: [email protected]
    >> To unsubscribe send a blank email to $subst('Email.Unsub')
    >>
    >
    >---
    >You are currently subscribed to ntdev as: [email protected]
    >To unsubscribe send a blank email to $subst('Email.Unsub')
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    Sure I agree, but I'm still having trouble with Max saying:

    "When a PTE is made valid, the reference count for the page table containing
    the PTE is incremented. And vice versa. "

    The reference on the Page Table might or might not be OS MM policy, but he
    is saying here, unless I'm misreading, that the reference count on the PTE
    is also incremented, and that would make all PTE's effectively non-pageable.


    > -----Original Message-----
    > From: [email protected] [mailto:[email protected]]
    > Sent: Monday, April 03, 2000 11:13 AM
    > To: NT Developers Interest List
    > Subject: [ntdev] Re: Are page tables paged out?
    >
    >
    > Please forgive any errors in this description. It has been a long
    > time since I studied this and I am doing this off the top of my head.
    >
    > Page table entries are hardware dependent. In fact, it would
    > be possible
    > (I think MIPS is this way) to NOT have a hardware PTE. The hardware
    > implementations that I have looked at tdo not have "reference counts"
    > in the PTEs. They would effectively be meaningless. They
    > would have a "valid"
    > bit as well as some kind of "dirty" bit to indicate that
    > there was a recent
    > write to the page so that page cleanup can make sure that
    > section files
    > get written out to the disk before pages are removed.
    >
    > The reference count referred to is probably the one in the Page Frame
    > Database. This reference count is used to "lock" down pages. The PFN
    > database is an OS construct and has NOTHING to do with the hardware
    > except that it helps the OS track physical memory pages. By
    > OS definition,
    > a page with a non-zero reference count in it's PFN database
    > entry is NOT
    > pageable. This is essentially how MmProbeAndLockPages locks pages.
    > It's a different mechanism, but similar that is used to lock
    > pages within
    > a working set. A process has a working set list that works
    > in a similar
    > manner to the PFN database to provide for working set trimming.
    >
    > I would imagine that what Max was saying is that when a page
    > mapped by a
    > PTE is paged in, the reference count for the page containing
    > the PTE gets
    > incremented to prevent it's being paged. I am not certain
    > that this happens,
    > but it certainly makes sense. In theory, however, there
    > would be no reason
    > why an OS couldn't choose to page PTE pages without paging
    > the pages that
    > they map. It wouldn't make sense to do this since a page fault would
    > need to be rendered the next time a reference happened.
    > Therefore, it makes
    > sense to only page out a PTE page when all the pages it maps
    > are paged out.
    > I suppose that if you had some pages mapped by a PTE that were locked
    > (probably for a DMA transfer), you might decide to page the PTE page
    > since ALL the pages it maps cannot be paged. I wouldn't even
    > be surprised
    > if that happens in the case of a severe memory resource
    > crisis. It certainly
    > would be okay in theory to do.
    >
    >
    > just my opinion based aon a potentially faulty recall
    > mechanism. The old
    > brain is not quite what it used to be.
    >
    >
    > Rick Cadruvi...
    >
    >
    > >All right, color me stupid, but why is it exactly that a
    > page cannot be
    > >paged out if its reference count is not zero? Wouldn't the
    > reference count
    > >be the number of valid references to this particular
    > virtual->physical
    > >mapping? Wouldn't a zero count indicate that the particular
    > mapping, if it
    > >is pageable, should be discarded? I thought, and perhaps I
    > am very mistaken
    > >here, that pageable virtual->physical mappings can always be
    > paged out as
    > >long as they are not explicitly 'pinned down', as in
    > MmProbeAndLockPages.
    > >
    > >> -----Original Message-----
    > >> From: Maxim S. Shatskih [mailto:[email protected]]
    > >> Sent: Monday, April 03, 2000 3:42 AM
    > >> To: NT Developers Interest List
    > >> Subject: [ntdev] Re: Are page tables paged out?
    > >>
    > >>
    > >> > Really? So in other words, a page in the page table can
    > >> only be paged out
    > >> > if all of the pages that it describes have been paged out
    > >> themselves?
    > >> i.e.
    > >> > the page tables are the last thing to get paged out?
    > >>
    > >> Certainly. And can you imagine any other ways of pageable
    > page tables?
    > >> When a PTE is made valid, the reference count for the page
    > >> table containing
    > >> the PTE is incremented. And vice versa. Page can be swapped
    > >> out only if
    > >> refcount == 0.
    > >>
    > >> Max
    > >>
    > >>
    > >>
    > >> ---
    > >> You are currently subscribed to ntdev as: [email protected]
    > >> To unsubscribe send a blank email to
    > $subst('Email.Unsub')
    > >>
    > >
    > >---
    > >You are currently subscribed to ntdev as: [email protected]com
    > >To unsubscribe send a blank email to $subst('Email.Unsub')
    >
    > ---
    > You are currently subscribed to ntdev as: [email protected]
    > To unsubscribe send a blank email to $subst('Email.Unsub')
    >
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    I think Max refers to shared pages only. These pages have a count reference
    because actually they are not actual pages but pointers to prototype entry
    pages.
    The pages are not discarded except if its reference (in the Page Frame
    Database) counter is zero.
    However this has nothing to do with normal, non shared pages.

    Inaki.

    > -----Original Message-----
    > From: Roddy, Mark
    > Sent: lunes 3 de abril de 2000 20:02
    > To: NT Developers Interest List
    > Subject: [ntdev] Re: Are page tables paged out?
    >
    > Sure I agree, but I'm still having trouble with Max saying:
    >
    > "When a PTE is made valid, the reference count for the page table
    > containing
    > the PTE is incremented. And vice versa. "
    >
    > The reference on the Page Table might or might not be OS MM policy, but he
    > is saying here, unless I'm misreading, that the reference count on the PTE
    > is also incremented, and that would make all PTE's effectively
    > non-pageable.
    >
    >
    > > -----Original Message-----
    > > From: [email protected] [mailto:[email protected]]
    > > Sent: Monday, April 03, 2000 11:13 AM
    > > To: NT Developers Interest List
    > > Subject: [ntdev] Re: Are page tables paged out?
    > >
    > >
    > > Please forgive any errors in this description. It has been a long
    > > time since I studied this and I am doing this off the top of my head.
    > >
    > > Page table entries are hardware dependent. In fact, it would
    > > be possible
    > > (I think MIPS is this way) to NOT have a hardware PTE. The hardware
    > > implementations that I have looked at tdo not have "reference counts"
    > > in the PTEs. They would effectively be meaningless. They
    > > would have a "valid"
    > > bit as well as some kind of "dirty" bit to indicate that
    > > there was a recent
    > > write to the page so that page cleanup can make sure that
    > > section files
    > > get written out to the disk before pages are removed.
    > >
    > > The reference count referred to is probably the one in the Page Frame
    > > Database. This reference count is used to "lock" down pages. The PFN
    > > database is an OS construct and has NOTHING to do with the hardware
    > > except that it helps the OS track physical memory pages. By
    > > OS definition,
    > > a page with a non-zero reference count in it's PFN database
    > > entry is NOT
    > > pageable. This is essentially how MmProbeAndLockPages locks pages.
    > > It's a different mechanism, but similar that is used to lock
    > > pages within
    > > a working set. A process has a working set list that works
    > > in a similar
    > > manner to the PFN database to provide for working set trimming.
    > >
    > > I would imagine that what Max was saying is that when a page
    > > mapped by a
    > > PTE is paged in, the reference count for the page containing
    > > the PTE gets
    > > incremented to prevent it's being paged. I am not certain
    > > that this happens,
    > > but it certainly makes sense. In theory, however, there
    > > would be no reason
    > > why an OS couldn't choose to page PTE pages without paging
    > > the pages that
    > > they map. It wouldn't make sense to do this since a page fault would
    > > need to be rendered the next time a reference happened.
    > > Therefore, it makes
    > > sense to only page out a PTE page when all the pages it maps
    > > are paged out.
    > > I suppose that if you had some pages mapped by a PTE that were locked
    > > (probably for a DMA transfer), you might decide to page the PTE page
    > > since ALL the pages it maps cannot be paged. I wouldn't even
    > > be surprised
    > > if that happens in the case of a severe memory resource
    > > crisis. It certainly
    > > would be okay in theory to do.
    > >
    > >
    > > just my opinion based aon a potentially faulty recall
    > > mechanism. The old
    > > brain is not quite what it used to be.
    > >
    > >
    > > Rick Cadruvi...
    > >
    > >
    > > >All right, color me stupid, but why is it exactly that a
    > > page cannot be
    > > >paged out if its reference count is not zero? Wouldn't the
    > > reference count
    > > >be the number of valid references to this particular
    > > virtual->physical
    > > >mapping? Wouldn't a zero count indicate that the particular
    > > mapping, if it
    > > >is pageable, should be discarded? I thought, and perhaps I
    > > am very mistaken
    > > >here, that pageable virtual->physical mappings can always be
    > > paged out as
    > > >long as they are not explicitly 'pinned down', as in
    > > MmProbeAndLockPages.
    > > >
    > > >> -----Original Message-----
    > > >> From: Maxim S. Shatskih [mailto:[email protected]]
    > > >> Sent: Monday, April 03, 2000 3:42 AM
    > > >> To: NT Developers Interest List
    > > >> Subject: [ntdev] Re: Are page tables paged out?
    > > >>
    > > >>
    > > >> > Really? So in other words, a page in the page table can
    > > >> only be paged out
    > > >> > if all of the pages that it describes have been paged out
    > > >> themselves?
    > > >> i.e.
    > > >> > the page tables are the last thing to get paged out?
    > > >>
    > > >> Certainly. And can you imagine any other ways of pageable
    > > page tables?
    > > >> When a PTE is made valid, the reference count for the page
    > > >> table containing
    > > >> the PTE is incremented. And vice versa. Page can be swapped
    > > >> out only if
    > > >> refcount == 0.
    > > >>
    > > >> Max
    > > >>
    > > >>
    > > >>
    > > >> ---
    > > >> You are currently subscribed to ntdev as: [email protected]
    > > >> To unsubscribe send a blank email to
    > > $subst('Email.Unsub')
    > > >>
    > > >
    > > >---
    > > >You are currently subscribed to ntdev as: [email protected]
    > > >To unsubscribe send a blank email to $subst('Email.Unsub')
    > >
    > > ---
    > > You are currently subscribed to ntdev as: [email protected]
    > > To unsubscribe send a blank email to $subst('Email.Unsub')
    > >
    >
    > ---
    > You are currently subscribed to ntdev as: [email protected]
    > To unsubscribe send a blank email to $subst('Email.Unsub')
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    Hey I know, maybe we could all spend the next few days inventing
    explanations that work or don't work for 'what max refers to' :-)

    I agree that shared pages are reference counted.

    The point that interested me was the issue of does a page table ever page
    out while there are paged-in PTE's referenced from that page table. I'm
    still not convinced that the page table is referenced counted by each paged
    in PTE. However I am convinced that PTEs are not referenced counted by each
    referring Page Table, the 'vice versa' portion of 'what max referred to'.

    More to the point, this thread at some point asserted that Page Tables would
    rarely if ever page out, and I just don't think that that is true.

    > -----Original Message-----
    > From: I?aki Castillo [mailto:[email protected]]
    > Sent: Monday, April 03, 2000 2:18 PM
    > To: NT Developers Interest List
    > Subject: [ntdev] Re: Are page tables paged out?
    >
    >
    > I think Max refers to shared pages only. These pages have a
    > count reference
    > because actually they are not actual pages but pointers to
    > prototype entry
    > pages.
    > The pages are not discarded except if its reference (in the Page Frame
    > Database) counter is zero.
    > However this has nothing to do with normal, non shared pages.
    >
    > Inaki.
    >
    > > -----Original Message-----
    > > From: Roddy, Mark
    > > Sent: lunes 3 de abril de 2000 20:02
    > > To: NT Developers Interest List
    > > Subject: [ntdev] Re: Are page tables paged out?
    > >
    > > Sure I agree, but I'm still having trouble with Max saying:
    > >
    > > "When a PTE is made valid, the reference count for the page table
    > > containing
    > > the PTE is incremented. And vice versa. "
    > >
    > > The reference on the Page Table might or might not be OS MM
    > policy, but he
    > > is saying here, unless I'm misreading, that the reference
    > count on the PTE
    > > is also incremented, and that would make all PTE's effectively
    > > non-pageable.
    > >
    > >
    > > > -----Original Message-----
    > > > From: [email protected] [mailto:[email protected]]
    > > > Sent: Monday, April 03, 2000 11:13 AM
    > > > To: NT Developers Interest List
    > > > Subject: [ntdev] Re: Are page tables paged out?
    > > >
    > > >
    > > > Please forgive any errors in this description. It has been a long
    > > > time since I studied this and I am doing this off the top
    > of my head.
    > > >
    > > > Page table entries are hardware dependent. In fact, it would
    > > > be possible
    > > > (I think MIPS is this way) to NOT have a hardware PTE.
    > The hardware
    > > > implementations that I have looked at tdo not have
    > "reference counts"
    > > > in the PTEs. They would effectively be meaningless. They
    > > > would have a "valid"
    > > > bit as well as some kind of "dirty" bit to indicate that
    > > > there was a recent
    > > > write to the page so that page cleanup can make sure that
    > > > section files
    > > > get written out to the disk before pages are removed.
    > > >
    > > > The reference count referred to is probably the one in
    > the Page Frame
    > > > Database. This reference count is used to "lock" down
    > pages. The PFN
    > > > database is an OS construct and has NOTHING to do with
    > the hardware
    > > > except that it helps the OS track physical memory pages. By
    > > > OS definition,
    > > > a page with a non-zero reference count in it's PFN database
    > > > entry is NOT
    > > > pageable. This is essentially how MmProbeAndLockPages
    > locks pages.
    > > > It's a different mechanism, but similar that is used to lock
    > > > pages within
    > > > a working set. A process has a working set list that works
    > > > in a similar
    > > > manner to the PFN database to provide for working set trimming.
    > > >
    > > > I would imagine that what Max was saying is that when a page
    > > > mapped by a
    > > > PTE is paged in, the reference count for the page containing
    > > > the PTE gets
    > > > incremented to prevent it's being paged. I am not certain
    > > > that this happens,
    > > > but it certainly makes sense. In theory, however, there
    > > > would be no reason
    > > > why an OS couldn't choose to page PTE pages without paging
    > > > the pages that
    > > > they map. It wouldn't make sense to do this since a page
    > fault would
    > > > need to be rendered the next time a reference happened.
    > > > Therefore, it makes
    > > > sense to only page out a PTE page when all the pages it maps
    > > > are paged out.
    > > > I suppose that if you had some pages mapped by a PTE that
    > were locked
    > > > (probably for a DMA transfer), you might decide to page
    > the PTE page
    > > > since ALL the pages it maps cannot be paged. I wouldn't even
    > > > be surprised
    > > > if that happens in the case of a severe memory resource
    > > > crisis. It certainly
    > > > would be okay in theory to do.
    > > >
    > > >
    > > > just my opinion based aon a potentially faulty recall
    > > > mechanism. The old
    > > > brain is not quite what it used to be.
    > > >
    > > >
    > > > Rick Cadruvi...
    > > >
    > > >
    > > > >All right, color me stupid, but why is it exactly that a
    > > > page cannot be
    > > > >paged out if its reference count is not zero? Wouldn't the
    > > > reference count
    > > > >be the number of valid references to this particular
    > > > virtual->physical
    > > > >mapping? Wouldn't a zero count indicate that the particular
    > > > mapping, if it
    > > > >is pageable, should be discarded? I thought, and perhaps I
    > > > am very mistaken
    > > > >here, that pageable virtual->physical mappings can always be
    > > > paged out as
    > > > >long as they are not explicitly 'pinned down', as in
    > > > MmProbeAndLockPages.
    > > > >
    > > > >> -----Original Message-----
    > > > >> From: Maxim S. Shatskih [mailto:[email protected]]
    > > > >> Sent: Monday, April 03, 2000 3:42 AM
    > > > >> To: NT Developers Interest List
    > > > >> Subject: [ntdev] Re: Are page tables paged out?
    > > > >>
    > > > >>
    > > > >> > Really? So in other words, a page in the page table can
    > > > >> only be paged out
    > > > >> > if all of the pages that it describes have been paged out
    > > > >> themselves?
    > > > >> i.e.
    > > > >> > the page tables are the last thing to get paged out?
    > > > >>
    > > > >> Certainly. And can you imagine any other ways of pageable
    > > > page tables?
    > > > >> When a PTE is made valid, the reference count for the page
    > > > >> table containing
    > > > >> the PTE is incremented. And vice versa. Page can be swapped
    > > > >> out only if
    > > > >> refcount == 0.
    > > > >>
    > > > >> Max
    > > > >>
    > > > >>
    > > > >>
    > > > >> ---
    > > > >> You are currently subscribed to ntdev as:
    > [email protected]
    > > > >> To unsubscribe send a blank email to
    > > > $subst('Email.Unsub')
    > > > >>
    > > > >
    > > > >---
    > > > >You are currently subscribed to ntdev as: [email protected]
    > > > >To unsubscribe send a blank email to
    > $subst('Email.Unsub')
    > > >
    > > > ---
    > > > You are currently subscribed to ntdev as: [email protected]
    > > > To unsubscribe send a blank email to
    > $subst('Email.Unsub')
    > > >
    > >
    > > ---
    > > You are currently subscribed to ntdev as: [email protected]
    > > To unsubscribe send a blank email to $subst('Email.Unsub')
    >
    > ---
    > You are currently subscribed to ntdev as: [email protected]
    > To unsubscribe send a blank email to $subst('Email.Unsub')
    >
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    > long as they are not explicitly 'pinned down', as in MmProbeAndLockPages.

    Yes. And they are pinned down by incrementing the reference count.
    You can think of the reference count as of the "pin count".
    The number of mapping - the number of PTEs referencing the page - is not a
    reference count, but a _share_ count - a different thing.

    Max
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    > I would imagine that what Max was saying is that when a page mapped by a
    > PTE is paged in, the reference count for the page containing the PTE gets
    > incremented to prevent it's being paged.

    This happens - I have spent some time investigating the assembly code of
    MmAccessFault and other routines - so, I can say that this is for sure.

    Throwing the page away from the physical memory required update to its
    PTE to place a swapfile address in it. So, the PTE - and the page table
    page - must be resident at this moment. That's why it's impossible to page
    out the page table while it has even transition (standby list) PTEs, not to
    say
    present PTEs.

    Max
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    > The reference on the Page Table might or might not be OS MM policy, but he
    > is saying here, unless I'm misreading, that the reference count on the PTE
    > is also incremented, and that would make all PTE's effectively non-

    No. PTEs has no reference counts - only physical pages (PFN entries) has
    such.

    Max
  • OSR_Community_UserOSR_Community_User Member Posts: 110,217
    > The pages are not discarded except if its reference (in the Page Frame
    > Database) counter is zero.
    > However this has nothing to do with normal, non shared pages.

    PFN database keeps both ShareCount (for shared pages. This is number of
    present PTEs referencing the page) and ReferenceCount (this is "pin count"
    incremented in MmProbeAndLockPages).
    So, even non-shared pages have a reference count.
    The idea described by me is something like - "MmProbeAndLock is done
    on the page table the number of times equal to the number of
    present/transition PTEs in it".

    Max
Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. Sign in or register to get started.

Upcoming OSR Seminars
OSR has suspended in-person seminars due to the Covid-19 outbreak. But, don't miss your training! Attend via the internet instead!
Internals & Software Drivers 7 February 2022 Live, Online
Kernel Debugging 21 March 2022 Live, Online
Developing Minifilters 23 May 2022 Live, Online
Writing WDF Drivers 12 September 2022 Live, Online