tell windows not to touch pages of memory (hibernation problem solved, sort of)

NTDEV
1

Part of the initialization my driver does is to gift some pages of
memory back to xen to balance the books (it had to takes some pages from
xen for something else). This leaves some memory ‘holes’, and it turns
out that hibernation copies these pages to the buffers it passes to my
dump/hiber storpot driver. It appears that touching these pages causes
the VM to blow up.

I will approach this problem from both directions - it is in fact more a
bug in Xen than Windows - but is there a way to tell Windows that pages
of memory I have allocated are never to be touched, not even by the
hiber or dump driver? (the dump driver doesn’t actually double buffer
the pages it seems so it never touches them, just hands me the physical
address).

So I’m looking for something that does one of the following:
. Tells windows that the physical page isn’t important for hibernation
(and crash dump if possible)
. Tells windows that the physical page doesn’t exist anymore
. Tells windows that the physical page is bad or something
With the net result being that hiber and dump never touch the page…

Thanks

James

2

Have you already tried MmRemovePhysicalMemory (declared in ntddk.h) or
MmMarkPhysicalMemoryAsBad (same parameters as MmRemovePhysicalMemory)?

On Fri, 21 Jan 2011 13:28:48 +0100, James Harper
wrote:
> Part of the initialization my driver does is to gift some pages of
> memory back to xen to balance the books (it had to takes some pages from
> xen for something else). This leaves some memory ‘holes’, and it turns
> out that hibernation copies these pages to the buffers it passes to my
> dump/hiber storpot driver. It appears that touching these pages causes
> the VM to blow up.
>
> I will approach this problem from both directions - it is in fact more a
> bug in Xen than Windows - but is there a way to tell Windows that pages
> of memory I have allocated are never to be touched, not even by the
> hiber or dump driver? (the dump driver doesn’t actually double buffer
> the pages it seems so it never touches them, just hands me the physical
> address).
>
> So I’m looking for something that does one of the following:
> . Tells windows that the physical page isn’t important for hibernation
> (and crash dump if possible)
> . Tells windows that the physical page doesn’t exist anymore
> . Tells windows that the physical page is bad or something
> With the net result being that hiber and dump never touch the page…
>
> Thanks
>
> James

3

The balloon driver has to take memory away ‘for a while’ and then give
it back, so marking it as bad would be,well, bad.
I’m guessing remove is going to have a problem if the memory is in use.

MSFT has probably implemented a balloon driver for their hypervisor,
so they may also have solved the hibernate problem.

Mark Roddy

On Fri, Jan 21, 2011 at 8:07 AM, Cay Bremer wrote:
> Have you already tried MmRemovePhysicalMemory (declared in ntddk.h) or
> MmMarkPhysicalMemoryAsBad (same parameters as MmRemovePhysicalMemory)?
>
>
> On Fri, 21 Jan 2011 13:28:48 +0100, James Harper
> wrote:
>>
>> Part of the initialization my driver does is to gift some pages of
>> memory back to xen to balance the books (it had to takes some pages from
>> xen for something else). This leaves some memory ‘holes’, and it turns
>> out that hibernation copies these pages to the buffers it passes to my
>> dump/hiber storpot driver. It appears that touching these pages causes
>> the VM to blow up.
>>
>> I will approach this problem from both directions - it is in fact more a
>> bug in Xen than Windows - but is there a way to tell Windows that pages
>> of memory I have allocated are never to be touched, not even by the
>> hiber or dump driver? (the dump driver doesn’t actually double buffer
>> the pages it seems so it never touches them, just hands me the physical
>> address).
>>
>> So I’m looking for something that does one of the following:
>> . Tells windows that the physical page isn’t important for hibernation
>> (and crash dump if possible)
>> . Tells windows that the physical page doesn’t exist anymore
>> . Tells windows that the physical page is bad or something
>> With the net result being that hiber and dump never touch the page…
>>
>> Thanks
>>
>> James
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars
> visit:http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

4

Actually Mark, we use a very different design in these areas. Xen (to my
naïve understanding) uses a cooperative model for VM state changes. They
actually run code in the guest as part of saving a VM, migrating a VM, etc.
Hyper-V uses a model where VM transitions can happen on any instruction
boundary and there’s no interaction with code in the guest.

Consequently, you can’t expect a guest to do anything regarding ballooning
or anything else that would run code in the guest.

As for James’s specific problem, we let the guest touch those pages as part
of crashdump and/or hibernation and we just show it a fake page that’s
mapped into every “missing” page that’s been ballooned. The contents don’t
matter, really.

Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group

This post implies no warranties and confers no rights.

“Mark Roddy” wrote in message news:xxxxx@ntdev…

The balloon driver has to take memory away ‘for a while’ and then give
it back, so marking it as bad would be,well, bad.
I’m guessing remove is going to have a problem if the memory is in use.

MSFT has probably implemented a balloon driver for their hypervisor,
so they may also have solved the hibernate problem.

Mark Roddy

On Fri, Jan 21, 2011 at 8:07 AM, Cay Bremer wrote:
> Have you already tried MmRemovePhysicalMemory (declared in ntddk.h) or
> MmMarkPhysicalMemoryAsBad (same parameters as MmRemovePhysicalMemory)?
>
>
> On Fri, 21 Jan 2011 13:28:48 +0100, James Harper
> wrote:
>>
>> Part of the initialization my driver does is to gift some pages of
>> memory back to xen to balance the books (it had to takes some pages from
>> xen for something else). This leaves some memory ‘holes’, and it turns
>> out that hibernation copies these pages to the buffers it passes to my
>> dump/hiber storpot driver. It appears that touching these pages causes
>> the VM to blow up.
>>
>> I will approach this problem from both directions - it is in fact more a
>> bug in Xen than Windows - but is there a way to tell Windows that pages
>> of memory I have allocated are never to be touched, not even by the
>> hiber or dump driver? (the dump driver doesn’t actually double buffer
>> the pages it seems so it never touches them, just hands me the physical
>> address).
>>
>> So I’m looking for something that does one of the following:
>> . Tells windows that the physical page isn’t important for hibernation
>> (and crash dump if possible)
>> . Tells windows that the physical page doesn’t exist anymore
>> . Tells windows that the physical page is bad or something
>> With the net result being that hiber and dump never touch the page…
>>
>> Thanks
>>
>> James
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars
> visit:http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

5

Just for the sake of completeness, there are also MmAddPhysicalMemory and
MmMarkPhysicalMemoryAsGood. It might be worth a try.

On Fri, 21 Jan 2011 15:17:25 +0100, Mark Roddy
wrote:
> The balloon driver has to take memory away ‘for a while’ and then give
> it back, so marking it as bad would be,well, bad.
> I’m guessing remove is going to have a problem if the memory is in use.
>
> MSFT has probably implemented a balloon driver for their hypervisor,
> so they may also have solved the hibernate problem.
>
> Mark Roddy
>
>
>
> On Fri, Jan 21, 2011 at 8:07 AM, Cay Bremer wrote:
>> Have you already tried MmRemovePhysicalMemory (declared in ntddk.h) or
>> MmMarkPhysicalMemoryAsBad (same parameters as MmRemovePhysicalMemory)?
>>
>>
>> On Fri, 21 Jan 2011 13:28:48 +0100, James Harper
>> wrote:
>>>
>>> Part of the initialization my driver does is to gift some pages of
>>> memory back to xen to balance the books (it had to takes some pages
>>> from
>>> xen for something else). This leaves some memory ‘holes’, and it turns
>>> out that hibernation copies these pages to the buffers it passes to my
>>> dump/hiber storpot driver. It appears that touching these pages causes
>>> the VM to blow up.
>>>
>>> I will approach this problem from both directions - it is in fact more
>>> a
>>> bug in Xen than Windows - but is there a way to tell Windows that pages
>>> of memory I have allocated are never to be touched, not even by the
>>> hiber or dump driver? (the dump driver doesn’t actually double buffer
>>> the pages it seems so it never touches them, just hands me the physical
>>> address).
>>>
>>> So I’m looking for something that does one of the following:
>>> . Tells windows that the physical page isn’t important for hibernation
>>> (and crash dump if possible)
>>> . Tells windows that the physical page doesn’t exist anymore
>>> . Tells windows that the physical page is bad or something
>>> With the net result being that hiber and dump never touch the page…
>>>
>>> Thanks
>>>
>>> James

6

>

Actually Mark, we use a very different design in these areas. Xen (to my
na?ve understanding) uses a cooperative model for VM state changes. They
actually run code in the guest as part of saving a VM, migrating a VM, etc.
Hyper-V uses a model where VM transitions can happen on any instruction
boundary and there’s no interaction with code in the guest.

Consequently, you can’t expect a guest to do anything regarding ballooning
or anything else that would run code in the guest.

That’s not quite right. Xen can run a PV (ParaVirtualised - ‘fully cooperative’) VM where the VM runs on Xen rather than on an emulated standard x86 machine. Obviously Windows would need a custom HAL and kernel to do that. Then you have HVM mode where Xen presents a standard x86 platform (BIOS, emulated network card, emulated vga adapter, emulated IDE interface, etc). In that mode, Xen handles the state change (save/restore/live migration) completely by itself and apart from maybe seeing a network cable unplug/plug event the VM has no idea that anything happened.

Once you run PV drivers running on a HVM VM though, we are indeed back to a ‘(semi-)cooperative model’ again. Xen detects that PV drivers are loaded and hands off shutdown/reboot/save/migrate etc events via xenbus (inter VM communications channel). This allows the PV drivers to disconnect from the backend device (I don’t tell Windows that this has happened though) and then reconnect again. Xen still takes care of some of the state changes though.

As for James’s specific problem, we let the guest touch those pages as part
of crashdump and/or hibernation and we just show it a fake page that’s
mapped into every “missing” page that’s been ballooned. The contents don’t
matter, really.

That’s what I was hoping to do. A 1-many mapping of ‘machine’[1] to ‘physical’[2] frames might be problematic though if xen is only written with a 1:1 maping in mind. I think it has been discussed before so I’ll check the archives.

Having slept on it, I think I can work around it. HVM provides a PCI device which contains an interrupt and some memory space, which is what my bus driver attaches to. The memory space is just a hole for mapping various pages into. Having mapped the pages from Xen into that hole I then need to give Xen some pages back to keep the total page count correct or save/migrate doesn’t work (I call this a bug, but it was considered a ‘feature’ when it was first discussed :). I think I should be able to allocate normal non-paged memory, give that to xen, then tell xen to map it’s pages into that ‘hole’ instead, which means hibernate shouldn’t blow up as there are no ‘holes’.

One thing I haven’t implemented in my driver is PoD (Populate on Demand) where the VM starts out with a certain amount of RAM, but only some of it is populated. The driver in this case would need to allocate the unpopulated memory before Windows touches it, then ask Xen to populate it and then free it back to Windows as required. My above solution isn’t going to stop hibernate crashing when it comes across unpopulated memory holes though.

James

[1] ‘machine’ being actual physical page of memory as seen by Xen
[2] ‘physical’ being a physical page of memory as seen by the VM

7

>

Just for the sake of completeness, there are also MmAddPhysicalMemory
and
MmMarkPhysicalMemoryAsGood. It might be worth a try.

I looked at that a long time ago. Doesn’t sound like the sort of thing
that would get me WHQL though.

There is also the possibility that hibernate will read the memory and
write it to the hiberfile anyway. In a normal machine this wouldn’t have
any side effects and would be easier than checking the page against a
bad/missing list each time.

Thanks

James