Hooking up or wrapping kernel calls?

NTDEV
1

Hello,

I’m running a project that involves virtualization with Windows XP as a guest OS. For the project I need to be able to get some low-level information from the kernel. (E.g. address ranges for DMA mappings) The problem is that I need no get that info even at early stages during the boot - i.e. before any device driver is initialized.

Normally one would write some driver to intercept such information. But I doubt it would be possible to load and initialize this driver before any of the device drivers.

So the only other option I see is to somehow alter kernel behavior, i.e. create kind of wrapper around some of the exported functions. I don’t want to mess with the kernel itself, so the best option would be to create my own kernel version that would proxy all the calls to the real (original) kernel. Is it feasible? How do I do it?

I already tried to create such a proxying module, but came to a roadblock. ntldr seems to load only four files by itself: kernel (ntoskrnl.exe), bootvid.dll, hal.dll and kdcom.dll. And it doesn’t mind any of the import declarations of kernel. Can anybody, please, confirm that?

So, how can I create such proxying module? Or does my problem have simpler solution?

(Let me note that currently I’m interested only in Windows XP OS on x86 architecture. Also it may seem that the whole purpose of virtualization is to abstract the guest OS and host OS, but believe me I really need the described feature.)

Thanks ahead!

2

Ntoskrnl imports are in general not resolved by standard dllimport resolution by the loader like most other PE binaries. This detouring technique will not work.

  • S

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@gmail.com
Sent: Wednesday, March 25, 2009 3:02 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Hooking up or wrapping kernel calls?

Hello,

I’m running a project that involves virtualization with Windows XP as a guest OS. For the project I need to be able to get some low-level information from the kernel. (E.g. address ranges for DMA mappings) The problem is that I need no get that info even at early stages during the boot - i.e. before any device driver is initialized.

Normally one would write some driver to intercept such information. But I doubt it would be possible to load and initialize this driver before any of the device drivers.

So the only other option I see is to somehow alter kernel behavior, i.e. create kind of wrapper around some of the exported functions. I don’t want to mess with the kernel itself, so the best option would be to create my own kernel version that would proxy all the calls to the real (original) kernel. Is it feasible? How do I do it?

I already tried to create such a proxying module, but came to a roadblock. ntldr seems to load only four files by itself: kernel (ntoskrnl.exe), bootvid.dll, hal.dll and kdcom.dll. And it doesn’t mind any of the import declarations of kernel. Can anybody, please, confirm that?

So, how can I create such proxying module? Or does my problem have simpler solution?

(Let me note that currently I’m interested only in Windows XP OS on x86 architecture. Also it may seem that the whole purpose of virtualization is to abstract the guest OS and host OS, but believe me I really need the described feature.)

Thanks ahead!

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

3

wrote in message news:xxxxx@ntdev…
> Hello,
>
> I’m running a project that involves virtualization with Windows XP as a
> guest OS. For the project I need to be able to get some low-level
> information from the kernel. (E.g. address ranges for DMA mappings) The
> problem is that I need no get that info even at early stages during the
> boot - i.e. before any device driver is initialized.
>
> Normally one would write some driver to intercept such information. But I
> doubt it would be possible to load and initialize this driver before any
> of the device drivers.
>
> So the only other option I see is to somehow alter kernel behavior, i.e.
> create kind of wrapper around some of the exported functions. I don’t want
> to mess with the kernel itself, so the best option would be to create my
> own kernel version that would proxy all the calls to the real (original)
> kernel. Is it feasible? How do I do it?
>
> I already tried to create such a proxying module, but came to a roadblock.
> ntldr seems to load only four files by itself: kernel (ntoskrnl.exe),
> bootvid.dll, hal.dll and kdcom.dll. And it doesn’t mind any of the import
> declarations of kernel. Can anybody, please, confirm that?
>
> So, how can I create such proxying module? Or does my problem have simpler
> solution?

You can get any devices properties without hooking at anytime either using
the bios interrupt at boot time or the via the kernel itself later.

Usually legitimate virtualization software for PCs used abstract ‘idealized’
hardware. You don’t need to emulate the real hardware configured by the host
OS. So you can abstract any dma engine configuration as far as it is
consistent with the pc hardware standards and your hypervisor can filter and
forward the guest requests to the real dma on the host if needed.

But you will need to emulate an exact copy of your hardware if you try to
silently virtualized an already installed os - like the trendy virtualized
rootkits - otherwise you will get lots of messages at best or confuse the os
(bsod). To prevent this vulnerability you could try to be the first to
virtualize the cpu at boot but one can show that it is never granted (even
bios can be infected).
Signing with a tpm chip your boot process would alleviate most of the common
threats.

>
> (Let me note that currently I’m interested only in Windows XP OS on x86
> architecture. Also it may seem that the whole purpose of virtualization is
> to abstract the guest OS and host OS, but believe me I really need the
> described feature.)'

>
> Thanks ahead!
>
>
J/

4

Ken, thanks for clarification.

jchristatos, unfortunately virtualization technic I am using is not usual (however I’m not building rootkit or something). I don’t know the exact interface of a virtualized device beforehand, so (I guess) I’m not able to ask device or kernel for the needed info. Moreover, I need to capture dma allocations before the device knows about them. So it is also basically useless to ask kernel for such info later on. And I doubt there is a standard way to ask kernel for dma allocations. Also note that target device driver would be a third party one (and obviously unknown).

Any ideas?

5

What exactly do you mean by a “dma allocation”?

Mark Roddy

On Wed, Mar 25, 2009 at 6:00 AM, wrote:
> Ken, thanks for clarification.
>
> jchristatos, unfortunately virtualization technic I am using is not usual (however I’m not building rootkit or something). I don’t know the exact interface of a virtualized device beforehand, so (I guess) I’m not able to ask device or kernel for the needed info. Moreover, I need to capture dma allocations before the device knows about them. So it is also basically useless to ask kernel for such info later on. And I doubt there is a standard way to ask kernel for dma allocations. Also note that target device driver would be a third party one (and obviously unknown).
>
> Any ideas?
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>

6

Mark, sorry for confusion. I really meant DMA memory allocations, i.e. address ranges used for DMA. E.g. input and output parameters of AllocateCommonBuffer from DMA_OPERATIONS structure.

7

wrote in message news:xxxxx@ntdev…
> Ken, thanks for clarification.
>
> jchristatos, unfortunately virtualization technic I am using is not usual
> (however I’m not building rootkit or something).

Well, either you’re using the cpu vt-x extensions or binary translation or a
combination.

Other companies are sandboxing applications at a higher level and filtering
a restricted set of Apis not the os or devices.

>I don’t know the exact interface of a virtualized device beforehand,

You should define it yourself, you are virtualizing a device and presenting
a hardware api to the guest os whatever you do with the real hardware behind
doesn’t count. You could present a dma to the guest and use programmed io to
the real disk if you wish.

>so (I guess) I’m not able to ask device or kernel for the needed info.
>Moreover, I need to capture dma allocations before the device >knows about
>them.
As I said this is only valid if for some reason (?) you must have a 1-1
relationship between real/virtualized device in which case a small
pass-through device could do after detecting/enumerating devices.

Dma is not the only problem to run a guest os: graphics, usb, etc… should
be there too, not a small task.

>So it is also basically useless to ask kernel for such info later on. And I
>doubt there is a standard way to ask kernel for >dma allocations. >Also
>note that target device driver would be a third party one (and obviously
>unknown).
You shouldn’t care of device driver at this level of abstraction.

>
> Any ideas?
>
My 2 cents…
J/

8

What you are describing is a need to have your own HAL, but that has
been a non-starter with Microsoft for quite a few years now. Perhaps
instead you need to go down the road to virtualized bus driver hell
that has been documented here quite extensively by the Xen folks.
Their stuff is all gpl so you can take a look at what they’ve
accomplished.

Mark Roddy

On Wed, Mar 25, 2009 at 8:22 AM, wrote:
> Mark, sorry for confusion. I really meant DMA memory allocations, i.e. address ranges used for DMA. E.g. input and output parameters of AllocateCommonBuffer from DMA_OPERATIONS structure.
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>

9

> I’m running a project that involves virtualization with Windows XP as a guest OS.

Any virtualization which requires excessive patching (beyound virtual device drivers) to the guest OS is evil.

There is free MS Virtual PC, there is MS’s Hyper-V, included to Server 2008. For money, you can buy VMWare’s ESX Server. All these products do not install the ugly hacks to the guest OSes.

What is the need in any other virtualization, of Windows at least?

For the project I need to be able to get some low-level information from the kernel. (E.g. address
ranges for DMA mappings) The problem is that I need no get that info even at early stages during the
boot - i.e. before any device driver is initialized.

Before the driver is initialized, there is no DMA mappings.

Also - what is “DMA mappings” in Windows? Windows does not know this term.

I already tried to create such a proxying module, but came to a roadblock. ntldr seems to load only
four files by itself: kernel (ntoskrnl.exe), bootvid.dll, hal.dll and kdcom.dll.

+NLS tables, +SYSTEM registry hive, +all boot drivers and boot FS.

(Let me note that currently I’m interested only in Windows XP OS on x86 architecture. Also it may
seem that the whole purpose of virtualization is to abstract the guest OS and host OS

Once again: virtualization which requires excessive patching of the guest OS is just plain mis-architectured.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

10

>Moreover, I need to capture dma allocations before the device knows about them.

There is no such allocations. DMA common buffers are allocated after the driver load in MN_START_DEVICE (device instance init).

Actually, patching the guest OS to fit the arbitrary limits of the lame hypervisor is the bad idea. The proper way is to patch or even rearchitect the hypervisor instead.

Any memory in Windows is DMAble. For 16bit ISA DMA, the limit is 16MB, for 32bit PCI DMA, the limit is 3GB, for DualAddresCycle and newer version of PCI - no limit.

If the hypervisor cannot support DMA to these ranges without restrictions - then it is misarchitected.

If the hypervisor has some special notion of “DMAble memory”, without supporting guest-emulated DMA all over the guest’s memory - then it is mis-architected.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

11

> Mark, sorry for confusion. I really meant DMA memory allocations, i.e. address ranges used for DMA.

Why do you need them? Just understand that all memory is DMAble, and DMA is executed not only over common buffers, but over any custom MDLs too.

So, the hypervisor must support DMA over all guest’s memory.

Personally, I see no hardships in accessing all guest memory from hypervisor-side virtual device server, which is the DMA virtualization.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

12

> that has been documented here quite extensively by the Xen folks.

From the postings by James and Wayne here on this forum, I can conclude only the following:

  1. Xen hypervisor (or at least it’s IO bus model) is mis-architectered piece of junk, compared to its MS’s and VMWare’s competitors.

  2. James and Wayne personally are great talented guys who spend their time dealing with the lameness of the Xen hypervisor, working around its limitations from the guest OS side (probably it would be by far more proper to work around them from the hypervisor’s side, changing the IO bus model).

For instance: why create a custom DMA adapter object in the guest for Xen? why Xen cannot accept the guest-side addresses thru the control registers of the virtual device and access them? or the hypervisor does not know the address spaces of its guests?

Why not design and emulate the single PCI-based (Xen has complete PCI emulation anyway) device with PCI interrupt and only 2 registers, one holding a common buffer with all of the info and another is written to send a command? When the guest driver writes to the command reg, Xen intercepts it, gets the previously set address of the common buffer, accesses the command in this common buffer and executes it. With such an approach, you can have exactly 1 HVCall per IRP/SRB/packet chain, and 1 virtual interrupt on completion.

Why not make an ACPI table node for the guest capable of interrupt generation and implement the root of the IO bus this way, like Hyper-V does?

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

13

Thanks to everyone for your replies.

I totally agree that virtualization should not imply excessive guest patching. Actually, I was thinking for some time now about rearchitecturing some parts of the project.

However just out of curiosity I would like to get an answer: what detouring technics are possible for situation described above? I.e. how can one intercept or wrap kernel calls even at early boot stages (e.g. when bootload drivers are initialized)?

14

> how can one intercept or wrap kernel calls even at early boot stages (e.g. when bootload drivers

are initialized)?

Well, you have to write your own bootloader and patch ntoskrnl.exe (for example, extend its .TEXT section by modifying PE header and place your hooking code into it) before its entry point is invoked. Therefore, you have to play quite a few dirty tricks in your bootloader - you have to hook BIOS calls before transferring control to NTLDR…

Anton Bassov

15

> From the postings by James and Wayne here on this forum, I can conclude only the following:

  1. Xen hypervisor (or at least it’s IO bus model) is mis-architectered piece of junk, compared
    to its MS’s and VMWare’s competitors.

Well, as usually, you judge everything from Windows programmer’s perspective. Don’t forget that XEN is Linux-based project, so that it just makes an assumption that the guest can be painlessly patched. It is just not designed to host Windows unless host’s CPU has VT support…

virtualization which requires excessive patching of the guest OS is just plain mis-architectured.

Open you mind, Maxim - everything depends on your objectives. Don’t forget that the very idea of virtual machine (i.e. IBM’s VM back in 1964) is simply allowing the OS to run multiple copies of itself, so that guest’s kernel does not necessarily have to be blissfully ignorant of the host’s presence, especially if it offers potential overhead reduction.I do agree with you as long as you speak about Windows guest, but the world is not limited to Windows, is it…

  1. James and Wayne personally are great talented guys who spend their time dealing
    with the lameness of the Xen hypervisor    , working around its limitations from the guest OS side

Well, figuratively speaking, they are just trying to modify a nail in such way that it may be used by someone equipped with a screwdriver, rather than hammer…

(probably it would be by far more proper to work around them from the hypervisor’s side, changing
the IO bus model).

Totally agree - indeed, adding hammer capabilities to a screwdriver seems to be more reasonable approach, compared to adjusting nails to it…

For instance: why create a custom DMA adapter object in the guest for Xen?
why Xen cannot accept the guest-side addresses thru the control registers of the virtual device
and access them? or the hypervisor does not know the address spaces of its guests? Why
not design and emulate the single PCI-based (Xen has complete PCI emulation anyway) device
with PCI interrupt and only 2 registers, one holding a common buffer with all of the info and
another is written to send a command?

Well, who holds you back from starting something like “WinXEN project” that is capable of running unmodified guests regardless of CPU’s support for VY??? After all, XEN’s code is released under GPL…

Anton Bassov

16

>

> I’m running a project that involves virtualization with Windows XP
as
a guest OS.

Any virtualization which requires excessive patching (beyound virtual
device drivers) to the guest OS is evil.

There is free MS Virtual PC, there is MS’s Hyper-V, included to Server
2008. For money, you can buy VMWare’s ESX Server. All these products
do
not install the ugly hacks to the guest OSes.

How sure are you of this?

When I patched the windows kernel to remove any reference to the APIC
TASKPRI Register, the performance increase in XP was quite noticeable.
I’m quite sure that other virtualisation solutions, including the ones
you mention, do that too. Another option is to switch to a non-ACPI HAL,
but that’s a bad idea for other reasons.

James

17

I think people not working on virtualization are under the impression
that it is now all mostly handled by hardware vt support and that
‘messy hacks’ are no longer needed. Everyone else knows different :slight_smile:

Mark Roddy

On Fri, Mar 27, 2009 at 7:00 AM, James Harper
wrote:
>>
>> > I’m running a project that involves virtualization with Windows XP
> as
>> a guest OS.
>>
>> Any virtualization which requires excessive patching (beyound virtual
>> device drivers) to the guest OS is evil.
>>
>> There is free MS Virtual PC, there is MS’s Hyper-V, included to Server
>> 2008. For money, you can buy VMWare’s ESX Server. All these products
> do
>> not install the ugly hacks to the guest OSes.
>
> How sure are you of this?
>
> When I patched the windows kernel to remove any reference to the APIC
> TASKPRI Register, the performance increase in XP was quite noticeable.
> I’m quite sure that other virtualisation solutions, including the ones
> you mention, do that too. Another option is to switch to a non-ACPI HAL,
> but that’s a bad idea for other reasons.
>
> James
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
>

18

James, do your homework. Hook up a kernel debugger to those systems before
you authoritatively say what they do.

Microsoft Virtual PC used to patch guest OSes before hardware virtualization
was available. It doesn’t now.

Hyper-V has never done that. With that said, starting with Windows Server
2008, our operating systems behave differently if the Microsoft hypercall
interface is detected. This isn’t patching. It’s built in and available
when any hypervisor advertizes that interface. (As a final note, TPR
behavior isn’t among the things that change.)

–
Jake Oshins
Hyper-V I/O Architect
Windows Kernel Team

This post implies no warranties and confers no rights.

“James Harper” wrote in message
news:xxxxx@ntdev…
>>
>> > I’m running a project that involves virtualization with Windows XP
> as
>> a guest OS.
>>
>> Any virtualization which requires excessive patching (beyound virtual
>> device drivers) to the guest OS is evil.
>>
>> There is free MS Virtual PC, there is MS’s Hyper-V, included to Server
>> 2008. For money, you can buy VMWare’s ESX Server. All these products
> do
>> not install the ugly hacks to the guest OSes.
>
> How sure are you of this?
>
> When I patched the windows kernel to remove any reference to the APIC
> TASKPRI Register, the performance increase in XP was quite noticeable.
> I’m quite sure that other virtualisation solutions, including the ones
> you mention, do that too. Another option is to switch to a non-ACPI HAL,
> but that’s a bad idea for other reasons.
>
> James
>

19

> I think people not working on virtualization are under the impression that it is now all mostly

handled by hardware vt support and that ‘messy hacks’ are no longer needed.

I think that their beliefs are based upon the assumption that., unless the host tells the guest it is being debugged, the host will be unable to run 64-bit Windows guests if it patches their kernels…

Anton Bassov

20

> starting with Windows Server 2008, our operating systems behave differently if the Microsoft

hypercall interface is detected. This isn’t patching. It’s built in and available when any hypervisor
advertizes that interface.

Well, unless this interface is properly documented and is available for use by non-MSFT hypervisors, the above statement alone is sufficient to trigger yet another lawsuit against MSFT…

Anton Bassov