strange interrupt behaviour when HAL is Standard PC

[James_Harper]

To make my Xen drivers work more like they are talking to ‘real’
hardware, I have the bus driver enumerate the children with a resource
requirement of the same interrupt as the bus driver, so each device
connects to the same interrupt. On my test system I have the bus driver
and 3 device drivers - 2 x xenvbd (block device driver) and 1 x xennet
(network driver). They load in that order so for every interrupt that
occurs for the bus driver, the ISR for each child device is also called.
Each device always tells windows that the interrupt was not for itself
to ensure that the next device is always called.

So for example, if an interrupt occurs which is really a xen event for
the second xenvbd device, the ISR call list goes like:

1. bus driver isr is called and it works out which child device(s) the
   event is for, and sets a bit in a shared memory area for that device.
2. xenvbd #1 isr checks the shared memory area, notices that the
   interrupt is not for it, and returns immediately
3. xenvbd #2 isr checks the shared memory area, notices that the
   interrupt is for it, and does its processing
4. xennet isr checks the shared memory area, notices that the interrupt
   is not for it, and returns immediately

Aside from the obvious performance hit of having additional overhead per
isr, it works really well, except (I’ve just found out) when the hal
type is set to ‘Standard PC’ instead of ‘ACPI xxx’. What happens then is
that the bus driver, xenvbd #1, and xenvbd #2 ISR’s are always called,
but xennet (NDIS 5.1 driver) stops getting for every interrupt after the
first 500 or so interrupts. Instead it only gets called every 3-20(-ish)
interrupts. My test environment is Windows 2003 R2 SP2

It’s almost like NDIS is assuming that I am getting spurious interrupts
and trying to save me some time or something.

Under the ACPI HAL, the PCI device gets an IRQ of 28, but under Standard
PC, I am obviously limited to IRQ’s <16 so I get IRQ 5. Not sure if
that’s important, except that maybe the PIC behaves differently than an
APIC would, or maybe it’s an artefact of virtualisation?

Can anyone offer any suggestions? Or any way to approach this
differently? Is there a way to create a virtual interrupt driver (eg
something that exposes IRQ resources) under Windows so that the child
devices can attach to my virtual interrupts?

Merry Christmas all!

Thanks

James

[Mark_Roddy]

James,
Having worked with your xenbus driver, it is my opinion that your design of
‘always returning false to insure everyone gets called’ is problematic.
Shared level-sensitive interrupts don’t work that way. Instead what should
happen is that each ‘device asserting an interrupt’ should keep its
interrupt assertion ‘on’ until its associated function driver explicitly
turns that interrupt assertion off and returns true from its isr. Shared
latched interrupts sort of work the way your design is implemented, but the
‘always call everyone on the vector’ function is implemented in NT’s chained
interrupt handler and the function drivers should once again return true iff
the interrupt was for them.

Mark Roddy

On Wed, Dec 24, 2008 at 10:27 PM, James Harper <
xxxxx@bendigoit.com.au> wrote:

To make my Xen drivers work more like they are talking to ‘real’
hardware, I have the bus driver enumerate the children with a resource
requirement of the same interrupt as the bus driver, so each device
connects to the same interrupt. On my test system I have the bus driver
and 3 device drivers - 2 x xenvbd (block device driver) and 1 x xennet
(network driver). They load in that order so for every interrupt that
occurs for the bus driver, the ISR for each child device is also called.
Each device always tells windows that the interrupt was not for itself
to ensure that the next device is always called.

So for example, if an interrupt occurs which is really a xen event for
the second xenvbd device, the ISR call list goes like:

1. bus driver isr is called and it works out which child device(s) the
   event is for, and sets a bit in a shared memory area for that device.
2. xenvbd #1 isr checks the shared memory area, notices that the
   interrupt is not for it, and returns immediately
3. xenvbd #2 isr checks the shared memory area, notices that the
   interrupt is for it, and does its processing
4. xennet isr checks the shared memory area, notices that the interrupt
   is not for it, and returns immediately

Aside from the obvious performance hit of having additional overhead per
isr, it works really well, except (I’ve just found out) when the hal
type is set to ‘Standard PC’ instead of ‘ACPI xxx’. What happens then is
that the bus driver, xenvbd #1, and xenvbd #2 ISR’s are always called,
but xennet (NDIS 5.1 driver) stops getting for every interrupt after the
first 500 or so interrupts. Instead it only gets called every 3-20(-ish)
interrupts. My test environment is Windows 2003 R2 SP2

It’s almost like NDIS is assuming that I am getting spurious interrupts
and trying to save me some time or something.

Under the ACPI HAL, the PCI device gets an IRQ of 28, but under Standard
PC, I am obviously limited to IRQ’s <16 so I get IRQ 5. Not sure if
that’s important, except that maybe the PIC behaves differently than an
APIC would, or maybe it’s an artefact of virtualisation?

Can anyone offer any suggestions? Or any way to approach this
differently? Is there a way to create a virtual interrupt driver (eg
something that exposes IRQ resources) under Windows so that the child
devices can attach to my virtual interrupts?

Merry Christmas all!

Thanks

James

---

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

[Peter_Viscarola_OSR]

Is putting this whole “interrupt fanout” logic in the DOMU (virtual client) the wrong approach anyhow?

Aren’t you just doing this to work-around the initial Xen implementation to signal the DOMU drivers using a single interrupt? Isn’t Xen being modified, or has already been modified by some vendors, to send more than one interrupt to the DOMU?

Not being a Linux/Xen weenie, I could be mistaken… but I certainly remember discussions about making some flavor of Xen support one interrupt per device, at least at one point with “the right people” in the Linux/Xen community…

Peter
OSR

[James_Harper]

> James,

Having worked with your xenbus driver, it is my opinion that your
design
of ‘always returning false to insure everyone gets called’ is
problematic.
Shared level-sensitive interrupts don’t work that way. Instead what
should
happen is that each ‘device asserting an interrupt’ should keep its
interrupt assertion ‘on’ until its associated function driver
explicitly
turns that interrupt assertion off and returns true from its isr.
Shared
latched interrupts sort of work the way your design is implemented,
but
the ‘always call everyone on the vector’ function is implemented in
NT’s
chained interrupt handler and the function drivers should once again
return true iff the interrupt was for them.

The problem is that to avoid a race, I have to clear Xen’s ‘interrupt
pending’ flag before I check to see which devices the interrupt is
for. So unless I call all the devices in turn, interrupts will get
missed.

I have been trying to figure out a way around this, but the current way
works well (aside from the NDIS issue under the Standard PC HAL). Every
path I explore to come up with a solution appears to be a dead end.

I agree with you that the solution is hardly ideal, but I’m stuck with
it for now, and even if the xen qemu implementation changes to allow
more interrupts, I’m still stuck supporting legacy code.

What I have done though is to go back to the Xen bus driver making a
callback to the NDIS driver directly, rather than doing it by letting
the ISR flow down. This should cut out a bit of overhead and latency,
although I fear I’m drifting further away from WHQL compliance. For
scsiport drivers I’m stuck with ISR delivery of events, although
scsiport appears to deliver every interrupt correctly.

Thanks for the feedback.

James

[James_Harper]

>

Is putting this whole “interrupt fanout” logic in the DOMU (virtual
client) the wrong approach anyhow?

I’d be hesitant to use the word ‘wrong’, as it’s the only one available,
but certainly with a holistic view there are better ways.

Aren’t you just doing this to work-around the initial Xen
implementation
to signal the DOMU drivers using a single interrupt? Isn’t Xen being
modified, or has already been modified by some vendors, to send more
than
one interrupt to the DOMU?

Not being a Linux/Xen weenie, I could be mistaken… but I certainly
remember discussions about making some flavor of Xen support one
interrupt
per device, at least at one point with “the right people” in the
Linux/Xen
community…

I raised that on the xen-devel mailing list, and it’s probably been
raised before by others. It would actually be done inside qemu - Linux
solves the problem by creating virtual interrupts at the kernel level -
a luxury we don’t have under windows

I will post on xen-devel in a minute in case there has been work to that
effect that I’ve missed, and possibly implement it myself as I’m
reasonably familiar with qemu code these days.

I think that the biggest hurdle will be the mechanism to define where to
put the ‘interrupt pending’ flag - not so much the logic, but getting
everyone to agree on how that should be approached. Fortunately on that
list everyone is reasonably well behaved when it comes to subjective
discussions like that *ducks and runs for cover*

Thanks

James

[James_Harper]

>

Is putting this whole “interrupt fanout” logic in the DOMU (virtual
client) the wrong approach anyhow?

Some background info (I think you know this Peter but others are
probably unfamiliar with xen) - xen deals in event channels, which allow
one ‘domain’ (virtual machine) to signal another ‘domain’. Normally this
is between Dom0 (which talks to physical hardware devices) and other
domains (other domains are typically referred to as DomU where the U
stands for unprivileged)

When a windows DomU has an event pending, the xen hypervisor signals it
by asserting an interrupt line. That interrupt is attached to a virtual
PCI device. By having this virtual PCI device, it gives my PV drivers a
PCI id to attach to, and an interrupt to connect to.

If I want to use more than a single interrupt I’m not sure about the
mechanism I should use to get Windows to allocate them to me. In a
previous version of the drivers I was using ‘asm {int xx}’ instructions,
which others have pointed out is a flawed idea and as I found out, has
performance problems. I was able to add an interrupt requirement to
child devices in my bus driver, and Windows appeared quite happy to
assign an interrupt number and vector to my interrupt though. I’m not
sure if this would be sufficient to just give Xen the IRQ number and
have it signal the individual device that way.

Under Linux PV domains (where the kernel knows it is being virtualised),
the Linux kernel allows attaching an event to an interrupt but the way
this works is that when the event comes in, the Linux kernel just calls
the ISR directly - no need to worry about APIC’s etc. I don’t have the
luxury of being able to insert code into Windows ‘above’ the interrupt
delivery mechanism though, or at least can’t do so in a way that
Microsoft would appreciate (unless someone here can tell me a way?)

Any comments or discussion would be much appreciated.

Thanks

James

[OSR_Community_User]

From what I undersand, there would be some work involved to make the virtual PCI card use more than one interrupt. Even if there were more though, there would have to be one for each device instance and that just doesn’t scale.

For our Win2k - 2k8 driverset, the PCI card represents the Disk driver (as I remember that yours does). The bus is root enumerated, and the NICs have a normal WDM relationship with the bus. The interrupt logic works like this:

The disk driver receives and interrupt.
look at bits and determine who has work
queue a DPC to the bus for any work to be done other than disk (NIC or XenBus)
do any disk work
return true if any work was done

[Peter_Viscarola_OSR]

I’d be hesitant to use the word ‘wrong’, as it’s the only one available, but certainly with a holistic view there are better ways.

Quite right… of course. “Wrong” was clearly the, er, wrong word. “Not the best overall solution”, perhaps?

To be clear, my question was more out of curiosity – whether you’d considered the QEMU change or whether somebody had already done this. I know it’s been discussed (during Xen development – there are references in some of the original papers to extending the one interrupt design to more interrupts – and again internally within specific vendors), and I’m curious as to the state of things.

BTW, I’m not saying things CAN’T be made to work with one interrupt… obviously, there are several vendors who’ve written numerous flavors of Linux PV drivers already. Some of these even perform exceptionally well.

I’d be interested to hear what xen-devel says.

As to WHQL compliance… I have never seen an NDIS or SCSI PV driver that didn’t take “considerable liberties” that would be at odds with WHQL requirements to achieve good performance. In our experience here at OSR (which I admit is not entirely definitive in this area) there is no alternative.

The whole topic of what the best architectural approach to supporting Linux PV within Windows drivers is a really interesting one – if a rather arcane one. Every time there a Windows NIC and SCSI PV driver pair are written, I say to myself “well, THAT must be the absolutely last pair of Windows NIC and SCSI PV drivers anyone will need to write”… but they keep on coming!

Peter
OSR

[Maxim_S_Shatskih]

>When a windows DomU has an event pending, the xen hypervisor signals it

by asserting an interrupt line.

This is absolutely OK and is very similar to what Hyper-V does.

Your only problem is actually SCSIPORT, which does not support virtual miniports. STORPORT which does support them is on 2003 and later only, and, BTW, Hyper-V’s virtual SCSI (storvsc) is the STORPORT-based miniport not available on XP.

If you need XP/w2k support - then you can:
a) write a full storage port
b) do some hacks about creating a really virtual SCSIPORT’s miniport
c) patch the Xen hypervisor to expose an interrupt per each guest’s storage controller, not only for a bus root device.

With 2003 and later, you have virtual STORPORT miniports. Write one which will send all requests down to its PDO.

Then write the similar NDIS-WDM miniport for network, and the similar drivers for all other XenBus hardware.

The PDOs will belong to XenBus driver which will communicate to Xen via the fake device and interrupts.

Once again - SCSIPORT (i.e. w2k and XP) is the only issue.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

[James_Harper]

>

>When a windows DomU has an event pending, the xen hypervisor signals
it
>by asserting an interrupt line.

This is absolutely OK and is very similar to what Hyper-V does.

Your only problem is actually SCSIPORT, which does not support virtual
miniports. STORPORT which does support them is on 2003 and later only,
and, BTW, Hyper-V’s virtual SCSI (storvsc) is the STORPORT-based
miniport
not available on XP.

If you need XP/w2k support - then you can:
a) write a full storage port
b) do some hacks about creating a really virtual SCSIPORT’s miniport
c) patch the Xen hypervisor to expose an interrupt per each guest’s
storage controller, not only for a bus root device.

With 2003 and later, you have virtual STORPORT miniports. Write one
which
will send all requests down to its PDO.

Separate to the scsiport ‘block device driver’, where much of the SCSI
commands are emulated, I also do scsi passthrough where SCSI commands
are sent in the Dom0 onto a real SCSI bus to a real SCSI device.
Initially this was written as a scsiport driver, and worked, but I
attempted to rewrite it as a storport driver, and just couldn’t get it
working. I found that:
. hw config and port config structure members documented as ‘will be set
to xxx by default’ were in fact not actually set to xxx
. Members documented as ‘driver must not modify this’ had to be modified
to correct the above
. storport would never, ever, hand me buffers larger than 16k, despite
having set all the write parameters to allow this. This was a
dealbreaker as HP’s L&TT’s would get SCSI requests failed by storport
(I’d never see them) when it attempted to do things like update the
firmware of my tape drive.

I think maybe that last one was because the storport device was on the
‘Internal’ bus, not the PCI bus, and so storport was imposing some
limitations on me.

Anyway, I would be reluctant to want to go down the storport path
again…

Then write the similar NDIS-WDM miniport for network, and the similar
drivers for all other XenBus hardware.

The PDOs will belong to XenBus driver which will communicate to Xen
via
the fake device and interrupts.

Once again - SCSIPORT (i.e. w2k and XP) is the only issue.

Thanks for the input.

James

[OSR_Community_User]

Hello,

Several thought on the subject:

First of all I think you should check VirtIO implementation. It should be hypervisor agnostic and was also implemented for QEMU.

Now without going into hypervisor depended discussion but keeping strictly to the guest, here are my 2 cents:

* If you want to have PV bus driver I see several approaches. In any case you should have 1 PCI device which will be governed by your bus driver and it will allocate and generate the interrupts for all the devices on PV bus.

1. Child devices are never handling real interrupts. Bus driver will handle the interrupt and dispatch DPC for the registered child device when needed.
2. Bus driver will call child device callback while in ISR.

** In both case please remember that your bus driver still may share interrupt with other device. Also you will need to create the interface for your bus, though which child devices will be able to register\deregister callbacks\DPCs.
** Bus driver approach has its WHQL limitations.

* Other option is to create PCI device for each PV device you want to create. Let QEMU and Windows handle interrupt sharing.

Best regards,
Yan Vugenfirer.

[Jake_Oshins]

This is good advice.

–
Jake Oshins
Hyper-V I/O Architect
Windows Kernel Team

This post implies no warranties and confers no rights.

---

wrote in message news:xxxxx@ntdev…
> Hello,
>
> Several thought on the subject:
>
> First of all I think you should check VirtIO implementation. It should be
> hypervisor agnostic and was also implemented for QEMU.
>
>
> Now without going into hypervisor depended discussion but keeping strictly
> to the guest, here are my 2 cents:
>
> * If you want to have PV bus driver I see several approaches. In any case
> you should have 1 PCI device which will be governed by your bus driver and
> it will allocate and generate the interrupts for all the devices on PV
> bus.
> 1. Child devices are never handling real interrupts. Bus driver will
> handle the interrupt and dispatch DPC for the registered child device when
> needed.
> 2. Bus driver will call child device callback while in ISR.
>
> In both case please remember that your bus driver still may share
> interrupt with other device. Also you will need to create the interface
> for your bus, though which child devices will be able to
> register\deregister callbacks\DPCs.
> Bus driver approach has its WHQL limitations.
>
> * Other option is to create PCI device for each PV device you want to
> create. Let QEMU and Windows handle interrupt sharing.
>
> Best regards,
> Yan Vugenfirer.
>
>

[James_Harper]

>

BTW, I’m not saying things CAN’T be made to work with one interrupt…
obviously, there are several vendors who’ve written numerous flavors
of
Linux PV drivers already. Some of these even perform exceptionally
well.

Under Linux you can do whatever you want (eg rewrite whole sections of
the kernel if required), so there are none of these limitations

I’d be interested to hear what xen-devel says.

As to WHQL compliance… I have never seen an NDIS or SCSI PV driver
that
didn’t take “considerable liberties” that would be at odds with WHQL
requirements to achieve good performance. In our experience here at
OSR
(which I admit is not entirely definitive in this area) there is no
alternative.

The whole topic of what the best architectural approach to supporting
Linux PV within Windows drivers is a really interesting one – if a
rather
arcane one. Every time there a Windows NIC and SCSI PV driver pair
are
written, I say to myself “well, THAT must be the absolutely last pair
of
Windows NIC and SCSI PV drivers anyone will need to write”… but they
keep on coming!

I have exchanged a few posts with the xen-devel list and now have some
ideas about how a multiple irq scenario might work. I’m trying to come
up with a solution to another problem at the moment (virtualised TPR
writes are slow), so it’s something I’ll have to get back to.

James

[OSR_Community_User]

I find the whole “In linux you can…” to be inappropriate. It doesn’t make
sense in general, and even discussing it in a Windows group is silly beyond
comprehension. Sure, you can, and then you become a maintainer of the only
version of linux that can run your driver. From a business viewpoint, this
sucks. In the entire history I have been involved with Unix, it has never
made sense to modify the kernel to run something that was intended for
distribution beyond the machine so modified. I find the whole “you can
change anything” argument fallacious in the extreme. The consequences of
these actions are profound. I even watched projects crash and burn because
kernel modifications were made that meant that two different
programs/drivers could not coexist on the same machine because they required
mutually-incompatible changes, and they could not run on unmodified kernels
at all.
joe

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of James Harper
Sent: Monday, December 29, 2008 5:49 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] strange interrupt behaviour when HAL is Standard PC

BTW, I’m not saying things CAN’T be made to work with one interrupt…
obviously, there are several vendors who’ve written numerous flavors
of
Linux PV drivers already. Some of these even perform exceptionally
well.

Under Linux you can do whatever you want (eg rewrite whole sections of the
kernel if required), so there are none of these limitations

I’d be interested to hear what xen-devel says.

As to WHQL compliance… I have never seen an NDIS or SCSI PV driver
that
didn’t take “considerable liberties” that would be at odds with WHQL
requirements to achieve good performance. In our experience here at
OSR
(which I admit is not entirely definitive in this area) there is no
alternative.

The whole topic of what the best architectural approach to supporting
Linux PV within Windows drivers is a really interesting one – if a
rather
arcane one. Every time there a Windows NIC and SCSI PV driver pair
are
written, I say to myself “well, THAT must be the absolutely last pair
of
Windows NIC and SCSI PV drivers anyone will need to write”… but they
keep on coming!

I have exchanged a few posts with the xen-devel list and now have some ideas
about how a multiple irq scenario might work. I’m trying to come up with a
solution to another problem at the moment (virtualised TPR writes are slow),
so it’s something I’ll have to get back to.

James

---

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

–
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.

[James_Harper]

(corrected top posting so that this thread actually makes sense…)

> >
> > BTW, I’m not saying things CAN’T be made to work with one
interrupt…
> > obviously, there are several vendors who’ve written numerous
flavors
> > of Linux PV drivers already. Some of these even perform
> > exceptionally well.
>
> Under Linux you can do whatever you want (eg rewrite whole sections
> of the
> kernel if required), so there are none of these limitations

I find the whole “In linux you can…” to be inappropriate. It
doesn’t
make
sense in general, and even discussing it in a Windows group is silly
beyond
comprehension. Sure, you can, and then you become a maintainer of the
only
version of linux that can run your driver. From a business viewpoint,
this
sucks. In the entire history I have been involved with Unix, it has
never
made sense to modify the kernel to run something that was intended for
distribution beyond the machine so modified. I find the whole “you
can
change anything” argument fallacious in the extreme. The consequences
of
these actions are profound. I even watched projects crash and burn
because
kernel modifications were made that meant that two different
programs/drivers could not coexist on the same machine because they
required
mutually-incompatible changes, and they could not run on unmodified
kernels at all.

Well… someone asked about why we can’t take the same approach to
resolve the interrupt limitations as is taken under Linux, and I pointed
out that to do so would require patching the Windows kernel, which isn’t
readily possible beyond some very simple instruction substitutions.

I’m not sure I agree with your argument… The changes required to make
Xen work under Linux were maintained as separate patches for quite a
while, and are now merged into the ‘official’ Linux tree. Linux (as a
Xen guest) can still work without those patches, it’s just that it works
better with them. Early versions of VMWare required updates to the Linux
kernel for optimal behaviour. I think Oracle under some version of Unix
required kernel patches too. It makes things more difficult, but it
doesn’t make failure a certainty.

I’m just about to start writing some code to patch the Windows kernel
(substitute ‘TPR write’ instructions with ‘call my code’ instructions)
which will require modification to the Xen hypervisor (not sure yet if
the hypervisor or my drivers will do the actual patching, but the
hypervisor will need to tell me where those writes are occurring). Based
on discussions I’ve had with the maintainers of that code, unless I do a
really bad job of it my modifications should be accepted and form part
of the next version.

I do agree that this thread is heading towards a not-relevant-to-ntdev
direction though… we’d better finish up before Jan 1, and be sure not
to resort to name-calling and other personal attacks in the meantime

James

[Calvin_Guan-2]

It’s beyond OT but it reminds me that some TOE vendors submitted patch?for Tcp offload?to work in Linux but was rejected by the linux maintainer – calling it “toe is junk, will never make it to the kernel”. Yes, you can patch anything you want, but it’s still a no-go because no oem will take it?b/c it’s out-of-the-box.

Glad that FreeBSD is committed to support it in v8.0. Man, on a Windows system with *multiple* teaming 10Gb ports, TOE definitely has its merit. Well, time will tell.

–
Calvin Guan
Broadcom Corp.
Connecting Everything(r)

----- Original Message ----
From: Joseph M. Newcomer
To: Windows System Software Devs Interest List
Sent: Monday, December 29, 2008 4:56:11 PM
Subject: RE: [ntdev] strange interrupt behaviour when HAL is Standard PC

I find the whole “In linux you can…” to be inappropriate.? It doesn’t make
sense in general, and even discussing it in a Windows group is silly beyond
comprehension.? Sure, you can, and then you become a maintainer of the only
version of linux that can run your driver.? From a business viewpoint, this
sucks.? In the entire history I have been involved with Unix, it has never
made sense to modify the kernel to run something that was intended for
distribution beyond the machine so modified.? I find the whole “you can
change anything” argument fallacious in the extreme.? The consequences of
these actions are profound.? I even watched projects crash and burn because
kernel modifications were made that meant that two different
programs/drivers could not coexist on the same machine because they required
mutually-incompatible changes, and they could not run on unmodified kernels
at all.?
??? ??? ??? ??? ??? joe

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of James Harper
Sent: Monday, December 29, 2008 5:49 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] strange interrupt behaviour when HAL is Standard PC

>
> BTW, I’m not saying things CAN’T be made to work with one interrupt…
> obviously, there are several vendors who’ve written numerous flavors
of
> Linux PV drivers already.? Some of these even perform exceptionally
well.

Under Linux you can do whatever you want (eg rewrite whole sections of the
kernel if required), so there are none of these limitations

>
> I’d be interested to hear what xen-devel says.
>
> As to WHQL compliance… I have never seen an NDIS or SCSI PV driver
that
> didn’t take “considerable liberties” that would be at odds with WHQL
> requirements to achieve good performance.? In our experience here at
OSR
> (which I admit is not entirely definitive in this area) there is no
> alternative.
>
> The whole topic of what the best architectural approach to supporting
> Linux PV within Windows drivers is a really interesting one – if a
rather
> arcane one.? Every time there a Windows NIC and SCSI PV driver pair
are
> written, I say to myself “well, THAT must be the absolutely last pair
of
> Windows NIC and SCSI PV drivers anyone will need to write”… but they
> keep on coming!

I have exchanged a few posts with the xen-devel list and now have some ideas
about how a multiple irq scenario might work. I’m trying to come up with a
solution to another problem at the moment (virtualised TPR writes are slow),
so it’s something I’ll have to get back to.

James

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

–
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

__________________________________________________________________
Yahoo! Canada Toolbar: Search from anywhere on the web, and bookmark your favourite sites. Download it now at
http://ca.toolbar.yahoo.com.

[Ken_Johnson]

Unless you are doing something particularly clever to hide your tracks, that will most likely bugcheck all x64 Windows guests (PatchGuard).

? S

-----Original Message-----
From: James Harper
Sent: Monday, December 29, 2008 19:16
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] strange interrupt behaviour when HAL is Standard PC

(corrected top posting so that this thread actually makes sense…)

> > >
> > > BTW, I’m not saying things CAN’T be made to work with one
interrupt…
> > > obviously, there are several vendors who’ve written numerous
flavors
> > > of Linux PV drivers already. Some of these even perform
> > > exceptionally well.
> >
> > Under Linux you can do whatever you want (eg rewrite whole sections
> > of the
> > kernel if required), so there are none of these limitations
>
> I find the whole “In linux you can…” to be inappropriate. It
doesn’t
> make
> sense in general, and even discussing it in a Windows group is silly
> beyond
> comprehension. Sure, you can, and then you become a maintainer of the
> only
> version of linux that can run your driver. From a business viewpoint,
> this
> sucks. In the entire history I have been involved with Unix, it has
never
> made sense to modify the kernel to run something that was intended for
> distribution beyond the machine so modified. I find the whole “you
can
> change anything” argument fallacious in the extreme. The consequences
of
> these actions are profound. I even watched projects crash and burn
> because
> kernel modifications were made that meant that two different
> programs/drivers could not coexist on the same machine because they
> required
> mutually-incompatible changes, and they could not run on unmodified
> kernels at all.

Well… someone asked about why we can’t take the same approach to
resolve the interrupt limitations as is taken under Linux, and I pointed
out that to do so would require patching the Windows kernel, which isn’t
readily possible beyond some very simple instruction substitutions.

I’m not sure I agree with your argument… The changes required to make
Xen work under Linux were maintained as separate patches for quite a
while, and are now merged into the ‘official’ Linux tree. Linux (as a
Xen guest) can still work without those patches, it’s just that it works
better with them. Early versions of VMWare required updates to the Linux
kernel for optimal behaviour. I think Oracle under some version of Unix
required kernel patches too. It makes things more difficult, but it
doesn’t make failure a certainty.

I’m just about to start writing some code to patch the Windows kernel
(substitute ‘TPR write’ instructions with ‘call my code’ instructions)
which will require modification to the Xen hypervisor (not sure yet if
the hypervisor or my drivers will do the actual patching, but the
hypervisor will need to tell me where those writes are occurring). Based
on discussions I’ve had with the maintainers of that code, unless I do a
really bad job of it my modifications should be accepted and form part
of the next version.

I do agree that this thread is heading towards a not-relevant-to-ntdev
direction though… we’d better finish up before Jan 1, and be sure not
to resort to name-calling and other personal attacks in the meantime

James

—
NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

[Maxim_S_Shatskih]

> distribution beyond the machine so modified. I find the whole "you can

change anything" argument fallacious in the extreme.

You can change anything, but the chances Linus will include it in the official branch of his kernel are so smaaall… the same small as to influence the design or API of next Windows version.

kernel modifications were made that meant that two different
programs/drivers could not coexist on the same machine because they required
mutually-incompatible changes

Yes, and the Linux maintaners also know this very well. That’s why it will be hard to commit your hackery to the official distro.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

[Maxim_S_Shatskih]

>Glad that FreeBSD is committed to support it in v8.0. Man, on a Windows system with *multiple*

What is amazing for me that Windows has ToE support (to some degree) since at least w2k, Linux and FreeBSD still have none, and there are myth believers that consider these UNIXen to be faster then Windows.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

[Maxim_S_Shatskih]

>(substitute ‘TPR write’ instructions with ‘call my code’ instructions)

Custom HAL with custom KeRaiseIrql would help

which will require modification to the Xen hypervisor (not sure yet if

Well, what I see here is the issue is a minor bit of misarchitecture in Xen.

Task: implement a generic way of IO request passing from guest to host under Xen, which will be able to support different kinds of emulated hardware in guests.

Proper solution: add message passing feature to Xen hypervisor using the “hypervisor call” primitive, then add message server to the host and message-sending client to the guest.

Improper solution as I see it: introduce an emulated PCI device with an interrupt to guests to use for this message-passing.

Actually, what is going on is the usage of emulated PCI device with BARs, config space, interrupt etc to implement a hypervisor call.

Isn’t this bad? implementing it as an invalid CPU opcode is simpler and more natural.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com