Question about interrupt levels

[OSR_Community_User]

Hi, People,

This is an off-beat request, but hey, I specialize in that kind of stuff.

One of my chips came from the fab with its interrupt line disconnected.
However, the chip can easily write to system memory space from its dma
command queue, so, I generate interrupts by having its dma “channel program”
to poke at the Local APIC’s Interrupt Register and generate an MSI interrupt
at the end of my dma. The channel program also writes the chip’s interrupt
status register to a location in my device extension, which is eventually
examined by the DPC.

This works very nice, well, sort of. I bumped into a problem: the interrupt
comes edge-triggered even if I set it up to be level-triggered, I just read
it in the Intel small print. If I let IoConnectInterrupt allocate the
interrupt vector as “shared”, my 64-bit WinXP shares my interrupt vector
with the on-motherboard network card, and I quickly lose my network because
of the edge triggered interrupt on a shared level-triggered vector. If I do
not share that line, the system runs out of resources and fails to allocate
an interrupt vector to the network card. Damned if I do, damned if I don’t!

What I want is to allocate some lonely vector somewhere out of the reach of
the PCI bus, and dedicate it to this weird scheme. I cannot see any other
way to fabricate an interrupt, and I do not want to resort to polling to
handle my dma completions. Do any of you have any suggestions ?

Thanks!

Alberto.

[OSR_Community_User]

Alberto:

Have you looked at IoConnectInterruptEx for CONNECT_FULLY_SPECIFIED? I
don’t know enough about this subject to say whether it will address your
problem, but the documentation for FULLY_SPECIFIED seems like it might
be worth a look, if you haven’t already, as it allows you to specify a
vector, mode and whether it is shareably. Although it is listed as
Vista, there is a note about it on other platforms:

“Versions: Available on Windows Vista and later versions of the Windows
operating system. Drivers that must also work on Windows 2000, Windows
XP, or Windows Server 2003 can instead link to iointex.lib to use the
routine.”

Good luck,

mm

Alberto Moreira wrote:

Hi, People,

This is an off-beat request, but hey, I specialize in that kind of stuff.

One of my chips came from the fab with its interrupt line disconnected.
However, the chip can easily write to system memory space from its dma
command queue, so, I generate interrupts by having its dma “channel
program” to poke at the Local APIC’s Interrupt Register and generate an
MSI interrupt at the end of my dma. The channel program also writes the
chip’s interrupt status register to a location in my device extension,
which is eventually examined by the DPC.

This works very nice, well, sort of. I bumped into a problem: the
interrupt comes edge-triggered even if I set it up to be
level-triggered, I just read it in the Intel small print. If I let
IoConnectInterrupt allocate the interrupt vector as “shared”, my 64-bit
WinXP shares my interrupt vector with the on-motherboard network card,
and I quickly lose my network because of the edge triggered interrupt on
a shared level-triggered vector. If I do not share that line, the system
runs out of resources and fails to allocate an interrupt vector to the
network card. Damned if I do, damned if I don’t!

What I want is to allocate some lonely vector somewhere out of the reach
of the PCI bus, and dedicate it to this weird scheme. I cannot see any
other way to fabricate an interrupt, and I do not want to resort to
polling to handle my dma completions. Do any of you have any suggestions ?

Thanks!

Alberto.

[anton_bassov]

Alberto,

I am afraid you are going to fail on x64 because of the PatchGuard - it would not allow you to
modify IDT, would it??? Furthermore, even if you try it on 32-bit XP, there is no guarantee that it will work on any machine. Don’t forget that not all motherboards are equipped with IOAPIC - if the target motherboard does not support IOAPIC, the local APICs on all CPUs are going to be disabled in BIOS settings, so that you will have no chance to take advantage of the whole thing.

To summarize, I would not do it, unless I target some known-in-advance machine(s) with known-in-advance configuration…

Anton Bassov

[OSR_Community_User]

What you need is MSI. The MSI support in Vista is exactly what you have described. So if you were running on Vista, you would fabricate an interrupt resource requirement that looked like an MSI. You would then use IoConnectInterruptEx using CONNECT_MESSAGE_BASED, and the output of that routine would give you an address and data that can be used to trigger the interrupt via your DMA engine. Because it’s an MSI, the ISR would be connected properly for edge triggered mode.

I don’t think you can do it on XP.

Dave

[anton_bassov]

> What you need is MSI.

Exactly. Once XP does not support it, Alberto asks if it makes sense to implement it yourself…

I don’t think you can do it on XP.

Sure you can. However, there are quite a few small nasty details here. Therefore, I would not do it, unless my target machine is known in advance…

Anton Bassov

[OSR_Community_User]

Anton, and guys,

I don’t need to modify the IDT. If you want details, take a look at the
Intel IA32 Systems Programming Guide, Volume 1: section 8.6 teaches how to
generate interrupts inside the processor. You have to write a quadword to
the Interrupt Command Register which includes the vector the interrupt
should use.

I have actually used this in several machines, both 32 and 64 bit, under XP,
Vista and 2003, and so far - knock on wood - no problems. Except of course
this issue of sharing interrupts.

Since I wrote my other post yesterday, I found a few things. I first tried
to tell IoConnectInterrupt to give me the vector nonshared, and the
consequence was, the network subsystem couldn’t allocate a vector and it
wouldn’t come up. I then found that if I really tightened the window between
the point my dma engine writes the hw status to the device extension and the
time I clear it, the probability of the network to vector an interrupt
through my chip’s ISR dropped down to negligible. By the way, an
InterlockedAnd call gave me the ideal way to clear the hw status while
reading it to test if the interrupt is mine!

It also looks like I can configure the interrupt vector as LevelSensitive
and yet give it an edge at interrupt time - it seems that, well, an edge is
followed by a level anyway, so, the hardware’s happy and I don’t seem to
lose interrupts. My skeletons and brain aneurisms tumble happily and I
haven’t experienced a problem yet, both under Vista 32-bit and XP64 which
are my main targets at this moment.

Last but not least, I have added a Registry entry to my driver: if
MsiInterrupts = 1, it generates this fake MSI interrupt. If it is zero, I
have a kernel-side thread that polls the chip every n milliseconds (also
configurable through a Registry entry) for completions. And if the user sets
MsiInterrupts = 2, I generate IPI interrupts instead - yes, it works! So, if
the local APIC isn’t accessible, the polling will pick up the completions.
It’s a trifle slower, but hey, it seems to work!

I may have to add a tweak or two to make it run on AMD platforms, I haven’t
tried it yet.

Alberto.

----- Original Message -----
From:
To: “Windows System Software Devs Interest List”
Sent: Monday, November 26, 2007 11:45 PM
Subject: RE:[ntdev] Question about interrupt levels

> Alberto,
>
> I am afraid you are going to fail on x64 because of the PatchGuard - it
> would not allow you to
> modify IDT, would it??? Furthermore, even if you try it on 32-bit XP,
> there is no guarantee that it will work on any machine. Don’t forget that
> not all motherboards are equipped with IOAPIC - if the target motherboard
> does not support IOAPIC, the local APICs on all CPUs are going to be
> disabled in BIOS settings, so that you will have no chance to take
> advantage of the whole thing.
>
> To summarize, I would not do it, unless I target some known-in-advance
> machine(s) with known-in-advance configuration…
>
> Anton Bassov
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

[OSR_Community_User]

Hi, Martin,

Yes, I’m going to try it next. I got to a point where I no longer hog the
network, but it’s a bit of an iffy implementation, so, I’ll see if I can use
CONNECT_FULLY_SPECIFIED and make it work somehow. This week is a bit iffy
because my people are out in Chicago, as we speak, attending the major event
for our industry, which is the annual conference of the Radiological Society
of North America. My boss and my VP are both at our RSNA booth, and I am on
call to fix whatever problems they find during the demos, so, I don’t know
if I’ll have spare cycles to try something right now! But as soon as people
come back and the pressure abates, I’ll let you guys know how it went.

Thanks for the suggestion!

Alberto.

----- Original Message -----
From: “Martin O’Brien”
Newsgroups: ntdev
To: “Windows System Software Devs Interest List”
Sent: Monday, November 26, 2007 11:41 PM
Subject: Re:[ntdev] Question about interrupt levels

> Alberto:
>
> Have you looked at IoConnectInterruptEx for CONNECT_FULLY_SPECIFIED? I
> don’t know enough about this subject to say whether it will address your
> problem, but the documentation for FULLY_SPECIFIED seems like it might be
> worth a look, if you haven’t already, as it allows you to specify a
> vector, mode and whether it is shareably. Although it is listed as Vista,
> there is a note about it on other platforms:
>
> “Versions: Available on Windows Vista and later versions of the Windows
> operating system. Drivers that must also work on Windows 2000, Windows XP,
> or Windows Server 2003 can instead link to iointex.lib to use the
> routine.”
>
> Good luck,
>
> mm
>
>
>
>
>
>
>
>
> Alberto Moreira wrote:
>> Hi, People,
>>
>> This is an off-beat request, but hey, I specialize in that kind of stuff.
>>
>> One of my chips came from the fab with its interrupt line disconnected.
>> However, the chip can easily write to system memory space from its dma
>> command queue, so, I generate interrupts by having its dma “channel
>> program” to poke at the Local APIC’s Interrupt Register and generate an
>> MSI interrupt at the end of my dma. The channel program also writes the
>> chip’s interrupt status register to a location in my device extension,
>> which is eventually examined by the DPC.
>>
>> This works very nice, well, sort of. I bumped into a problem: the
>> interrupt comes edge-triggered even if I set it up to be level-triggered,
>> I just read it in the Intel small print. If I let IoConnectInterrupt
>> allocate the interrupt vector as “shared”, my 64-bit WinXP shares my
>> interrupt vector with the on-motherboard network card, and I quickly lose
>> my network because of the edge triggered interrupt on a shared
>> level-triggered vector. If I do not share that line, the system runs out
>> of resources and fails to allocate an interrupt vector to the network
>> card. Damned if I do, damned if I don’t!
>>
>> What I want is to allocate some lonely vector somewhere out of the reach
>> of the PCI bus, and dedicate it to this weird scheme. I cannot see any
>> other way to fabricate an interrupt, and I do not want to resort to
>> polling to handle my dma completions. Do any of you have any suggestions
>> ?
>>
>> Thanks!
>>
>>
>> Alberto.
>>
>>
>>
>>
>>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

[OSR_Community_User]

The problem, of course, is XP, I don’t know to what extent they support the
functionality provided by IoConnectInterruptEx on that compatibility
library, I forget its name. I’m going to try it next, maybe I’ll strike gold
? This is a jolly complicated chip, just to give you an idea, I can have
four chips on a board running on a 4-processor configuration, each with 4Mb
of on-board memory. So, you can easily imagine, much of my work these days
is to circumvent hardware problems! The VMetro and the Logic Analyzer are my
best friends.

I find it interesting that apparently the people who designed the DDK and
WDK seem to assume that chips will work fine every time! I wish there were a
few bits and pieces here and there to help people like me, who has to run
things on chips that aren’t always running 100% correct. When I wrote OpenGL
ICDs for a living it was the same thing: chips are a fickle lot.

Thanks for the suggestion!

Alberto.

----- Original Message -----
From:
To: “Windows System Software Devs Interest List”
Sent: Tuesday, November 27, 2007 12:43 AM
Subject: RE:[ntdev] Question about interrupt levels

> What you need is MSI. The MSI support in Vista is exactly what you have
> described. So if you were running on Vista, you would fabricate an
> interrupt resource requirement that looked like an MSI. You would then
> use IoConnectInterruptEx using CONNECT_MESSAGE_BASED, and the output of
> that routine would give you an address and data that can be used to
> trigger the interrupt via your DMA engine. Because it’s an MSI, the ISR
> would be connected properly for edge triggered mode.
>
> I don’t think you can do it on XP.
>
> Dave
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

[anton_bassov]

Alberto,

I don’t need to modify the IDT. If you want details, take a look at the Intel IA32
Systems Programming Guide, Volume 1: section 8.6 teaches how to generate
interrupts inside the processor. You have to write a quadword to the Interrupt
Command Register which includes the vector the interrupt should use.

In fact, you don’t have to tell me how to raise interrupts via ICR - I did it quite a few times myself, on one occasion even in a commercial product ( on this occasion I had an explicit permission from my client to assume that the target machine runs MP APIC HAL on x86-based system, so that I found the whole thing really enjoyable). Therefore, my objections stand as following:

1. Interrupts get raised via ICR by the *software*, rather than hardware ( actually, this is how HalRequestSoftwareInterrupt() works on APIC HAL ). Interrupt, raised by writing to ICR, does not get accepted right on the spot. This is why ICR has DeliveryStatus field - it indicates whether you can write to ICR right on the spot, or wait until pending interrupt gets accepted, i.e. DeliveryStatus bit gets cleared. Every CPU in the system has its own ICR, and HalRequestSoftwareInterrupt() is frequently-used function, For example, it gets called every time you queue a DPC (although not every invocation of HalRequestSoftwareInterrupt() results in write to ICR), and this function does not expect anyone, apart from CPU it currently runs on, to access the given CPU’s ICR.

I hope by now you understand that making some bus agent, other than CPU X, write to CPU X’s ICR is really a no-brainer - what is going to happen if this writes somehow collides with the one done by HalRequestSoftwareInterrupt() ??? Therefore, you should not use ICR if you want to raise interrupt in the hardware. Instead, you should use MSI - your MSI-capable hardware has to write contents of the Message Data register to the address, specified by the Message Address register, i.e. ICR should be completely out of play here. Certainly, XP does not support MSI, so that you have to do some extra work here - it is really good to feel yourself like an OS designer, rather than just driver writer, don’t you think?

2. No matter how interrupt gets raised, it has to get handled somehow. Therefore, if you want to use some currently unused vector, you have to register its handler in IDT, so that you have to modify IDT anyway. Certainly, you can try to use IoConnectInterrupt() for this purpose, but I believe it is a bad idea - if you feel in a position to play around with HAL, then you should do everything yourself from the very beginning to the very end, rather than partly relying on OS-provided functions…

Anton Bassov

[OSR_Community_User]

I hope it helps. Just out of curiosity, where did this post come from?
I just noticed it in Thunderbird, somewhere in the middle of that
absurd thread about RAZZLE.CMD, which is kind of weird. I guess it was
always there, and I just hadn’t noticed it before. Not that it matters,
but just sort of wondering.

Good luck,

mm

Alberto Moreira wrote:

Hi, Martin,

Yes, I’m going to try it next. I got to a point where I no longer hog
the network, but it’s a bit of an iffy implementation, so, I’ll see if I
can use CONNECT_FULLY_SPECIFIED and make it work somehow. This week is a
bit iffy because my people are out in Chicago, as we speak, attending
the major event for our industry, which is the annual conference of the
Radiological Society of North America. My boss and my VP are both at our
RSNA booth, and I am on call to fix whatever problems they find during
the demos, so, I don’t know if I’ll have spare cycles to try something
right now! But as soon as people come back and the pressure abates, I’ll
let you guys know how it went.

Thanks for the suggestion!

Alberto.

----- Original Message ----- From: “Martin O’Brien”

> Newsgroups: ntdev
> To: “Windows System Software Devs Interest List”
> Sent: Monday, November 26, 2007 11:41 PM
> Subject: Re:[ntdev] Question about interrupt levels
>
>
>> Alberto:
>>
>> Have you looked at IoConnectInterruptEx for CONNECT_FULLY_SPECIFIED?
>> I don’t know enough about this subject to say whether it will address
>> your problem, but the documentation for FULLY_SPECIFIED seems like it
>> might be worth a look, if you haven’t already, as it allows you to
>> specify a vector, mode and whether it is shareably. Although it is
>> listed as Vista, there is a note about it on other platforms:
>>
>> “Versions: Available on Windows Vista and later versions of the
>> Windows operating system. Drivers that must also work on Windows 2000,
>> Windows XP, or Windows Server 2003 can instead link to iointex.lib to
>> use the
>> routine.”
>>
>> Good luck,
>>
>> mm
>>
>>
>>
>>
>>
>>
>>
>>
>> Alberto Moreira wrote:
>>> Hi, People,
>>>
>>> This is an off-beat request, but hey, I specialize in that kind of
>>> stuff.
>>>
>>> One of my chips came from the fab with its interrupt line
>>> disconnected. However, the chip can easily write to system memory
>>> space from its dma command queue, so, I generate interrupts by having
>>> its dma “channel program” to poke at the Local APIC’s Interrupt
>>> Register and generate an MSI interrupt at the end of my dma. The
>>> channel program also writes the chip’s interrupt status register to a
>>> location in my device extension, which is eventually examined by the
>>> DPC.
>>>
>>> This works very nice, well, sort of. I bumped into a problem: the
>>> interrupt comes edge-triggered even if I set it up to be
>>> level-triggered, I just read it in the Intel small print. If I let
>>> IoConnectInterrupt allocate the interrupt vector as “shared”, my
>>> 64-bit WinXP shares my interrupt vector with the on-motherboard
>>> network card, and I quickly lose my network because of the edge
>>> triggered interrupt on a shared level-triggered vector. If I do not
>>> share that line, the system runs out of resources and fails to
>>> allocate an interrupt vector to the network card. Damned if I do,
>>> damned if I don’t!
>>>
>>> What I want is to allocate some lonely vector somewhere out of the
>>> reach of the PCI bus, and dedicate it to this weird scheme. I cannot
>>> see any other way to fabricate an interrupt, and I do not want to
>>> resort to polling to handle my dma completions. Do any of you have
>>> any suggestions ?
>>>
>>> Thanks!
>>>
>>>
>>> Alberto.
>>>
>>>
>>>
>>>
>>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>
>

[Jake_Oshins]

Actually, Anton, Alberto isn’t trying to raise an interrupt with the ICR.
He’s programming his device to generate an MSI. And he thinks he doesn’t
need to modify the IDT either, since he’s just trying to use the vector
(level-triggered, shared) that the PnP manager allotted him.

This works, sort of. As Alberto says, he has managed to get the race
conditions inherent in his approach to “almost negligible” levels. There
are still race conditions, and his approach will still hang the machine
occasionally. But he seems comfortable with that.

I’m actually surprised, given all the posts that he’s made in the past, that
he doesn’t feel more comfortable using his own IDT vector and eliminating
the races entirely.

• Jake

wrote in message news:xxxxx@ntdev…
> Alberto,
>
>> I don’t need to modify the IDT. If you want details, take a look at the
>> Intel IA32
>> Systems Programming Guide, Volume 1: section 8.6 teaches how to generate
>> interrupts inside the processor. You have to write a quadword to the
>> Interrupt
>> Command Register which includes the vector the interrupt should use.
>
> In fact, you don’t have to tell me how to raise interrupts via ICR - I did
> it quite a few times myself, on one occasion even in a commercial product
> ( on this occasion I had an explicit permission from my client to assume
> that the target machine runs MP APIC HAL on x86-based system, so that I
> found the whole thing really enjoyable). Therefore, my objections stand as
> following:
>
> 1. Interrupts get raised via ICR by the software, rather than hardware
> ( actually, this is how HalRequestSoftwareInterrupt() works on APIC HAL ).
> Interrupt, raised by writing to ICR, does not get accepted right on the
> spot. This is why ICR has DeliveryStatus field - it indicates whether you
> can write to ICR right on the spot, or wait until pending interrupt gets
> accepted, i.e. DeliveryStatus bit gets cleared. Every CPU in the system
> has its own ICR, and HalRequestSoftwareInterrupt() is frequently-used
> function, For example, it gets called every time you queue a DPC (although
> not every invocation of HalRequestSoftwareInterrupt() results in write to
> ICR), and this function does not expect anyone, apart from CPU it
> currently runs on, to access the given CPU’s ICR.
>
> I hope by now you understand that making some bus agent, other than CPU X,
> write to CPU X’s ICR is really a no-brainer - what is going to happen if
> this writes somehow collides with the one done by
> HalRequestSoftwareInterrupt() ??? Therefore, you should not use ICR if you
> want to raise interrupt in the hardware. Instead, you should use MSI -
> your MSI-capable hardware has to write contents of the Message Data
> register to the address, specified by the Message Address register, i.e.
> ICR should be completely out of play here. Certainly, XP does not support
> MSI, so that you have to do some extra work here - it is really good to
> feel yourself like an OS designer, rather than just driver writer, don’t
> you think?
>
> 2. No matter how interrupt gets raised, it has to get handled somehow.
> Therefore, if you want to use some currently unused vector, you have to
> register its handler in IDT, so that you have to modify IDT anyway.
> Certainly, you can try to use IoConnectInterrupt() for this purpose, but I
> believe it is a bad idea - if you feel in a position to play around with
> HAL, then you should do everything yourself from the very beginning to the
> very end, rather than partly relying on OS-provided functions…
>
> Anton Bassov
>

[anton_bassov]

Jake,

Actually, Anton, Alberto isn’t trying to raise an interrupt with the ICR. He’s programming
his device to generate an MSI.

This is what I thought after having read his *original* post. However, the subsequent one came to the contradiction with the above assumption - he made it clear that he intends to use ICR for this purpose. Look at what he said in it:

[begin quote]

…if you want details, take a look at the Intel IA32 Systems Programming Guide, Volume 1: section 8.6 teaches how to generate interrupts inside the processor. You have to write a quadword to the Interrupt Command Register which includes the vector the interrupt should use.

[end quote]

As you can see, in the second post he already speaks about ICR…

And he thinks he doesn’t need to modify the IDT either, since he’s just trying to use the vector > (level-triggered, shared) that the PnP manager allotted him.

Again, the same story - the intention to use system-allocated vector turned up only in the subsequent posts. However, in his *original* one he spoke about using some unused vector (which is easily understandable - after all, re-using a vector that has been allocated by PnP simply defeats the point of the whole exercise, in the first place). Look at what he originally said:

[begin quote]

What I want is to allocate some lonely vector somewhere out of the reach of the PCI bus, and dedicate it to this weird scheme.

[end quote]

I’m actually surprised, given all the posts that he’s made in the past, that he doesn’t feel more > comfortable using his own IDT vector and eliminating the races entirely.

I think he just feels uneasy - on one hand, he wants to make it work, but, on the other hand, he, apparently, does not want to do something that you, guys, want to be “definitely no-no” for us, driver writers…

Anton Bassov

[OSR_Community_User]

Jake, Anton, Guys,

Anton is right about my intentions. I’m not trying to just generate an MSI
interrupt, I’m trying to program the ICR to generate an interrupt: because
this particular chip is incapable of generating interrupts. I did not want
to directly trap the IDT for several reasons. I wouldn’t know which vector
to use that would guarantee exclusivity and yet be portable to all machines
under all HALs. I would have to go through hoops to make it work in 64-bit
Windows. I wanted to keep the current driver structure intact and not to
insert a facility that might not map easily into the Linux and Solaris
versions of the driver. My my other chip versions can generate interrupts ok
and don’t need this kluge. Last but not least, this environment is so
render-bound that performance at interrupt time isn’t that relevant anyway,
and the driver is large enough and it does lots more than handling dma and
interrupts. Why add complexity and a service-intensive facility if I could
find an easier way ?

So, the solution was, try to generate an interrupt that the OS cannot
distinguish from a real MSI interrupt. This was first suggested by my
Japanese CTO, we did some experimentation on our Chip Diagnostics program
(by the way, if you want an ill-behaved program that would make you cringe,
try that one! I put in several rather debatable driver facilities to support
my hardware guys.). It looked pretty decent, so we went that way. Still, I
left the polling code intact so that one can revert to it using a Registry
entry, its main drawback is that the timer latency affects the chip’s
performance.

I since found that the networking issue I had wasn’t due to throwing edge
triggered interrupts at a levelSensitive vector - it works flawlessly from
all I’ve seen so far - but from a window I had during which any interrupt
from the networking card would vector directly into my ISR. This window
arises because there’s a finite amount of time between the moment my
hardware sets the interrupt status in host memory and the time the ISR looks
at the identifier and clears it - the act of generating an interrupt and its
hw state is no longer atomic! This window was rather large, so, I tightened
it up to be only a few cycles, and the networking card is now happy.

The next chips revs do not have this problem, so, I hope that in good time
this will be history!

Alberto.

----- Original Message -----
From:
To: “Windows System Software Devs Interest List”
Sent: Wednesday, November 28, 2007 2:05 AM
Subject: RE:[ntdev] Question about interrupt levels

> Jake,
>
>>Actually, Anton, Alberto isn’t trying to raise an interrupt with the ICR.
>>He’s programming
>> his device to generate an MSI.
>
> This is what I thought after having read his original post. However, the
> subsequent one came to the contradiction with the above assumption - he
> made it clear that he intends to use ICR for this purpose. Look at what he
> said in it:
>
> [begin quote]
>
> …if you want details, take a look at the Intel IA32 Systems Programming
> Guide, Volume 1: section 8.6 teaches how to generate interrupts inside the
> processor. You have to write a quadword to the Interrupt Command Register
> which includes the vector the interrupt should use.
>
> [end quote]
>
> As you can see, in the second post he already speaks about ICR…
>
>> And he thinks he doesn’t need to modify the IDT either, since he’s just
>> trying to use the vector > (level-triggered, shared) that the PnP manager
>> allotted him.
>
> Again, the same story - the intention to use system-allocated vector
> turned up only in the subsequent posts. However, in his original one he
> spoke about using some unused vector (which is easily understandable -
> after all, re-using a vector that has been allocated by PnP simply defeats
> the point of the whole exercise, in the first place). Look at what he
> originally said:
>
> [begin quote]
>
> What I want is to allocate some lonely vector somewhere out of the reach
> of the PCI bus, and dedicate it to this weird scheme.
>
> [end quote]
>
>
>> I’m actually surprised, given all the posts that he’s made in the past,
>> that he doesn’t feel more > comfortable using his own IDT vector and
>> eliminating the races entirely.
>
> I think he just feels uneasy - on one hand, he wants to make it work, but,
> on the other hand, he, apparently, does not want to do something that you,
> guys, want to be “definitely no-no” for us, driver writers…
>
> Anton Bassov
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

[OSR_Community_User]

I’m well aware that this isn’t an ideal solution! I’m using it to enable my
people to use one particular rev of the chip while the next generation
doesn’t come back from the fab. If my chip tries to write to the ICR and
that write hits a window where there’s another interrupt pending, the
interrupt is not going to get generated, but my polling logic will pick up
the completion at the driver’s timer tick, which is usually set to 1
millisecond or so, because the poller bases its decision on the sequence
number register that’s incremented by the dma channel program prior to
generating the interrupt.

In fact, thank you - you may have explained one of the side effects that I
was trying to figure out and so far couldn’t! I noticed that even though
interrupts were being generated ok, I would get timeouts if I turned the
polling timer off, which effectively disables the polling logic. My driver
can recover from timeouts, but that generates a performance hickup. What may
be happening is that the chip writes to the ICR at the wrong time and the
interrupt doesn’t get generated; if the polling logic picks it up, the
completion is handled at the polling timer tick; but if the polling logic is
off, that interrupt is lost.

Alberto.

----- Original Message -----
From:
To: “Windows System Software Devs Interest List”
Sent: Tuesday, November 27, 2007 11:51 PM
Subject: RE:[ntdev] Question about interrupt levels

> Alberto,
>
>> I don’t need to modify the IDT. If you want details, take a look at the
>> Intel IA32
>> Systems Programming Guide, Volume 1: section 8.6 teaches how to generate
>> interrupts inside the processor. You have to write a quadword to the
>> Interrupt
>> Command Register which includes the vector the interrupt should use.
>
> In fact, you don’t have to tell me how to raise interrupts via ICR - I did
> it quite a few times myself, on one occasion even in a commercial product
> ( on this occasion I had an explicit permission from my client to assume
> that the target machine runs MP APIC HAL on x86-based system, so that I
> found the whole thing really enjoyable). Therefore, my objections stand as
> following:
>
> 1. Interrupts get raised via ICR by the software, rather than hardware
> ( actually, this is how HalRequestSoftwareInterrupt() works on APIC HAL ).
> Interrupt, raised by writing to ICR, does not get accepted right on the
> spot. This is why ICR has DeliveryStatus field - it indicates whether you
> can write to ICR right on the spot, or wait until pending interrupt gets
> accepted, i.e. DeliveryStatus bit gets cleared. Every CPU in the system
> has its own ICR, and HalRequestSoftwareInterrupt() is frequently-used
> function, For example, it gets called every time you queue a DPC (although
> not every invocation of HalRequestSoftwareInterrupt() results in write to
> ICR), and this function does not expect anyone, apart from CPU it
> currently runs on, to access the given CPU’s ICR.
>
> I hope by now you understand that making some bus agent, other than CPU X,
> write to CPU X’s ICR is really a no-brainer - what is going to happen if
> this writes somehow collides with the one done by
> HalRequestSoftwareInterrupt() ??? Therefore, you should not use ICR if you
> want to raise interrupt in the hardware. Instead, you should use MSI -
> your MSI-capable hardware has to write contents of the Message Data
> register to the address, specified by the Message Address register, i.e.
> ICR should be completely out of play here. Certainly, XP does not support
> MSI, so that you have to do some extra work here - it is really good to
> feel yourself like an OS designer, rather than just driver writer, don’t
> you think?
>
> 2. No matter how interrupt gets raised, it has to get handled somehow.
> Therefore, if you want to use some currently unused vector, you have to
> register its handler in IDT, so that you have to modify IDT anyway.
> Certainly, you can try to use IoConnectInterrupt() for this purpose, but I
> believe it is a bad idea - if you feel in a position to play around with
> HAL, then you should do everything yourself from the very beginning to the
> very end, rather than partly relying on OS-provided functions…
>
> Anton Bassov
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

[OSR_Community_User]

Alberto, you have your terminology slightly mixed up. The ICR is a register inside the per-processor local APIC. This can only be written by software running on the processor. An MSI, which has an encoding that looks an awful lot like the ICR encoding, is sent to a different address and is handled by a different feature of the processor. If you are using your DMA engine to send a PCI write to an address in order to generate an interrupt, you aren’t targeting the ICR.

Dave

[OSR_Community_User]

You are right that the ICR is only used to generate IPI interrupts. I rather
write to the Message Address Register when I want to generate MSI
interrupts: my Registry entry allows me to choose which one to generate, IPI
or MSI. The access works like a charm, looks to me like once the bridge puts
the transaction on the front-side bus, some processor decodes it and its
APIC runs the access. When I target the ICR, I get an IPI interrupt, when I
taret the Message Register I get an MSI interrupt!

Alberto.

----- Original Message -----
From:
To: “Windows System Software Devs Interest List”
Sent: Wednesday, November 28, 2007 12:57 PM
Subject: RE:[ntdev] Question about interrupt levels

> Alberto, you have your terminology slightly mixed up. The ICR is a
> register inside the per-processor local APIC. This can only be written by
> software running on the processor. An MSI, which has an encoding that
> looks an awful lot like the ICR encoding, is sent to a different address
> and is handled by a different feature of the processor. If you are using
> your DMA engine to send a PCI write to an address in order to generate an
> interrupt, you aren’t targeting the ICR.
>
> Dave
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer