MSI interrupt

OSR_Community_User · July 20, 2011, 5:48am

Hi,

I am writing a miniport driver for HBA which supports MSI(multi message MSI - 32 msgs)

I have some doubts regarding MSI.

I am requesting for 32 message in the .inf file. But still only 16 messages are allocated. Why cant i get 32 messages? Is this a limitation?
I have registered my MSI_ISR as follows:
pConfigInfo->HwMSInterruptRoutine = MSI_ISR;
pConfigInfo->InterruptSynchronizationMode = InterruptSynchronizePerMessage;

Is this correct? or Do i need to use IoConnectInterruptEx? What is the use of IoConnectInterruptEx when i have already registered my MSI_ISR in port_config_data?

I have used InterruptSynchronizePerMessage so it means MSI_ISR call is synchronized on message basis and if i am accessing some shared data i should use the StorPortAcquireMSISpinLock. correct?

But StorPortAcquireMSISpinLock call is not getting success in my MSI_ISR() function.
ULONG OldIrql = 0x00;
StorPortAcquireMSISpinLock(pDevExt, MessageID, &OldIrql);

Thanks

gokul_tv · July 20, 2011, 6:27am

How did you find only 16 messages are allocated? Can you share the
portion of inf that you request the number of messages.
That’s correct. You do-not want to use IoConnectInterruptEx.
What is the error status? Or are you hitting a deadlock?

>StorPortAcquireMSISpinLock(pDevExt, MessageID, &OldIrql);
Is it the exact cut & paste from your code. Then you will definitely hit the
deadlock. You cannot acquire the same MSI spinlock from the same (MSI_ISR)
thread.

Regards,
Gokul T V

OSR_Community_User · July 20, 2011, 6:59am

I checked by calling StorPortGetMSiInfo in a loop with message id = 0 …31. For messageid =16,
i got the return value of 0xc1000007

Here is the inf file

HKR,“Interrupt Management”,0x00000010
HKR,“Interrupt Management\MessageSignaledInterruptProperties”, 0x00000010
HKR,“Interrupt Management\MessageSignaledInterruptProperties”, MSISupported,0x00010001,1
HKR,“Interrupt Management\MessageSignaledInterruptProperties”, MessageNumberLimit,0x00010001,32

Then what is the benefit of IoConnectInterruptEx?
No, there is no deadlock. It doesn’t return success status for StorPortAcquireMSISpinLock call. Yes that is the code from my driver. What is that i am doing wrong here? How can i use StorPortAcquireMSISpinLock then?

Thanks

gokul_tv · July 20, 2011, 7:30am

I do-not see any issue with the inf. Check whether the PCI function is
asking for 32 messages (IIRC, it is placed in PCI config - MSIx
capabilities). Get help from your hardware guys.
The IoConnectInterruptEx API is for WDM driver. The storport miniport
driver should use only StorportXXX APIs.
As per the MSDN doc, in case of errors the StorPortAcquireMSISpinLock API
either returns STOR_STATUS_INVALID_PARAMETER or STOR_STATUS_NOT_IMPLEMENTED.
If it is STOR_STATUS_INVALID_PARAMETER, check whether HwDevExt is NULL?
BTW, what is the Windows OS Version & storport version? Did you apply any
storport hotfixes?

Regards,
Gokul T V

On Wed, Jul 20, 2011 at 4:29 PM, wrote:

> 1.
> I checked by calling StorPortGetMSiInfo in a loop with message id = 0
> …31. For messageid =16,
> i got the return value of 0xc1000007
>
> Here is the inf file
>
> HKR,“Interrupt Management”,0x00000010
> HKR,“Interrupt Management\MessageSignaledInterruptProperties”, 0x00000010
> HKR,“Interrupt Management\MessageSignaledInterruptProperties”,
> MSISupported,0x00010001,1
> HKR,“Interrupt Management\MessageSignaledInterruptProperties”,
> MessageNumberLimit,0x00010001,32
>
> 2. Then what is the benefit of IoConnectInterruptEx?
>
> 3. No, there is no deadlock. It doesn’t return success status for
> StorPortAcquireMSISpinLock call. Yes that is the code from my driver. What
> is that i am doing wrong here? How can i use StorPortAcquireMSISpinLock
> then?
>
> Thanks
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

OSR_Community_User · July 20, 2011, 10:35am

Gokul,

I will check with the capability registers and see how many message has been allocated for my device.

In the below link they mention that a max of message allocation depends on ratio of cores/threads.
Whether this would be the reason for less allocation of messages?

http://208.13.232.222/showthread.cfm?link=198541

I am working on Windows 7 with no storport fixes. No, Device Extension is not NULL.

Thanks

gokul_tv · July 20, 2011, 12:37pm

Quite possible (thanks for sharing the link). Anyhow its worth checking
with your hardware team.
What is the return value of the StorPortAcquireMSISpinlock?

Regards,
Gokul T V

On Wed, Jul 20, 2011 at 8:04 PM, wrote:

> Gokul,
>
> 1. I will check with the capability registers and see how many message has
> been allocated for my device.
>
> In the below link they mention that a max of message allocation depends on
> ratio of cores/threads.
> Whether this would be the reason for less allocation of messages?
>
> http://208.13.232.222/showthread.cfm?link=198541
>
> 3. I am working on Windows 7 with no storport fixes. No, Device Extension
> is not NULL.
>
> Thanks
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

Mark_Johnson · July 20, 2011, 6:21pm

There is a simple, definitive answer to your problem. Windows only supports 16 MSI Interrupt messages, even though the PCI hardware spec allows for 32. But if your device supports MSI-X, Windows supports up to 2048 (provided you are using Vista or beyond, of course).

gokul_tv · July 21, 2011, 1:14am

Mark,

IIRC for MSI only one message alloted per PCI function? The OP mentions, he
is seeing 16 messages assigned for his PCI function - then it should MSIx,
right?

Regards,
Gokul T V

On Thu, Jul 21, 2011 at 3:49 AM, wrote:

> There is a simple, definitive answer to your problem. Windows only
> supports 16 MSI Interrupt messages, even though the PCI hardware spec allows
> for 32. But if your device supports MSI-X, Windows supports up to 2048
> (provided you are using Vista or beyond, of course).
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

OSR_Community_User · July 21, 2011, 4:21am

Gokul,

It’s not MSI-x. PCI support multi message MSI … upto 32 messages.

In MSI-X, it supports upto 2048 message.

Thanks

gokul_tv · July 21, 2011, 5:23am

Is you card PCI 3.0 compliant or PCI V2.2? I remember reading a MSFT doc
that says, for PCI v2.2 compliant adapters with multi-message support -
Windows vista & above support a maximum of 16 messages (although the PCI
v2.2 spec support 32 per device).

But in regard to storport, the Microsoft WinHEC2007 “NUMA IO Optimization”*
pptx (Slide-18), it is mentioned as follows,

“Windows driver support for MSI and MSI-X is identical”
“Windows will allocate one message if it cannot provide the specified
number”

*
http://download.microsoft.com/download/a/f/d/afdfd50d-6eb9-425e-84e1-b4085a80e34e/svr-t332_wh07.pptx

Since you said you are seeing 16 messages allocated - I asked you to verify
the PCI MSIx capabilities. BTW still you have not answered by 3rd question -
what is the error value you get for StorPortAcquireMSISpinLock.

Regards,
Gokul T V

Peter_Viscarola_OSR · July 21, 2011, 9:10am

> IIRC for MSI only one message alloted per PCI function

Nope. With EITHER MSI or MSI-x you can have multiple messages per PCIe function.

MSI-x adds the ability to target messages to specific processors, adds a higher message max, etc.

Peter
OSR

Mark_Johnson · July 21, 2011, 6:09pm

Gokul: I recently ran across definitive documentation indicating that MSI/MSI-X support is only available with Vista and later OSes, and that even though the PCIe docs indicate 32 max MSI messages, Windows limits MSI messages to 16 (but supports up to 2048 MSI-X messages, go figure). So your observations seem to confirm the documentation.

Peter_Viscarola_OSR · July 21, 2011, 6:28pm

This is absolutely correct. The support was coded during the Vista development cycle. As I understand it, it was considered for a potential back-port, but generally judged to be too big a change.

Peter
OSR

OSR_Community_User · July 25, 2011, 7:28am

Gokul,

I am unable to get the return value for StorPortAcquireMSISpinLock function.

When i call this function [Ret = StorPortAcquireMSISpinLock(pDevExt, MessageID, &Irql);], it gives the following error:

Assertion: *** DPC watchdog timeout
This is NOT a break in update time
This is most likely a BUG in an ISR
Perform a stack trace to find the culprit
The period will be doubled on continuation
Use gh to continue!!

nt!KeAccumulateTicks+0x3c5:
82a86e78 cd2c int 2Ch

kd> g (Next i tried to continue)
Continuing an assertion failure can result in the debuggee
being terminated (bugchecking for kernel debuggees).
If you want to ignore this assertion, use ‘ahi’.
If you want to force continuation, use ‘gh’ or ‘gn’.

I even check DeviceExtension is not Null and MessageID = 0. What might be the error?

Thanks

Maxim_S_Shatskih · July 25, 2011, 8:15am

> I am unable to get the return value for StorPortAcquireMSISpinLock function.

Deadlock, most probable cause is acquiring the lock twice.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

gokul_tv · July 25, 2011, 8:41am

Maxim is right… its a deadlock. With ‘InterruptSynchronizePerMessage’,
every message has its own unique MSI (interrupt) spinlock. Let’s say you
have 2 messages (message 0 and 1) allocated for your adapter and MISR is
called for message ‘0’ - the storport will acquire the MSISpinLock for
message ‘0’ before invoking miniport’s HwMSInterruptRoutine. If you try to
acquire the same MSISpinlock it will result in deadlock!

So in this case to synchronize across multiple message ISRs,

At message 0 - MISR - acquire message 1’s MSI spinlock.
At message 1 - MISR - acquire message 0’s MSI spinlock.

Regards,
Gokul T V

On Mon, Jul 25, 2011 at 4:56 PM, wrote:

> Gokul,
>
> I am unable to get the return value for StorPortAcquireMSISpinLock
> function.
>
> When i call this function [Ret = StorPortAcquireMSISpinLock(pDevExt,
> MessageID, &Irql);], it gives the following error:
>
> Assertion: *** DPC watchdog timeout
> This is NOT a break in update time
> This is most likely a BUG in an ISR
> Perform a stack trace to find the culprit
> The period will be doubled on continuation
> Use gh to continue!!
>
> nt!KeAccumulateTicks+0x3c5:
> 82a86e78 cd2c int 2Ch
>
> kd> g (Next i tried to continue)
> Continuing an assertion failure can result in the debuggee
> being terminated (bugchecking for kernel debuggees).
> If you want to ignore this assertion, use ‘ahi’.
> If you want to force continuation, use ‘gh’ or ‘gn’.
>
> I even check DeviceExtension is not Null and MessageID = 0. What might be
> the error?
>
> Thanks
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

OSR_Community_User · July 26, 2011, 7:19am

Thanks, I got the lock for message ID other than what came with the MSI_ISR call.

Gokul, in your example, when i am serving Isr with messageid = 0, i can control the access of the shared data structure by acquiring a lock on messageid =1. In this case, there are only 2 options(both mutual exclusive).

But If i am using 16 or more messages, how can i use this?

If you have any ideas, can you please share it. It would be of great help.

Thanks

gokul_tv · July 26, 2011, 8:23am

That’s why I limited with 2 messages in my example

If your design can accomadate, the easy thing would be push the processing
to your completion DPC (and by default it is serialized)

If not one other option is, make all MISRs sync to one particular message
spinlock. To elaborate let us say you have 4 messages,
HwMSInterruptRoutine - message 0 - NONE (Bcoz message 0 lock is acquired by
storport already)

message 1 - Sync to message 0 spinlock.
message 2 - Sync to message 0 spinlock.
message 3 - Sync to message 0 spinlock.

The latter one I’ve not experimented in detail. Let me know if this works or
share if you have some other ideas.

Regards,
Gokul T V