MSI synchronization

I have a driver that most of the time can run fine with N channels each with
an MSI interrupt. For a few rare operations I need to synchronize across
all the MSI interrupts, so far the obvious approach seems to be acquiring
all the interrupt spin locks in order. I’m wondering if anyone has
encountered this problem and has another approach to ensure that the
interrupt service routines and the other code that accesses the hardware is
all quiescent for these rare cases?

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

That’s fine, but make sure that you acquire them all at the highest DIRQL. (MSI messages will all be at the same IRQL, but MSI-X probably won’t be.)

  • Jake Oshins

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Wednesday, October 2, 2013 1:12 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] MSI synchronization

I have a driver that most of the time can run fine with N channels each with
an MSI interrupt. For a few rare operations I need to synchronize across
all the MSI interrupts, so far the obvious approach seems to be acquiring
all the interrupt spin locks in order. I’m wondering if anyone has
encountered this problem and has another approach to ensure that the
interrupt service routines and the other code that accesses the hardware is
all quiescent for these rare cases?

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Jake,

For MSI-X would a prioritized ordering (i.e. lowest IRQL to highest) and
using the reverse order on release be appropriate?

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jake Oshins
Sent: Wednesday, October 02, 2013 4:57 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] MSI synchronization

That’s fine, but make sure that you acquire them all at the highest DIRQL.
(MSI messages will all be at the same IRQL, but MSI-X probably won’t be.)

  • Jake Oshins

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Wednesday, October 2, 2013 1:12 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] MSI synchronization

I have a driver that most of the time can run fine with N channels each with
an MSI interrupt. For a few rare operations I need to synchronize across
all the MSI interrupts, so far the obvious approach seems to be acquiring
all the interrupt spin locks in order. I’m wondering if anyone has
encountered this problem and has another approach to ensure that the
interrupt service routines and the other code that accesses the hardware is
all quiescent for these rare cases?

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Well that depends somewhat on how you’re doing things. The safe approach would involve looking at all your assigned messages and finding the highest IRQL and then connecting all of those interrupts by supplying your own spinlock and then specifying that highest DIRQL as your SynchronizationIrql, which will force the same DIRQL for all the messages. Then pick an order and stick to it.

  • Jake Oshins
    Microsoft Kernel Team

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Wednesday, October 2, 2013 2:03 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] MSI synchronization

Jake,

For MSI-X would a prioritized ordering (i.e. lowest IRQL to highest) and
using the reverse order on release be appropriate?

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jake Oshins
Sent: Wednesday, October 02, 2013 4:57 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] MSI synchronization

That’s fine, but make sure that you acquire them all at the highest DIRQL.
(MSI messages will all be at the same IRQL, but MSI-X probably won’t be.)

  • Jake Oshins

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Wednesday, October 2, 2013 1:12 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] MSI synchronization

I have a driver that most of the time can run fine with N channels each with
an MSI interrupt. For a few rare operations I need to synchronize across
all the MSI interrupts, so far the obvious approach seems to be acquiring
all the interrupt spin locks in order. I’m wondering if anyone has
encountered this problem and has another approach to ensure that the
interrupt service routines and the other code that accesses the hardware is
all quiescent for these rare cases?

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Jake,

Unfortunately this is a Storport so I can’t do some of those things.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jake Oshins
Sent: Wednesday, October 02, 2013 6:19 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] MSI synchronization

Well that depends somewhat on how you’re doing things. The safe approach
would involve looking at all your assigned messages and finding the highest
IRQL and then connecting all of those interrupts by supplying your own
spinlock and then specifying that highest DIRQL as your SynchronizationIrql,
which will force the same DIRQL for all the messages. Then pick an order
and stick to it.

  • Jake Oshins
    Microsoft Kernel Team

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Wednesday, October 2, 2013 2:03 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] MSI synchronization

Jake,

For MSI-X would a prioritized ordering (i.e. lowest IRQL to highest) and
using the reverse order on release be appropriate?

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jake Oshins
Sent: Wednesday, October 02, 2013 4:57 PM
To: Windows System Software Devs Interest List
Subject: RE: [ntdev] MSI synchronization

That’s fine, but make sure that you acquire them all at the highest DIRQL.
(MSI messages will all be at the same IRQL, but MSI-X probably won’t be.)

  • Jake Oshins

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Wednesday, October 2, 2013 1:12 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] MSI synchronization

I have a driver that most of the time can run fine with N channels each with
an MSI interrupt. For a few rare operations I need to synchronize across
all the MSI interrupts, so far the obvious approach seems to be acquiring
all the interrupt spin locks in order. I’m wondering if anyone has
encountered this problem and has another approach to ensure that the
interrupt service routines and the other code that accesses the hardware is
all quiescent for these rare cases?

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

Don,

I’m assuming that you’ve specified InterruptSynchronizePerMessage in your PORT_CONFIGURATION_INFORMATION structure and that you don’t want to use InterruptSynchronizeAll.

I think what you have to do here is call StorPortAcquireSpinLock and request the InterruptLock. In other words, don’t try to use the MSI per message locks to do this, use the global lock to protect whatever resource they are sharing. I could be gauging this wrong because I’m not sure of the exact use case, but it seems like the message handlers are only accessing some shared resource rarely, so it shouldn’t be a huge performance hit.

All the documentation and presentations say “DO NOT USE THE INTERRUPT SPIN
LOCK IF YOU ARE USING MSI SPINLOCKS” for Storport. The performance hit of
only using the Interrupt spinlock will be unacceptable.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@gmail.com
Sent: Wednesday, October 02, 2013 7:44 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] MSI synchronization

Don,

I’m assuming that you’ve specified InterruptSynchronizePerMessage in your
PORT_CONFIGURATION_INFORMATION structure and that you don’t want to use
InterruptSynchronizeAll.

I think what you have to do here is call StorPortAcquireSpinLock and request
the InterruptLock. In other words, don’t try to use the MSI per message
locks to do this, use the global lock to protect whatever resource they are
sharing. I could be gauging this wrong because I’m not sure of the exact
use case, but it seems like the message handlers are only accessing some
shared resource rarely, so it shouldn’t be a huge performance hit.


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

The performance hit of acquiring the 1 global spin lock is going to be
less than acquring X MSI message locks. The lock is only global to the
interrupt handler for that device. It’s not global to all storport
miniports.

Regarding the documentation, where does it actually say to not use
InterruptSpinLock in your situtation because I don’t recall it saying
that? I know the documentation states to use StorPortAcquireMsiSpinLock
if and only if you’re synchronizing per message, but I don’t know
anywhere where it says you can’t use the global lock if you want to.
Normally, you wouldn’t want to because you might as well be using
SynchronizeAll if that’s the only lock you use, but for limited cases
where you need to synchronize everything its the best choice you’ve got
IMO. The alternative is to trust storport to do the right thing if you
just call StorPortAcquireMsiSpinLock for all messages, but to me the
documentation is a little fuzzy on if that’s even an intended use. I
get the impression from the documentation that those locks are only
intended for use to synchronize with a single message/hardware resource
(like NVMe queues using 1 message per queue).

Could you provide a link to the documentation that says you can’t use
the single lock when you’re synchronizing per message? I can’t find it,
and I’d like to read it.

Thanks

On 10/2/2013 5:31 PM, Don Burn wrote:

All the documentation and presentations say “DO NOT USE THE INTERRUPT SPIN
LOCK IF YOU ARE USING MSI SPINLOCKS” for Storport. The performance hit of
only using the Interrupt spinlock will be unacceptable.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@gmail.com
Sent: Wednesday, October 02, 2013 7:44 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] MSI synchronization

Don,

I’m assuming that you’ve specified InterruptSynchronizePerMessage in your
PORT_CONFIGURATION_INFORMATION structure and that you don’t want to use
InterruptSynchronizeAll.

I think what you have to do here is call StorPortAcquireSpinLock and request
the InterruptLock. In other words, don’t try to use the MSI per message
locks to do this, use the global lock to protect whatever resource they are
sharing. I could be gauging this wrong because I’m not sure of the exact
use case, but it seems like the message handlers are only accessing some
shared resource rarely, so it shouldn’t be a huge performance hit.


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

The problems is that if you want to use nterruptSynchronizePerMessage since
I have a device with a lot of MSI interrupts and a system with a lot of
cores, I have to use MSI spin locks. Now I could go to
InterruptSynchronizeAll but since the ISR and StartIo do a lot of work under
the lock instead of having N concurrent operations for device during normal
operations, I am constrained to 1 concurrent operation because things like
reset that occur very rarely need to synchronize across all the I/O paths.

The presentations on this stuff at the 2007 WinHEC and the 2008 DDC both
said use MSI spin locks or interrupt spin locks but not both. Now the
documentation for this is poor, but given the nature of MSI-X interrupts
running at difference DIRQL’s I cannot envision how this works. The
documentation on the INTERRUPT_SYNCHRONIZATION_MODE enumeration
http://msdn.microsoft.com/en-us/library/windows/hardware/ff559199(v=vs.85).a
spx certainly indicates that Storport will grab the MSI spin lock for an ISR
if the synchronization more is InterruptSynchronizePerMessage so what value
would I have with requesting the interrupt spin lock at that point anyway?

Unfortunately, it does not look like Storport does a decent job for this
situation, so it is either radical measures or take the major performance
hit.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Jeff Glass
Sent: Wednesday, October 02, 2013 9:29 PM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] MSI synchronization

The performance hit of acquiring the 1 global spin lock is going to be less
than acquring X MSI message locks. The lock is only global to the
interrupt handler for that device. It’s not global to all storport
miniports.

Regarding the documentation, where does it actually say to not use
InterruptSpinLock in your situtation because I don’t recall it saying that?
I know the documentation states to use StorPortAcquireMsiSpinLock if and
only if you’re synchronizing per message, but I don’t know anywhere where it
says you can’t use the global lock if you want to.
Normally, you wouldn’t want to because you might as well be using
SynchronizeAll if that’s the only lock you use, but for limited cases where
you need to synchronize everything its the best choice you’ve got IMO. The
alternative is to trust storport to do the right thing if you just call
StorPortAcquireMsiSpinLock for all messages, but to me the documentation is
a little fuzzy on if that’s even an intended use. I get the impression from
the documentation that those locks are only intended for use to synchronize
with a single message/hardware resource (like NVMe queues using 1 message
per queue).

Could you provide a link to the documentation that says you can’t use the
single lock when you’re synchronizing per message? I can’t find it, and I’d
like to read it.

Thanks

On 10/2/2013 5:31 PM, Don Burn wrote:

All the documentation and presentations say “DO NOT USE THE INTERRUPT
SPIN LOCK IF YOU ARE USING MSI SPINLOCKS” for Storport. The
performance hit of only using the Interrupt spinlock will be unacceptable.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of
xxxxx@gmail.com
Sent: Wednesday, October 02, 2013 7:44 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] MSI synchronization

Don,

I’m assuming that you’ve specified InterruptSynchronizePerMessage in
your PORT_CONFIGURATION_INFORMATION structure and that you don’t want
to use InterruptSynchronizeAll.

I think what you have to do here is call StorPortAcquireSpinLock and
request the InterruptLock. In other words, don’t try to use the MSI
per message locks to do this, use the global lock to protect whatever
resource they are sharing. I could be gauging this wrong because I’m
not sure of the exact use case, but it seems like the message handlers
are only accessing some shared resource rarely, so it shouldn’t be a huge
performance hit.


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer