MSI-X Latency

john_bloe · October 26, 2010, 2:52pm

>I finally narrow down to what causes the lengthy latency: when my application calls Createfile and >Readfile to read from the device in asynchronous way (i.e. FILE_FLAG_OVERLAPPED specified), the >lengthy latency is reduced to below 2us. In synchronous way, it can be somewhere between 3 and 90 >us. Can somebody explain this ?

In general that makes sense - async i/o is and should be faster than sync. You sacrifice simplicity for speed.

If this is a multithreaded app, http://winprogger.com/?p=991 may apply.

Satya
http://winprogger.com

Peter_Viscarola_OSR · October 26, 2010, 4:37pm

Dude… read the rest of the thread… or at least the thread title… or even just the first post in the thread.

The OP WROTE:

The OP is talking about the latency between requesting an interrupt and getting that interrupt delivered.

Peter
OSR

anton_bassov · October 26, 2010, 4:47pm

> With MSI, the governing stuff set by the OS can be flexible and allow the device to choose

what CPU to interrupt.

Well, IOAPIC is quite flexible thing as well, so that I don’t see much difference is in this respect. Please check IOAPIC Manual and Volume 3 of Developer’s Manual - you will see that there is practically one-to-one correspondence between fields in Message Address and Message Data Registers for MSI and IOAPIC’s redirection table…

This (and the ability of eliminating IOAPIC at all) is one of the major advantanges of MSI.

I would say that, as far as OS software is concerned, the major advantage of MSI is possibility to get dedicated interrupt vector…

Anton Bassov

Peter_Viscarola_OSR · October 26, 2010, 4:58pm

And you’d be at least partially correct.

The advantages of MSI:

Dedicated interrupt vectors (no sharing).
Device can have MULTIPLE interrupts… one to uniquely indicate each interrupt reason. This reduces the need to check the interrupt status register in the ISR. You can have up to 2048 interrupts per device with MSI-X, which is pretty cool.
Device can direct interrupt to specific machines (hmmm… I think that requires MSI-X)

Peter
OSR

Maxim_S_Shatskih · October 26, 2010, 5:10pm

> 2) Device can have MULTIPLE interrupts… one to uniquely indicate each interrupt reason. This

reduces the need to check the interrupt status register in the ISR. You can have up to 2048 interrupts
per device with MSI-X, which is pretty cool.

Also the device status bits are sent in the interrupt message itself, eliminating the need for a device status register read in the ISR.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

Alex_Chang · October 26, 2010, 5:11pm

I do appreciate all the feedbacks. I also agree with Peter that the IO fashion should NOT impact the latency of interrupt delivery. However, I am very sure that the OVERLAPPED stuff is the ONLY thing that causes the difference reliably and consistently. That’s why I opened up this thread. At first, I thought it happens to be CPU cache miss/hit or higher CPU IRQL masks, but it seems not.

anton_bassov · October 26, 2010, 5:32pm

> 2) Device can have MULTIPLE interrupts… one to uniquely indicate each interrupt reason. This reduces

the need to check the interrupt status register in the ISR.

Well, consider spurious interrupts because of the buggy MB; consider Alberto playing around with the local APIC requesting interrupts via ICR; etc - although you don’t share vector with anyone, there are still possibilities of your dedicated interrupt being raised for the reasons that have nothing to do with your device whatsoever. Therefore, checking a reason for interrupt would be not so bad idea…

Anton Bassov

anton_bassov · October 26, 2010, 6:29pm

> Also the device status bits are sent in the interrupt message itself, eliminating the need for a device

status register read in the ISR.

This is, indeed, yet another advantage of MSI - once interrupt gets raised as a memory write you don’t have to actually go to the device in order to check interrupt status…

Anton Bassov

Peter_Viscarola_OSR · October 27, 2010, 9:53am

OK… Assuming you still care about this (and I know that *I* do) let’s return to first principles then.

Can you describe, in a bit of detail please, where and how you are doing the timings? What, exactly, are you using for the triggers?

I *do* realize you said you’re using a LeCroy analyzer – presumably a PCIe bus analyzer – but I like to go step by step and be sure… and understand the exactly mechanics of your timing. Please be as explicit in your descriptions as you can. Assume I know nothing about your hardware environment.

There just HAS to be something subtle that you’re doing wrong, because from the MSI/MSI-X interrupt request to the first line of the ISR 2us is very reasonable… in fact, it’s downright pleasant. 90us is not good.

Can you do a trace with that LeCroy analyzer and see the instruction stream, by any chance? With symbols maybe?

Peter
OSR

Pavel_A1 · October 27, 2010, 10:05am

And, if possible, can you show the code snippet - how you register the interrupt and the ISR.
– pa

Peter_Viscarola_OSR · October 27, 2010, 10:30am

EXcellent suggestion. That would be useful in eliminating other issues we might have.

Peter
OSR

Alex_Chang · October 27, 2010, 1:44pm

Peter,

I put the LeCroy analyzer in between our device and a PCIe slot on MB to intercept the traffic. The device accepts an IO request, e.g., reading from the disk, when driver issues a PCIe MemWr and it issues MemWr to MSI vector to indicate the completions after all data gets transfer to system. I put in another MemWr to one of our memory-mapped register in the first line of my ISR to indicate the end of interrupt latency. Pretty straight forward and the analyzer records all the TLPs, including the MemWrs, and their specific time stamps/deltas. Again, only one request issued at a time for the measurement.

Here are the codes in FindAdapter routine…

// Populate all pPort Configuraiton Information fields…
pPCI->MaximumTransferLength = pDE->MaxTranSize;
pPCI->NumberOfPhysicalBreaks = pDE->PhysicalBreaks;
pPCI->MaximumNumberOfTargets = 1;

pPCI->NumberOfBuses = 1;
pPCI->ScatterGather = TRUE;
pPCI->AlignmentMask = 7;
pPCI->CachesData = TRUE;

// Support SRB_FUNCTION_RESET_DEVICE
pPCI->ResetTargetSupported = TRUE;
// Set the number of LUNs supported to 32.
pPCI->MaximumNumberOfLogicalUnits = 32;
// Set driver to run in full duplex mode
pPCI->SynchronizationModel = StorSynchronizeFullDuplex;
pDE->DeviceCap |= DEV_CAP_FULL_DUPLEX;
// Specify the size of SrbExtension
pPCI->SrbExtensionSize = sizeof(SRB_EXTENSION);
pDE->SrbExtSize = pPCI->SrbExtensionSize;

// For 64-bit systems, controller supports 64-bit addressing,
if (pPCI->Dma64BitAddresses == SCSI_DMA64_SYSTEM_SUPPORTED) {
pPCI->Dma64BitAddresses = SCSI_DMA64_MINIPORT_SUPPORTED;
pDE->DeviceCap |= DEV_CAP_64B_ADDR;
}

pPCI->InterruptSynchronizationMode = InterruptSynchronizePerMessage;
// Specify MSI ISR here
pPCI->HwMSInterruptRoutine = MSIISR;

return(SP_RETURN_FOUND);

john_bloe · October 27, 2010, 2:17pm

Peter,

You have issues.

Alex,

You need to look at all that that is going on between the start and end of measurement to see why it takes longer - not just your driver code.

Satya
http://www.winprogger.com

Pavel_A1 · October 27, 2010, 2:40pm

> pPCI->InterruptSynchronizationMode = InterruptSynchronizePerMessage;

// Specify MSI ISR here
pPCI->HwMSInterruptRoutine = MSIISR;

return(SP_RETURN_FOUND);

Ok so the actual ISR actually belongs to the class driver.
Before calling your routine, it may do somethng else.
–pa

Peter_Viscarola_OSR · October 27, 2010, 5:53pm

Wow… that’s a possible solution, except for the fact that I’m pretty sure that the class driver doesn’t do anything other than a couple of simple (static) checks, and then call the Miniport’s ISR.

Sigh…

And you need to get a clue and remember that you’re a guest here, with a low post count, no established reputation, and you’re missing the point of this thread.

BTW… Is your name John Bloe or Satya? Cuz, the web interface says “John Bloe” and your signing your posts “Satya”…

Peter
OSR

Alex_Grig · October 27, 2010, 6:45pm

Isn’t storport MSI routine actually called in a DPC which could be delayed by other DPCs and ISRs?

The documentation for InterruptSynchronizePerMessage says "the Storport driver calls the miniport driver’s HwMSInterruptRoutine routine at IRQL <= DISPATCH without acquiring any interrupt spin locks. Is it correct or doc bug?

john_bloe · October 27, 2010, 7:34pm

Take the chill pill Peter. Having seen OP’s code and listing the advantages of MSI 1,2,3 and the “sigh…” above, have you got anything to explain what the OP is seeing ? Tell me how relevant it is to discuss advantages of MSI in this thread ? Is the OP going to switch to different way of doing interrupts on PCIe ?

remember that you’re a guest here, with a low post count, no established reputation, and you’re >missing the point of this thread.

condescending poppycock.

BTW… Is your name John Bloe or Satya? Cuz, the web interface >says “John Bloe” and your signing your posts “Satya”…

It is not Dude. Dont mock people and keep those TPS reports coming, mmmmkay ?

Satya
http://www.winprogger.com

anton_bassov · October 27, 2010, 7:53pm

>> remember that you’re a guest here, with a low post count, no established reputation, and you’re

>missing the point of this thread.

condescending poppycock.

No, it is just a very obvious waring - after all, it is Peter who happens to be an owner of this site. What he actually says here is “Look - you are a guest with no established reputation so that if I delete your membership from this list no one is going to even notice it”. I just wonder how one could miss it…

Anton Bassov

Gary_Little-3 · October 27, 2010, 8:42pm

I learned quickly while in the Army that you never ever piss off the fella
cooking your food or handling your pay records. A corollary for the
internet is to never ever piss off the guy running the forum. Of course,
Peter is fairly thick skinned, and has been known to tolerate some rather
rampant stupidity before he pulls the plug on someone.

Gary G. Little
H (952) 223-1349
C (952) 454-4629
xxxxx@comcast.net

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@yahoo.com
Sent: Wednesday, October 27, 2010 6:36 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] MSI-X Latency

Take the chill pill Peter. Having seen OP’s code and listing the advantages
of MSI 1,2,3 and the “sigh…” above, have you got anything to explain what
the OP is seeing ? Tell me how relevant it is to discuss advantages of MSI
in this thread ? Is the OP going to switch to different way of doing
interrupts on PCIe ?

remember that you’re a guest here, with a low post count, no established
reputation, and you’re >missing the point of this thread.

condescending poppycock.

BTW… Is your name John Bloe or Satya? Cuz, the web interface >says “John
Bloe” and your signing your posts “Satya”…

It is not Dude. Dont mock people and keep those TPS reports coming, mmmmkay
?

Satya
http://www.winprogger.com

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

__________ Information from ESET Smart Security, version of virus signature
database 5569 (20101027) __________

The message was checked by ESET Smart Security.

http://www.eset.com

Alex_Grig · October 27, 2010, 11:24pm

“Also the device status bits are sent in the interrupt message itself,
eliminating the need for a device status register read in the ISR.”

I don’t think this is true for MSI-X.