Hi All,
I’ll need to write a windows driver for a PCI-E device soon. I have ever written a driver for a PCI.
I wonder the difference of them!
Thanks in advance!
For the most part there is no difference. If your PCI-E device supports
MSI interrupts you should be supporting them with IoConnectInterruptEx
that is the primary difference a driver writer would see.
Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr
“xxxxx@gmail.com” wrote in message
news:xxxxx@ntdev:
> Hi All,
>
> I’ll need to write a windows driver for a PCI-E device soon. I have ever written a driver for a PCI.
>
> I wonder the difference of them!
>
> Thanks in advance!
I agree with Don. About the only difference is support of MSI/MSI-X – The cool thing is that with MSI a device CAN have multiple interrupts it can send back to the host… which can make for some pretty cool functionality.
As a total aside, what’s WITH PCIe devices that don’t support MSI? It’s a required part of the spec… You would think that any bus interface logic for PCIe supports MSI, and thus you get it “for free” when you put the logic in your device. So why do I so frequently run into (custom developed) PCIe devices that don’t have MSI or MSI-X support? It sorta FEELS like these are just PCI devices that have been slapped into a new form factor.
Am I the only one who’s seeing this in devices for which they write drivers?
Peter
OSR
Peter,
You are definitely not the only one, I have worked on a high speed
communications device that should have had at least three interrupts,
but their PCI-E implementation only supported one. In this case I
believe it was the fault of the PCI interface chip. Ironically, I now
have a client who wants MSI even though the device will only ever have
one interrupt, but marketing wants MSI as a check off item.
Don Burn (MVP, Windows DKD)
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr
“xxxxx@osr.com” wrote in message news:xxxxx@ntdev:
> I agree with Don. About the only difference is support of MSI/MSI-X – The cool thing is that with MSI a device CAN have multiple interrupts it can send back to the host… which can make for some pretty cool functionality.
>
>
>
> As a total aside, what’s WITH PCIe devices that don’t support MSI? It’s a required part of the spec… You would think that any bus interface logic for PCIe supports MSI, and thus you get it “for free” when you put the logic in your device. So why do I so frequently run into (custom developed) PCIe devices that don’t have MSI or MSI-X support? It sorta FEELS like these are just PCI devices that have been slapped into a new form factor.
>
> Am I the only one who’s seeing this in devices for which they write drivers?
>
> Peter
> OSR
Peter wrote …
As a total aside, what’s WITH PCIe devices that don’t support MSI? It’s a required part of the spec… You would think that any bus interface logic for PCIe supports MSI, and thus you get it “for free” when you put the logic in your device. So why do I so frequently run into (custom developed) PCIe devices that don’t have MSI or MSI-X support? It sorta FEELS like these are just PCI devices that have been slapped into a new form factor. Am I the only one who’s seeing this in devices for which they write drivers?
… and I replied …
(smile) It’s likely due to a typical PCIe Gen2 core with all the “bells and whistles” (such as from Synopsys) costing much more than anyone on this list makes in year and needing a Palladium to properly debug versus a PCIe Gen1 core which is “compliant” (some of which are now open source) costing only a few grand and only needing the equivalent of a Dini FPGA board to properly debug … go figure …
Even worse are the designs which take a PCI core, bolt on a PCIe bridge chip and call it done. Ick, ew, argh and lord help you if you need to debug the PCI core …
I see much, much worse behaviour in the USB world silicon and firmware – check my post a few months ago warning the fellow whose HW team decided to power other junk with the power for USB … ick, ew, argh. These days I’m happy just to get a USB chipset to power up and get some sort of signal back, much less be able to develop against it. There is so much “wiggling” for signal timings, state transitions, power usage, etc. in USB cores (even from known good vendors, forget about the FPGA homebrews) … ick, ew, argh …
For the OP, agree with Don/ Peter – use one of the MS reference samples as a basis, look at the WinOF code for MSI-X support (nothing else, just the MSI-X!), use KMDF as a foundation and you’re good. Pay very close attention to handling ALL of the PnP notifications, though – PCIe can and will be started/ stopped, be rebalanced, do lot’s of Dx and Sx transitions, etc. KMDF will make handling all of this stuff easy as cake …
Dude! At least it *supported* MSI. I can see them thinking one interrupt is enough. Sort of old-style thinking, but in many cases it really IS enough, right? Once in your DPC, you’re gonna do ALL your interrupt service, unless you have a driver design (with appropriately supporting hardware) that uses dedicated DPC for each message (and in many cases, this isn’t a great win).
But I’ve seen PCIe devices that only issue LBIs. No MSI support at all.
Well, there’s nothing wrong with that. I’m all for it, in fact. It’s not just specsmanship.
Using MSI will not only get you a dedicated interrupt, but on a properly designed device will get you the ability to go from your ISR to your DPC without having to touch a device register (which would be a serializing operation).
Once in your DPC, again assuming a properly designed device, you can just check your shared memory data structures to determine which request(s) are complete… again, without having to touch a device register.
This is all good for device and system throughput.
Peter
OSR
Thank you, Mr. Howard. That’s good to know. You can see my hardware design experience extends all the way through telling people what they should do and what they should support in their devices and interfaces, and stops just short of being concerned about the cost (in terms of dollars and complexity) of implementing any of those recommendations 
On, TOTALLY. I suspect the “cost of entry” (in terms of dollars and intellectual capital) to design a USB device is just so much less than that for designing a PCIe device that there are more people who are able to “play.” That leads to more design with, well, ah, ahem, “interesting” implementation and subsequent edge conditions.
Again, thanks for the reply. I’d love to hear from more H/W folks reply about the whole “PCIe without any MSIs” issue, because I’m genuinely curious about some of the details.
Peter
OSR
Peter,
For high speed network controller with multiple hardware rx queue MSI-x directly associated (or mapped ) with several CPUs, is almost a must. More over due to virtualization ( in fact para virtualization ) we don’t get the near perfect performance on FC or NIC, so the next best is sr-iov a step forward from (direct assignment) DIO, where interrupt re-mapper and other hardware assist comes to play to make the functions go on their own…
So there are few scenarios where LBI just don’t cut it. Think about couple 40Gb NIC ports, and 8 hyper-thd’ed CPU’s on LBI, It reminds me the following -
“Easy lover, she will get a hold of you … Before you know it you will be on your knees …” Phil Collins.
-pro
On Apr 21, 2011, at 8:13 AM, xxxxx@osr.com wrote:
Thank you, Mr. Howard. That’s good to know. You can see my hardware design experience extends all the way through telling people what they should do and what they should support in their devices and interfaces, and stops just short of being concerned about the cost (in terms of dollars and complexity) of implementing any of those recommendations
On, TOTALLY. I suspect the “cost of entry” (in terms of dollars and intellectual capital) to design a USB device is just so much less than that for designing a PCIe device that there are more people who are able to “play.” That leads to more design with, well, ah, ahem, “interesting” implementation and subsequent edge conditions.
Again, thanks for the reply. I’d love to hear from more H/W folks reply about the whole “PCIe without any MSIs” issue, because I’m genuinely curious about some of the details.
Peter
OSR
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer
Sure… not arguing that at all. MSI with multiple interrupts can be a good thing. MSI-X can ALSO be a VERY good thing.
Of course, there are also plenty of multi-interrupt MSI/MSI-X device implementations that are kinda stupid. You know, we have 4 interrupts because we CAN. And no… I’m not saying this applies to the very high speed NIC cases.
I think the key words here are “MSI-X” and ALMOST, no??
Unless I’m mistaken, MSI-X (which you mentioned specifically) isn’t even supported by all chipsets/vendors. So, does this mean you’re dooming these vendor’s systems to “they don’t support high speed network controllers”?
Peter
OSR
IIRC, that there has to be one LBI (pin) implemented in any advance
controller, I think it is the spec mandates, so there is a choice for
those…
And for low interrupt count type device ( i.e. those are inherently slow)
they can always have LBI and lined up in a shared ISR chain!!!
So it is not a must, but as you caught “almost”.
Even in 10Gb controllers, I’ve seen few specs ( 'bout 5 vendors), they all
have MSI-X. Some of them even have multiple specialized processors in the
controller itself.
Yes it is not an absolute must…
-pro
Sure… not arguing that at all. MSI with multiple interrupts can be a
good thing. MSI-X can ALSO be a VERY good thing.Of course, there are also plenty of multi-interrupt MSI/MSI-X device
implementations that are kinda stupid. You know, we have 4 interrupts
because we CAN. And no… I’m not saying this applies to the very high
speed NIC cases.I think the key words here are “MSI-X” and ALMOST, no??
Unless I’m mistaken, MSI-X (which you mentioned specifically) isn’t even
supported by all chipsets/vendors. So, does this mean you’re dooming
these vendor’s systems to “they don’t support high speed network
controllers”?Peter
OSR
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
“Of course, there are also plenty of multi-interrupt MSI/MSI-X device
implementations that are kinda stupid. You know, we have 4 interrupts because
we CAN. And no… I’m not saying this applies to the very high speed NIC cases.”
People are unclear on the concept and often think they need a interrupt per target processor, while just ONE MSI-X vector (ISR) can be invoked on any processor, depending on MSI-X data posted from the device.
MSI-X has advantages over MSI …
Few of those are -
-
What if the device can signal wide variety of events (in band vs. side
band) ! -
Dynamic interrupt redirection (mapping)
-
NUMA
-
True virtual PCI functions ( 32 in band signals might not be good )
-pro
“Of course, there are also plenty of multi-interrupt MSI/MSI-X device
implementations that are kinda stupid. You know, we have 4 interrupts
because
we CAN. And no… I’m not saying this applies to the very high speed NIC
cases.”People are unclear on the concept and often think they need a interrupt
per target processor, while just ONE MSI-X vector (ISR) can be invoked on
any processor, depending on MSI-X data posted from the device.
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
> I’d love to hear from more H/W folks reply about the whole “PCIe without any MSIs” issue,
because I’m genuinely curious about some of the details.
Indeed, it would be interesting to hear about “PCIe without any MSIs” - after all, support for MSI or MSI-X
is, IIRC, mandatory for PCIe devices. I vaguely recall a thread on NTDEV some time ago where it was mentioned that even if PCIe device appears to signal interrupt via a pin there is still a message behind the scenes, i.e it is not really a “legacy” interrupt …
Anton Bassov
This is true, and I think it is virtualized, but also I’ve seen somewhere
that at least one out of four ( particularly INTA ) has to be there.
-pro
> I’d love to hear from more H/W folks reply about the whole “PCIe without
> any MSIs” issue,
> because I’m genuinely curious about some of the details.Indeed, it would be interesting to hear about “PCIe without any MSIs” -
after all, support for MSI or MSI-X
is, IIRC, mandatory for PCIe devices. I vaguely recall a thread on NTDEV
some time ago where it was mentioned that even if PCIe device appears to
signal interrupt via a pin there is still a message behind the scenes, i.e
it is not really a “legacy” interrupt …Anton Bassov
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
>… I’ve seen somewhere that at least one out of four ( particularly INTA ) has to be there.
Well, consider the OS version/configuration that does not support MSI (for example, WinXP or Linux with “pci=nomsi” argument specified as kernel boot option). If device does not support “legacy” interrupts it is simply unusable on such a system, right? Therefore, I think “legacy” interrupts are not going anywhere anytime soon - even if the whole thing is, in actuality, built on top of messages…
Anton Bassov
For inherently slow devices I don’t see the reason, why they will go away?
Except perhaps reducing pin counts ( ???). And bridges are there to tackle
these LBI ( emulate and post MSI )…
Now in the future, someone might say well we don’t need those bridges and we
have a way to have low-powered end points that will support direct PCIe etc.
… I just don’t know.
So you could be right here. Still I should be able to boot Linux on x386,
but what I would get is less.
But the fine line ( as always ) touched by Peter is MSIs ( not
particularly MSI-x with PCI-E)…
-pro
>… I’ve seen somewhere that at least one out of four ( particularly INTA
> ) has to be there.Well, consider the OS version/configuration that does not support MSI
(for example, WinXP or Linux with “pci=nomsi” argument specified as
kernel boot option). If device does not support “legacy” interrupts it is
simply unusable on such a system, right? Therefore, I think “legacy”
interrupts are not going anywhere anytime soon - even if the whole thing
is, in actuality, built on top of messages…Anton Bassov
NTDEV is sponsored by OSR
For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminarsTo unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer
wrote in message news:xxxxx@ntdev…
> I’ll need to write a windows driver for a PCI-E device soon. I have
> ever written a driver for a PCI.
>
> I wonder the difference of them!
Besides of MSIs, PCIe has link states and other features that Windows does
not seem to expose, but other OSes do.
Recently I got one “home-brewn FPGA” that works nicely in WinXP and Win7,
but crashed in new fedora.
Because it advertised link power caps in the config space that it does not
actually support, and linux happily starts banging on them.
Takeaway for me: testing only on Windows? prepare for surprises.
Good luck,
– pa
I’ve seen hardware that works on Linux that won’t even initialize properly on Windows.
So, let us generalize your learning so all may benefit: Test hardware on only one platform, or only one operating system, when you intend people to use that hardware on DIFFERENT platforms or VARIED operating systems? EXPECT surprises.
How a piece of hardware works under one OS really tells you almost nothing about how that hardware will work on any other OS. This should be common sense for any operating system level developer.
Peter
OSR
(smile) True dat! I’d actually generalize it a bit more … expect changes between HW rev’s, firmware rev’s, OS changes, MB rev’s … basically, everything external to your driver code can/ will/ should change on a daily, sometimes hourly basis. What does this mean for us as developers then?
-
Make it a point to have the HW folks make available individual version information for the board, the firmware on whatever you’re talking to and actual FPGA part they use for that firmware. Bonus points if they will put a unique serial number on the board (required for USB compliance) and a “made on” datestamp. These should be all in an EEPROM you can query, even better before the main firmware initializes. The best designs I’ve worked with expose an individual BAR which has a RO segment to the EEPROM and a small RW segment you can program with a simple address/data “poke”; this allows you to have a hardware debounce function as I talked about a few years ago here
-
Have a way to query the OS for version info as well as the motherboard vendor and version – I usually use the VID/PID’s of the motherboard components as AHCI sees them
Once you have that info, get in the habit of building an interface layer between your driver “guts” and the hardware you’re talking to; so, instead of directly dereferencing a mapped BAR pointer instead call a function in the “HWFoo” layer which then dereferences the mapped BAR pointer. Have this “HWFoo” layer be instantiated at runtime based on the version information from the HW (which is why it’s important to be able to query the info before the main firmware spins up). This will allow you flexibility when the HW guys change things up and you need to stay backward compatible with older HW
You should also do this for OS-specific interactions; even though KMDF abstracts away a lot of MB-specific stuff, there are places (such as MSI-X support) where the ability to determine what to do based on the OS and MB resources make a difference. A good example of this is in USB land where you can be running on a MB with USB 2.0 and [flaky] USB 3.0 controllers, Firewire with different OS drivers loaded, etc. Basically, treat MB and OS supplied external resources with the same interface layer as with your own HW and you’ll be much happier
Cheers!
> Well, there’s nothing wrong with that. I’m all for it, in fact. It’s not just specsmanship.
Using MSI will not only get you a dedicated interrupt, but on a properly designed device will get you the ability to
go from your ISR to your DPC without having to touch a device register (which would be a serializing operation).
Also, the device can have freedom in what CPU to interrupt on multi-core system.
–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com