PCIe bus contention

I see, during PCIe busmastering to system RAM, seemingly random pauses. The Xilinx PCIe block holds back the data flow, I assume in response to the Windows host throttling it. Suspecting contention with unknown processes’ memory transfers, I try stressing by running multiple memcpy() in separate processes… and the pauses in my busmastering go DOWN, i.e. performance improves… the maximum pauses are reduced at least by half.
I wildly speculate that the OS sees heavy memory activity and provides me with some memory block that is not in contention with that activity, whereas under normal conditions the OS is aware of contention but doesn’t care to optimize at that level.
Could that be? What could explain this? In what direction should I look to reduce these busmaster timeouts?
(I get the statistically same (roughly) timeouts on multiple machines, on the order of 12 microseconds, PCIe 3.0 x8)

The operating system is not involved at that level in any way. There is no throttling, and no management of memory blocks. The processor is just reading and writing memory addresses, whether it’s accessing memory or PCIe space. Everything you’re talking about is owned and managed by the PCIe root complex.

PCIe has the concept of “credits”, which is how it regulates how much you get to send. Some older root complexes did not do that very well. It’s also possible, I suppose, that you are seeing some very aggressive power management, so the processor goes to a lower power level when things aren’t active.

1 Like

that is a very wild speculation and does not seem possible. I suspect that the presence of other activity prevents low power states that become detrimental to your performance. Though they may be desirable to the end user. That is my wild speculation based on almost no evidence

2 Likes