PCIe packet sizes - driver effects?

NTDEV
1

We’ve got a custom PCIe board and we’re having trouble getting full bandwidth to the board. According to our hardware guys, we’re never seeing a data transfer from the motherboard of more than 64 bytes at a time (our PCIe core can handle a full 4K packet). We’ve tried two different cores (in an Altera FPGA) with no change in behaviour. We’ve also noticed that the motherboard/chipset/bios is ordering our card never to send more than 128 bytes in a packet (PCI config register).

The data transfers in question to the board are board-initiated DMA transfers. We do not have a PCIe analyzer, we are instead analyzing the performance of our FPGA.

The motherboards that we’ve tried so far are all AMD64 based, with the nVidia chipset (though we are running 32 bit windows XP PRO SP2). We’ve used an iWill and a Tyan motherboard so far, both with dual cpus. We’ve got some Intel systems on order, to see if they have any effect. Both tested motherboards

There are no other PCIe devices in the tested systems.

We’ve hunted through the BIOS on both systems, and nothing there seems to have an effect on this problem.

This appears to be some sort of chipset configuration or limitation. Does anyone know a way that a driver can affect this? Is there something I should be configuring differently? Is there some limitation that we don’t know about in the chipset?

Thanks!

Michael Kohne
xxxxx@kohne.org
“You must be smarter than the equipment you are trying to operate.”

2

It’s a limitation of Hypertransport. It’s max is 64 bytes, so the PCIE
interface to hyptertransport through the chipset will never transfer
more than 64 bytes at a time, even though the chipset can handle more.
I’m not sure why nvidia limits PCIE to 128 MB - probably something to do
with the HT limitation.

I have not yet found a way to work around this. If you figure it out (or
anyone else knows) please let me know. I don’t think there is anythign a
driver can do about it, though. The solution, I think, will have to come
from Nvidia.

Beverly

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of Michael Kohne
Sent: Monday, June 19, 2006 8:34 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] PCIe packet sizes - driver effects?

We’ve got a custom PCIe board and we’re having trouble getting full
bandwidth to the board. According to our hardware guys, we’re never
seeing a data transfer from the motherboard of more than 64 bytes at a
time (our PCIe core can handle a full 4K packet). We’ve tried two
different cores (in an Altera FPGA) with no change in behaviour. We’ve
also noticed that the motherboard/chipset/bios is ordering our card
never to send more than 128 bytes in a packet (PCI config register).

The data transfers in question to the board are board-initiated DMA
transfers. We do not have a PCIe analyzer, we are instead analyzing the
performance of our FPGA.

The motherboards that we’ve tried so far are all AMD64 based, with the
nVidia chipset (though we are running 32 bit windows XP PRO SP2). We’ve
used an iWill and a Tyan motherboard so far, both with dual cpus. We’ve
got some Intel systems on order, to see if they have any effect. Both
tested motherboards

There are no other PCIe devices in the tested systems.

We’ve hunted through the BIOS on both systems, and nothing there seems
to have an effect on this problem.

This appears to be some sort of chipset configuration or limitation.
Does anyone know a way that a driver can affect this? Is there something
I should be configuring differently? Is there some limitation that we
don’t know about in the chipset?

Thanks!

Michael Kohne
xxxxx@kohne.org
“You must be smarter than the equipment you are trying to operate.”

Questions? First check the Kernel Driver FAQ at
http://www.osronline.com/article.cfm?id=256

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer