64 bit transfers to PCI device

NTDEV
1

Hi,

I am hoping someone can help. We have a custom (inhouse) designed PCI
board that uses the Xilinx Virtex-II FPGA and their PCI core for a custom
interface application. It is a 64 bit 66Mhz PCI card. According to the
hardware designers, it follows the rules to be 64 bit compatible.

I am in charge of the software, specifically the device driver under
Windows 2000 Pro. Our development environment is MS Visual Studio.NET. I
used NuMega’s DriverWorks to create a driver that has a couple functions.
Its main function is open, and provide a DeviceIOControl method to memory
map one of the boards BAR spaces into the users calling process. This all
works fine, and 32 bit accesses to the board work fine. The user process
simply de-references a pointer offset based on the memory map of an
“unsigned __int32 *”.

However, I am attempting to do native 64 bit transfers to the PCI device.
I have already discovered that the MS compiler will not easily generate a
native 64 bit instruction (MOVQ), even if I declare the variable as
“unsigned __int64”. This is not a big deal, the inline assembly feature is
quite nice, and I have a simple inline routine that essentially does the
following (say for a write) where address is the device mapped address,
and data is the value to write…

MOVD edx,address
MOVQ mm0, data
MOVQ MMWORD PTR [edx], mm0

This result in two 32bit transfers to the PCI device (the data is
correct), as verified by a VMETRO PB615 PCI analyzer. According to the
hardware engineers, the system should attempt a 64bit transaction on the
PCI by asserting REQ64, and the board should respond with ACK64. The
analyzer NEVER sees REQ64 go active at any time, even through the reset,
POST, and reboot process, all the way through my software attempting a
64bit cycle as described above.

I can find little information on the subject to understand why a single 64
bit transaction is not performed instead of two 32 bit. I don’t know if
there is some setup I need to do to make sure the MMU, or chipset will
perform 64 bit transactions.

The hardware (the only one I have with a 64 bit PCI bus) is a SuperMicro
P4DSE motherboard with a Pentium 4 Xeon processor installed. This uses the
ServerWorks Grand Champion (GC-SL) chipset.

I hope someone has seen something similar, or knows more about it than I
do.

Thanks in advance,

Russ Poffenberger
xxxxx@nptest.com

2

Did you check if your destination address is
aligned on a 8 bytes boundary. It might address
the issue.

Guy Bonneau

-----Original Message-----
From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com]On Behalf Of Russ Poffenberger
Sent: Friday, October 25, 2002 12:50 PM
To: NT Developers Interest List
Subject: [ntdev] 64 bit transfers to PCI device

Hi,

I am hoping someone can help. We have a custom (inhouse) designed PCI
board that uses the Xilinx Virtex-II FPGA and their PCI core
for a custom
interface application. It is a 64 bit 66Mhz PCI card. According to the
hardware designers, it follows the rules to be 64 bit compatible.

I am in charge of the software, specifically the device driver under
Windows 2000 Pro. Our development environment is MS Visual
Studio.NET. I
used NuMega’s DriverWorks to create a driver that has a
couple functions.
Its main function is open, and provide a DeviceIOControl
method to memory
map one of the boards BAR spaces into the users calling
process. This all
works fine, and 32 bit accesses to the board work fine. The
user process
simply de-references a pointer offset based on the memory map of an
“unsigned __int32 *”.

However, I am attempting to do native 64 bit transfers to the
PCI device.
I have already discovered that the MS compiler will not
easily generate a
native 64 bit instruction (MOVQ), even if I declare the variable as
“unsigned __int64”. This is not a big deal, the inline
assembly feature is
quite nice, and I have a simple inline routine that
essentially does the
following (say for a write) where address is the device
mapped address,
and data is the value to write…

MOVD edx,address
MOVQ mm0, data
MOVQ MMWORD PTR [edx], mm0

This result in two 32bit transfers to the PCI device (the data is
correct), as verified by a VMETRO PB615 PCI analyzer. According to the
hardware engineers, the system should attempt a 64bit
transaction on the
PCI by asserting REQ64, and the board should respond with ACK64. The
analyzer NEVER sees REQ64 go active at any time, even through
the reset,
POST, and reboot process, all the way through my software attempting a
64bit cycle as described above.

I can find little information on the subject to understand
why a single 64
bit transaction is not performed instead of two 32 bit. I
don’t know if
there is some setup I need to do to make sure the MMU, or chipset will
perform 64 bit transactions.

The hardware (the only one I have with a 64 bit PCI bus) is a
SuperMicro
P4DSE motherboard with a Pentium 4 Xeon processor installed.
This uses the
ServerWorks Grand Champion (GC-SL) chipset.

I hope someone has seen something similar, or knows more
about it than I
do.

Thanks in advance,

Russ Poffenberger
xxxxx@nptest.com

You are currently subscribed to ntdev as: xxxxx@Matrox.COM
To unsubscribe send a blank email to %%email.unsub%%

3

At 01:47 PM 10/25/2002 -0400, you wrote:

Did you check if your destination address is
aligned on a 8 bytes boundary. It might address
the issue.

Thanks for the tip, but yes, it is definitely 8 byte aligned.

Russ Poffenberger
NPTest, Inc.
xxxxx@NPTest.com

4

I suspect it is more of a northbridge/southbridge issue. You may check your
BIOS setting in peripheral bus section to make sure things are set properly.
If that does not help, you need to talk to ServerWorks to seek a solution.

Bi

-----Original Message-----
From: Russ Poffenberger [mailto:xxxxx@nptest.com]
Sent: Friday, October 25, 2002 11:08 AM
To: NT Developers Interest List
Subject: [ntdev] RE: 64 bit transfers to PCI device

At 01:47 PM 10/25/2002 -0400, you wrote:

Did you check if your destination address is
aligned on a 8 bytes boundary. It might address
the issue.

Thanks for the tip, but yes, it is definitely 8 byte aligned.

Russ Poffenberger
NPTest, Inc.
xxxxx@NPTest.com

You are currently subscribed to ntdev as: xxxxx@appstream.com
To unsubscribe send a blank email to %%email.unsub%%

5

You can try Mindshare’s “PCI System Architecture”, by Tom Shanley and Don
Anderson, chapter 15 has a lot of information about 64-bit PCI. Looks like
the motherboard must assert REQ64# with RST#, that is, at bus reset time.
The REQ64# is pulled up by each 32-bit connector, and also by the
motherboard. The way I read the book is, the peripheral boards must sample
REQ64# @posedge(RST#). If there’s no REQ64# active at that time, boards
configure themselves for 32-bit transfers. Maybe your hw guys may want to
probe the REQ64# on the motherboard to see if it is being asserted by the
bridge and the bios ? And Bi is right again, maybe you want to call
ServerWorks’s support and see what’s going on, sometimes you may have to set
a jumper on the motherboard or do something at config time to get it to
handle 64-bit buses.

One thing to try is to do it by hand. :slight_smile: That is, pull the bus RST# pin
down, assert REQ64# by hand, give the RST# enough time, and pull it back up.
At this time, if the hardware is working, your board should know it’s in
64-bit mode. Now, if your board can act as an initiator, start a 64-bit
input and see what happens ?

Hope this helps,

Alberto.

-----Original Message-----
From: Russ Poffenberger [mailto:xxxxx@nptest.com]
Sent: Friday, October 25, 2002 12:50 PM
To: NT Developers Interest List
Subject: [ntdev] 64 bit transfers to PCI device

Hi,

I am hoping someone can help. We have a custom (inhouse) designed PCI
board that uses the Xilinx Virtex-II FPGA and their PCI core for a custom
interface application. It is a 64 bit 66Mhz PCI card. According to the
hardware designers, it follows the rules to be 64 bit compatible.

I am in charge of the software, specifically the device driver under
Windows 2000 Pro. Our development environment is MS Visual Studio.NET. I
used NuMega’s DriverWorks to create a driver that has a couple functions.
Its main function is open, and provide a DeviceIOControl method to memory
map one of the boards BAR spaces into the users calling process. This all
works fine, and 32 bit accesses to the board work fine. The user process
simply de-references a pointer offset based on the memory map of an
“unsigned __int32 *”.

However, I am attempting to do native 64 bit transfers to the PCI device.
I have already discovered that the MS compiler will not easily generate a
native 64 bit instruction (MOVQ), even if I declare the variable as
“unsigned __int64”. This is not a big deal, the inline assembly feature is
quite nice, and I have a simple inline routine that essentially does the
following (say for a write) where address is the device mapped address,
and data is the value to write…

MOVD edx,address
MOVQ mm0, data
MOVQ MMWORD PTR [edx], mm0

This result in two 32bit transfers to the PCI device (the data is
correct), as verified by a VMETRO PB615 PCI analyzer. According to the
hardware engineers, the system should attempt a 64bit transaction on the
PCI by asserting REQ64, and the board should respond with ACK64. The
analyzer NEVER sees REQ64 go active at any time, even through the reset,
POST, and reboot process, all the way through my software attempting a
64bit cycle as described above.

I can find little information on the subject to understand why a single 64
bit transaction is not performed instead of two 32 bit. I don’t know if
there is some setup I need to do to make sure the MMU, or chipset will
perform 64 bit transactions.

The hardware (the only one I have with a 64 bit PCI bus) is a SuperMicro
P4DSE motherboard with a Pentium 4 Xeon processor installed. This uses the
ServerWorks Grand Champion (GC-SL) chipset.

I hope someone has seen something similar, or knows more about it than I
do.

Thanks in advance,

Russ Poffenberger
xxxxx@nptest.com

You are currently subscribed to ntdev as: xxxxx@compuware.com
To unsubscribe send a blank email to %%email.unsub%%

The contents of this e-mail are intended for the named addressee only. It
contains information that may be confidential. Unless you are the named
addressee or an authorized designee, you may not copy or use it, or disclose
it to anyone else. If you received it in error please notify us immediately
and then destroy it.

6

> You can try Mindshare’s “PCI System Architecture”, by Tom Shanley
and Don

Anderson

The book is great. I can confirm this.

Max

7

> Its main function is open, and provide a DeviceIOControl method to
memory

map one of the boards BAR spaces into the users calling process.
This all

MmMapLockedPages(…UserMode…)

P4DSE motherboard with a Pentium 4 Xeon processor installed. This
uses the
ServerWorks Grand Champion (GC-SL) chipset.

The chipset possibly does not support 64bit transfers when the CPU is
a master.

Max

8

It’s only me getting Max e-mail’s twice ?

At 11:07 31/10/2002 +0300, you wrote:

> You can try Mindshare’s “PCI System Architecture”, by Tom Shanley
and Don
> Anderson

The book is great. I can confirm this.

Max

You are currently subscribed to ntdev as: xxxxx@jungo.com
To unsubscribe send a blank email to %%email.unsub%%

9

At 11:07 AM 10/31/2002 +0300, you wrote:

> You can try Mindshare’s “PCI System Architecture”, by Tom Shanley
and Don
> Anderson

The book is great. I can confirm this.

Max

Yes, it is a good book. I do have it, but of course what the chipsets and
PCI bridges do is open to a lot of interpretation.

Russ Poffenberger
NPTest, Inc.
xxxxx@NPTest.com