Reading from 64bit FPGA over PCI

Igal_Kroyter-2 · March 16, 2016, 6:27am

Hi,

I am trying to read from the FPGA which data bus width is 64bits.
I have received the PCI address and its length from ResourcesTranslated (starts at 0xF780_0000, length 0x80_0000).
I have converted the physical addresses to Kernel Virtual Addresses with
RegsBase = (ULONG64)MmMapIoSpace(REGSBasePA,REGSLength, MmNonCached );
(RegsBase is 0xFFFF_F880_0AA0_0000)
And tried to read the first memory mapped register with READ_REGISTER_ULONG64.
BTW, I tried to read other READ_REGISTER_xxxxx, too.
The value that I am receiving is 0xFFFF_FFFF_FFFF_FFFF.

My FPGA eng. who connected a chip-scope to the target FPGA claimed that he sees nor read or write access over the PCI to the FPGA :(.

Any ideas?

Igal

Tim_Roberts · March 16, 2016, 12:46pm

xxxxx@gmail.com wrote:

I am trying to read from the FPGA which data bus width is 64bits.
I have received the PCI address and its length from ResourcesTranslated (starts at 0xF780_0000, length 0x80_0000).

Are you using a kernel debugger? You can use the !pci command to
examine your PCI configuration space and make sure that number matches
your BAR. Those numbers do seem reasonable.

I have converted the physical addresses to Kernel Virtual Addresses with
RegsBase = (ULONG64)MmMapIoSpace(REGSBasePA,REGSLength, MmNonCached );
(RegsBase is 0xFFFF_F880_0AA0_0000)

That number also seems reasonable. You should not make RegsBase a
ULONG64, however. It wouldn’t be 64 bits long on a 32-bit system. You
should either use a pointer type (PVOID, PUCHAR or PULONG), or ULONG_PTR.

And tried to read the first memory mapped register with READ_REGISTER_ULONG64.
BTW, I tried to read other READ_REGISTER_xxxxx, too.

Can you show us the code? In which routine did you attempt this? Is it
possible, for example, that you tried to access the space before you had
mapped it? Show us the exact code where you mapped the space, and the
code where you try to use the space. Also, remember that
READ_REGISTER_ULONG64 is not available on 32-bit systems.

The value that I am receiving is 0xFFFF_FFFF_FFFF_FFFF.

That generally means you accessed an address where no device responded.
Are you behind a custom PCI bridge?

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Igal_Kroyter-2 · March 16, 2016, 1:56pm

Tim,

thanks for the reply.
The values that I have received are expected and agreed on the FPGA eng.

Here is the code:
for (unsigned int i = 0; i < ResourcesCount; ++i)
{
desc = WdfCmResourceListGetDescriptor(ResourcesTranslated, i);

if (!desc)
{
TraceEvents(TRACE_LEVEL_ERROR, TRACE_INIT, “%!FUNC! WdfResourceCmGetDescriptor failed”);
return STATUS_DEVICE_CONFIGURATION_ERROR;
}

switch (desc->Type)
{
case CmResourceTypeMemory:
if (!foundREGS && (desc->u.Memory.Length == FPGA_BAR0_LENGTH))
{
REGSBasePA = desc->u.Memory.Start;
REGSLength = desc->u.Memory.Length;
foundREGS = TRUE;
TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_INIT,
“%!FUNC! - Memory Resource [%I64X-%I64X] BAR0”,
desc->u.Memory.Start.QuadPart,
desc->u.Memory.Start.QuadPart + desc->u.Memory.Length-1);
}
else
TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_INIT,
“%!FUNC! - Memory Resource [%I64X-%I64X] ”,
desc->u.Memory.Start.QuadPart,
desc->u.Memory.Start.QuadPart + desc->u.Memory.Length);

break;
case CmResourceTypePort:
TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_INIT,
“%!FUNC! - Port Resource [%08I64X-%08I64X] ”,
desc->u.Port.Start.QuadPart,
desc->u.Port.Start.QuadPart + desc->u.Port.Length);
break;
case CmResourceTypeInterrupt:
TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_INIT,
“%!FUNC! - Interrupt Level %x Vector %x Affinity %08I64X”,
desc->u.Interrupt.Level,
desc->u.Interrupt.Vector,
desc->u.Interrupt.Affinity);

break;
case CmResourceTypeDevicePrivate:
TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_INIT,
“%!FUNC! - Device Private”);
break;
default:
TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_INIT,
“%!FUNC! - Unknown Resource Type %d”, desc->Type);
break;
}
}
if (foundREGS)
{

deviceContext->RegsBase = (PUCHAR) MmMapIoSpace(REGSBasePA,
REGSLength,
MmNonCached );

if (!deviceContext->RegsBase)
{
TraceEvents(TRACE_LEVEL_ERROR, TRACE_INIT,
“%!FUNC! - Unable to map Registers memory %08I64X, length %d”,
REGSBasePA.QuadPart,
REGSLength);
return STATUS_INSUFFICIENT_RESOURCES;
}

deviceContext->RegsLength = REGSLength;

TraceEvents(TRACE_LEVEL_INFORMATION, TRACE_INIT,
“%!FUNC! - Registers %p, length %x”,
deviceContext->RegsBase,
deviceContext->RegsLength
);

//
// Just for testing
//
UINT64 val = READ_REGISTER_ULONG64((volatile UINT64*)deviceContext->RegsBase);

This code is not going to be used on a 32bit machine, because the FPGA supports only 64it wide data, as I understand it a 32bit machine cannot access 64bit wide data.

The current system is running on an HPZ220 station with a Xilinx (Artix7) Evaluation board, with a Xilinx PCIe core. So actually the FPGA is sitting behind a bridge as it is on bus 2. Does it make a difference, once the CPU already provided a mapped memory?

Igal

Tim_Roberts · March 16, 2016, 3:13pm

xxxxx@gmail.com wrote:

The values that I have received are expected and agreed on the FPGA eng.

Here is the code:
//
// Just for testing
//
UINT64 val = READ_REGISTER_ULONG64((volatile UINT64*)deviceContext->RegsBase);

The READ_REGISTER_xxx APIs add the volatile. You don’t have to.

As an experiment, if you do these, do you see any cycles at the FPGA?
UINT64 val = *(PUINT64)deviceContext->RegsBase;
UINT val2 = *(PUINT)deviceContext->RegsBase;
USHORT val3 = *(PUSHORT)deviceContext->RegsBase;
UCHAR val4 = *(PUCHAR)deviceContext->RegsBase;

Do you have a PCIe bus analyzer? When your FPGA engineer says he’s not
seeing anything, does he mean there are no PCIe cycles, or does he mean
there’s nothing coming out of the PCIe IP block? Are they absolutely
sure that the PCIe IP is configured correctly?

This code is not going to be used on a 32bit machine, because the FPGA supports only 64it wide data, as I understand it a 32bit machine cannot access 64bit wide data.

Well, this becomes a marketing issue. Lots of people today still run
32-bit operating systems on their 64-bit-capable hardware. There are
enough intermediate steps between the CPU and your hardware that I’m not
sure you are always guaranteed that a 64-bit access gets translated to a
64-bit PCIe request. Your PCIe IP module might be translating between
the bus and the FPGA to soften that blow. In other words, it might
already work.

The current system is running on an HPZ220 station with a Xilinx (Artix7) Evaluation board, with a Xilinx PCIe core. So actually the FPGA is sitting behind a bridge as it is on bus 2. Does it make a difference, once the CPU already provided a mapped memory?

If it’s a bridge that is part of the computer, then one can probably
assume it is configured correctly. If it a bridge that is part of the
eval board, then one might want to make sure that ITS memory window was
also being configured.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Igal_Kroyter-2 · March 16, 2016, 4:30pm

Tim,

I have tried reading all the possible widths, all returning 0xfff depending on the relevant width.

I do not have a PCIe bus analyzer. The indication is on the other side of the core which indicates that there was a read or a write request. The core was raken as is from Xilinix, so I assume that it is OK.
The bridge is part of the motherboard of the station.

I hope it is not a silly question, but is it possible that the value of the address that I provide to the READ_REGISTER_xxx supposed to be physical address or should I provide only the lower 32bit of the virtual address?

Igal

Alex_Grig · March 16, 2016, 5:04pm

The BAR may not be readable until your driver goes through PNP_MN_START_DEVICE

Tim_Roberts · March 16, 2016, 7:55pm

xxxxx@gmail.com wrote:

I have tried reading all the possible widths, all returning 0xfff depending on the relevant width.

I do not have a PCIe bus analyzer. The indication is on the other side of the core which indicates that there was a read or a write request. The core was raken as is from Xilinix, so I assume that it is OK.

That may have been your first mistake, but that’s neither here nor there.

Are you using the kernel debugger? The !pci command can show you the
“command” register, which has bits to enable and disable the memory
space. There’s no reason why it should be disabled, but it’s worth
checking.

The bridge is part of the motherboard of the station.

I hope it is not a silly question, but is it possible that the value of the address that I provide to the READ_REGISTER_xxx supposed to be physical address or should I provide only the lower 32bit of the virtual address?

No, it has to be the full 64-bit virtual address. The processor ALWAYS
deals in virtual addresses. Always.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Igal_Kroyter-2 · March 17, 2016, 3:42am

Thanks for your reply.

Alex,

I have applied your tip:

Removed any access from the EvtDevicePrepareHardware and left there only mapping (updating device context with relevant PCI VA).
In the EvtDeviceD0Entry I have added the first access to the FPGA -> READ_REGISTER_ULONG64 (first address)

The result is that I am getting incorrect values.

Tim,

Hereafter the !pci 100 2 0 0 results from the kernel debugger:

!pci 100 2 0 0

PCI Configuration Space (Segment:0000 Bus:02 Device:00 Function:00)
Common Header:
00: VendorID 10ee Xilinx Corporation
02: DeviceID 7024
04: Command 0106 MemSpaceEn BusInitiate SERREn
06: Status 0010 CapList
08: RevisionID 00
09: ProgIF 00
0a: SubClass 80 Other Memory Controller
0b: BaseClass 05 Memory Controller
0c: CacheLineSize 0010 BurstDisabled
0d: LatencyTimer 00
0e: HeaderType 00
0f: BIST 00
10: BAR0 f7800000
14: BAR1 00000000
18: BAR2 00000000
1c: BAR3 00000000
20: BAR4 00000000
24: BAR5 00000000
28: CBCISPtr 00000000
2c: SubSysVenID 10ee
2e: SubSysID 0007
30: ROMBAR 00000000
34: CapPtr 40
3c: IntLine 00
3d: IntPin 00
3e: MinGnt 00
3f: MaxLat 00
Device Private:
40: 78034801 00000008 00016005 feeff00c
50: 00004990 00000000 00000000 00000000
60: 00020010 012c8e00 00003810 0003f442
70: 00420000 00000000 00000000 00000000
80: 00000000 00000002 00000000 00000000
90: 00000002 00000000 00000000 00000000
a0: 00000000 00000000 00000000 00000000
b0: 00000000 00000000 00000000 00000000
c0: 00000000 00000000 00000000 00000000
d0: 00000000 00000000 00000000 00000000
e0: 00000000 00000000 00000000 00000000
f0: 00000000 00000000 00000000 00000000
Capabilities:
40: CapID 01 PwrMgmt Capability
41: NextPtr 48
42: PwrMgmtCap 7803 PMED0 PMED1 PMED2 PMED3Hot Version=3
44: PwrMgmtCtrl 0008 DataScale:0 DataSel:0 D0

48: CapID 05 MSI Capability
49: NextPtr 60
4a: MsgCtrl MSIEnable MultipleMsgEnable:0 (0x1) MultipleMsgCapable:0 (0x1)
4c: MsgAddr feeff00c
50: MsData 4990

60: CapID 10 PCI Express Capability
61: NextPtr 00
62: Express Caps 0002 (ver. 2) Type:Endpoint
64: Device Caps 012c8e00
68: Device Control 3810 bcre/flr MRR:1K NS ap pf et MP:128 RO ur fe nf ce
6a: Device Status 0000 tp ap ur fe nf ce
6c: Link Caps 0003f442
70: Link Control 0000 es cc rl ld RCB:64 ASPM:None
72: Link Status 0042 scc lt lte NLW:x4 LS:2.5
84: DeviceCaps2 00000002 CTR:2 ctdis arifwd aor aoc32 aoc64 cas128 noro ltr TPH:0 OBFF:0 extfmt eetlp EETLPMax:0
88: DeviceControl2 0000 CTVal:0 ctdis arifwd aor aoeb idoreq idocom ltr OBFF:0 eetlp

Enhanced Capabilities:
100: CapID 0003 Serial Number Capability
Version 1
NextPtr 000

It looks like the memory space is enabled.

Any more ideas?

Igal

Igal_Kroyter-2 · March 17, 2016, 5:39am

Gentlemen,

the problem is with the FW, I made my FPGA eng. to rebuild the Xilinx core without any “goodies” that he inserted, and there is no problem accessing (read and write 64bit data) directly(with the KMDF driver) or via remote kernel debugger.

Thank you very much for the support learnt a lot from you.

Igal

Tim_Roberts · March 17, 2016, 12:39pm

xxxxx@gmail.com wrote:

the problem is with the FW, I made my FPGA eng. to rebuild the Xilinx core without any “goodies” that he inserted, and there is no problem accessing (read and write 64bit data) directly(with the KMDF driver) or via remote kernel debugger.

Thank you very much for the support learnt a lot from you.

Thank YOU for telling us what the problem was. This will help future
generations.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.