I have a Xlinx FPGA demoboard for PCIe. I configure it to PCIe 3.0 and X4 and learn to write PCIe driver.
But after insert into my PC’s motherboard PCIe X16 slot, I saw it worked in PCIe Gen1, X4.
Use SIV tools(http://www.rh-software.com/) can see my device max rate speed supported is 8.0G T/S(this is PCIe Gen3), but the actual speed is only 2.5G T/S(PCIe Gen1).
To read the specification of PCIe 3.0, I found that’s the result of negotiation between the PCIe devices.

I don’t know whether there’s something important to do in my driver.
I tried to write the PCIe configuration space ‘Link control capabilities register’ to retrain the PCIe link, but it doesn’t work.

xxxxx@sina.com

Hello lizzoe,

there is little to no(*) software configuration involved that determines
the PCI Express link speed and #lanes.

Verify the PC capabilities, the BIOS settings (it may limit the speed and
#lanes). Verify the slot you use is actually connected to a root complex
with Gen3 support (there is typically only one such slot, that directly
connects to the CPU’s PCIe root complex).

(*)Indeed on some devices (older FPGA’s) the retrain bit was sometimes
needed, but you already tried that.

This is not a software issue, but a system configuration issue.

You may want to use the built-in logic analyzer of your FPGA tools to debug
the link training. However, I would first verify the configuration.

Under Linux the ‘lspci -vvvn’ command is a free tool to verify capabilities
of your root complex and end point.

Regards,

Leon.

On Fri, Oct 28, 2016 at 2:52 AM, xxxxx@sina.com wrote:

> I have a Xlinx FPGA demoboard for PCIe. I configure it to PCIe 3.0 and X4
> and learn to write PCIe driver.
> But after insert into my PC’s motherboard PCIe X16 slot, I saw it worked in*
> PCIe Gen1, X4*.
> Use SIV tools(http://www.rh-software.com/) can see my device max rate
> speed supported is 8.0G T/S(this is PCIe Gen3), but the actual speed is
> only 2.5G T/S(PCIe Gen1).
> To read the specification of PCIe 3.0, I found that’s the result of
> negotiation between the PCIe devices.
>
> I don’t know whether there’s something important to do in my driver.
> I tried to write the PCIe configuration space ‘Link control capabilities
> register’ to retrain the PCIe link, but it doesn’t work.
>
>
>
> ------------------------------
> xxxxx@sina.com
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:> showlists.cfm?list=ntdev>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:></http:>

xxxxx@sina.com wrote:

I have a Xlinx FPGA demoboard for PCIe. I configure it to PCIe
3.0 and X4 and learn to write PCIe driver.
But after insert into my PC’s motherboard PCIe X16 slot, I saw it
worked in*PCIe Gen1, X4*.

What is your motherboard? Are you quite sure it supports Gen 3?

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

My motherboard is ASUS B150m-K (http://www.asus.com/Motherboards/B150M-K/specifications/)

Expansion Slots
1 x PCIe 3.0/2.0 x16 (x16 mode, gray)
2 x PCIe 3.0/2.0 x1
It hast only one X16 PCIe slot

Maillist can not send mail with attachments, otherwise I could show you my BIOS configuration…
In BIOS, the PCIE speed is “Auto”, I used to try to set it to “Gen3”, but it’s still not work.

After entering the windows desktop, I use SIV tools to check the actural configuration of my device, it’s 2.5G T/S only.

Bus-Numb-Fun
IRQ
Vendor-Dev-Sub_OEM-Rev
Class(4:63)
Vendor and Device Description Showing 15 of 15
0-01-0
16
8086-1091-00000000-07
PCI Bridge (0-1) X4@1 (X16@3)
Intel Core i5 PCIe Controller(X16)
1-00-0
16
10E1-8034-0007100EE-00
Memory Controller X4@1
Xilinx

Device’s detail information:
DCAP
0xC4
0x012C8003
Max Payload 1,024 (Supported) Power 75W
DCTL
0xC8
0x380
Max Payload 1,024 Max Read Request 1,024(current)
LCTL
0cD0
0x0000
Power Management Disabled
LCAP
0xCC
0x0043F043
Link Rate 8.0G T/S Width X4(Maximum)
LSTS
0xD2
0x1041
Link Rate 2.5G T/S Width X4(Negotiated)

LCAP is the “Link capabilities”, can be displayed correctly with the configuration that I have done to the FPGA
LSTS is the “Link status”, is the actually status of the PCIe, X4 , but only Gen1(2.5G T/S)

xxxxx@sina.com

From: Tim Roberts
Date: 2016-10-29 01:19
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Fw: Why does my PCIe device negotiated speed be 2.5G T/S

xxxxx@sina.com wrote:

I have a Xlinx FPGA demoboard for PCIe. I configure it to PCIe
3.0 and X4 and learn to write PCIe driver.
But after insert into my PC’s motherboard PCIe X16 slot, I saw it
worked in*PCIe Gen1, X4*.

What is your motherboard? Are you quite sure it supports Gen 3?

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:>

I’d second the suggestion that you boot Linux and run lspci. If both OSs show x4@1, it seems like a hardware issue.

I know some boards don’t like spread spectrum clocking, and some motherboards have an option to disable it.

Actually, Xilinx has a detailed PDF on debugging link training issues, https://www.xilinx.com/Attachment/Xilinx_Answer_42368_Debugging_Guide.pdf

Jan

From: on behalf of “xxxxx@sina.com”
Reply-To: Windows List
Date: Friday, October 28, 2016 at 9:41 PM
To: Windows List
Subject: Re: Re: [ntdev] Fw: Why does my PCIe device negotiated speed be 2.5G T/S

My motherboard is ASUS B150m-K (http://www.asus.com/Motherboards/B150M-K/specifications/)

Expansion Slots
1 x PCIe 3.0/2.0 x16 (x16 mode, gray)
2 x PCIe 3.0/2.0 x1
It hast only one X16 PCIe slot

Maillist can not send mail with attachments, otherwise I could show you my BIOS configuration…
In BIOS, the PCIE speed is “Auto”, I used to try to set it to “Gen3”, but it’s still not work.

After entering the windows desktop, I use SIV tools to check the actural configuration of my device, it’s 2.5G T/S only.

Bus-Numb-Fun

IRQ

Vendor-Dev-Sub_OEM-Rev

Class(4:63)

Vendor and Device Description Showing 15 of 15

0-01-0

16

8086-1091-00000000-07

PCI Bridge (0-1) X4@1 (X16@3)

Intel Core i5 PCIe Controller(X16)

1-00-0

16

10E1-8034-0007100EE-00

Memory Controller X4@1

Xilinx

Device’s detail information:
DCAP

0xC4

0x012C8003

Max Payload 1,024 (Supported) Power 75W

DCTL

0xC8

0x380

Max Payload 1,024 Max Read Request 1,024(current)

LCTL

0cD0

0x0000

Power Management Disabled

LCAP

0xCC

0x0043F043

Link Rate 8.0G T/S Width X4(Maximum)

LSTS

0xD2

0x1041

Link Rate 2.5G T/S Width X4(Negotiated)

LCAP is the “Link capabilities”, can be displayed correctly with the configuration that I have done to the FPGA
LSTS is the “Link status”, is the actually status of the PCIe, X4 , but only Gen1(2.5G T/S)
________________________________
xxxxx@sina.com

From: Tim Robertsmailto:xxxxx
Date: 2016-10-29 01:19
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: Re: [ntdev] Fw: Why does my PCIe device negotiated speed be 2.5G T/S

xxxxx@sina.com wrote:
>
> I have a Xlinx FPGA demoboard for PCIe. I configure it to PCIe
> 3.0 and X4 and learn to write PCIe driver.
> But after insert into my PC’s motherboard PCIe X16 slot, I saw it
> worked inPCIe Gen1, X4.
>

What is your motherboard? Are you quite sure it supports Gen 3?

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

—
NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:

—
NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></http:></http:></http:></mailto:xxxxx></mailto:xxxxx>

Hello Lizzoe,

pls. boot a Ubuntu DVD and run “sudo lspci -vvvn” and copy the full output
into an email (not only of your FPGA endpoint, just “all”).

That will allow us to verify that BOTH the RC and EP support Gen3 link
under its current configuration.

Regards,

Leon.

On Sat, Oct 29, 2016 at 6:41 AM, xxxxx@sina.com wrote:

> My motherboard is ASUS B150m-K (http://www.asus.com/Motherboards/B150M-K/
> specifications/)
>
> Expansion Slots
> 1 x PCIe 3.0/2.0 x16 (x16 mode, gray)
> 2 x PCIe 3.0/2.0 x1
> It hast only one X16 PCIe slot
>
> Maillist can not send mail with attachments, otherwise I could show you my
> BIOS configuration…
> In BIOS, the PCIE speed is “Auto”, I used to try to set it to “Gen3”,
> but it’s still not work.
>
> After entering the windows desktop, I use* SIV tools* to check the
> actural configuration of my device, it’s 2.5G T/S only.
>
> Bus-Numb-Fun
> IRQ
> Vendor-Dev-Sub_OEM-Rev
> Class(4:63)
> Vendor and Device Description Showing 15 of 15
> 0-01-0
> 16
> 8086-1091-00000000-07
> PCI Bridge (0-1) X4@1 (X16@3)
> Intel Core i5 PCIe Controller(X16)
> 1-00-0
> 16
> 10E1-8034-0007100EE-00
> Memory Controller X4@1
> Xilinx
>
> Device’s detail information:
> DCAP
> 0xC4
> 0x012C8003
> Max Payload 1,024 (Supported) Power 75W
> DCTL
> 0xC8
> 0x380
> Max Payload 1,024 Max Read Request 1,024(current)
> LCTL
> 0cD0
> 0x0000
> Power Management Disabled
> LCAP
> 0xCC
> 0x0043F043
> Link Rate 8.0G T/S Width X4(Maximum)
> LSTS
> 0xD2
> 0x1041
> Link Rate 2.5G T/S Width X4(Negotiated)
>
> LCAP is the “Link capabilities”, can be displayed correctly with the
> configuration that I have done to the FPGA
> LSTS is the “Link status”, is the actually status of the PCIe, X4 , but
> only Gen1(2.5G T/S)
> ------------------------------
> xxxxx@sina.com
>
>
> From: Tim Roberts
> Date: 2016-10-29 01:19
> To: Windows System Software Devs Interest List
> Subject: Re: [ntdev] Fw: Why does my PCIe device negotiated speed be
> 2.5G T/S
>
> xxxxx@sina.com wrote:
> >
> > I have a Xlinx FPGA demoboard for PCIe. I configure it to PCIe
> > 3.0 and X4 and learn to write PCIe driver.
> > But after insert into my PC’s motherboard PCIe X16 slot, I saw it
> > worked inPCIe Gen1, X4.
> >
>
> What is your motherboard? Are you quite sure it supports Gen 3?
>
> –
> Tim Roberts, xxxxx@probo.com
> Providenza & Boekelheide, Inc.
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:> showlists.cfm?list=ntdev>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
>
>
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:> showlists.cfm?list=ntdev>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:></http:></http:></http:>

I’m sorry, the whole result of lspci is too large to exeed the mail list maximum message size.

lspci

00:00.0 Host bridge: Intel Corporation Sky Lake Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation Sky Lake PCIe Controller (x16) (rev 07)
00:02.0 VGA compatible controller: Intel Corporation Sky Lake Integrated Graphics (rev 06)
00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)
00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #1 (rev 31)
00:17.0 SATA controller: Intel Corporation Device a102 (rev 31)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #5 (rev f1)
00:1c.7 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #8 (rev f1)
00:1d.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #9 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)
00:1f.3 Audio device: Intel Corporation Device a170 (rev 31)
00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)
01:00.0 Memory controller: Xilinx Corporation Device 8034
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

#lspci -s “xxxx” -vv
00:00.0 Host bridge: Intel Corporation Sky Lake Host Bridge/DRAM Registers (rev 07)
Subsystem: ASUSTeK Computer Inc. Device 8694
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0
Capabilities: [e0] Vendor Specific Information: Len=10 >

00:01.0 PCI bridge: Intel Corporation Sky Lake PCIe Controller (x16) (rev 07) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
Memory behind bridge: f6000000-f70fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [88] Subsystem: ASUSTeK Computer Inc. Device 8694
Capabilities: [80] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
Address: fee0f00c Data: 4151
Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #2, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <256ns, L1 <8us
ClockPM- Surprise- LLActRep- BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt+
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
Slot #1, PowerLimit 75.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
Changed: MRL- PresDet+ LinkState-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via WAKE# ARIFwd-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Via WAKE# ARIFwd-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [140 v1] Root Complex Link
Desc: PortNumber=02 ComponentID=01 EltType=Config
Link0: Desc: TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+
Addr: 00000000fed19000
Capabilities: [d94 v1] #19
Kernel driver in use: pcieport

01:00.0 Memory controller: Xilinx Corporation Device 8034
Subsystem: Xilinx Corporation Device 0007
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Interrupt: pin A routed to IRQ 11
Region 0: Memory at f7000000 (32-bit, non-prefetchable) [disabled] [size=2K]
Region 1: Memory at f6000000 (32-bit, non-prefetchable) [disabled] [size=16M]
Capabilities: [80] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [c0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

xxxxx@sina.com

From: Leon Woestenberg
Date: 2016-10-29 22:40
To: Windows System Software Devs Interest List
Subject: Re: Re: [ntdev] Fw: Why does my PCIe device negotiated speed be 2.5G T/S
Hello Lizzoe,

pls. boot a Ubuntu DVD and run “sudo lspci -vvvn” and copy the full output into an email (not only of your FPGA endpoint, just “all”).

That will allow us to verify that BOTH the RC and EP support Gen3 link under its current configuration.

Regards,

Leon.

On Sat, Oct 29, 2016 at 6:41 AM, xxxxx@sina.com wrote:
My motherboard is ASUS B150m-K (http://www.asus.com/Motherboards/B150M-K/specifications/)

Expansion Slots
1 x PCIe 3.0/2.0 x16 (x16 mode, gray)
2 x PCIe 3.0/2.0 x1
It hast only one X16 PCIe slot

Maillist can not send mail with attachments, otherwise I could show you my BIOS configuration…
In BIOS, the PCIE speed is “Auto”, I used to try to set it to “Gen3”, but it’s still not work.

After entering the windows desktop, I use SIV tools to check the actural configuration of my device, it’s 2.5G T/S only.

Bus-Numb-Fun
IRQ
Vendor-Dev-Sub_OEM-Rev
Class(4:63)
Vendor and Device Description Showing 15 of 15
0-01-0
16
8086-1091-00000000-07
PCI Bridge (0-1) X4@1 (X16@3)
Intel Core i5 PCIe Controller(X16)
1-00-0
16
10E1-8034-0007100EE-00
Memory Controller X4@1
Xilinx

Device’s detail information:
DCAP
0xC4
0x012C8003
Max Payload 1,024 (Supported) Power 75W
DCTL
0xC8
0x380
Max Payload 1,024 Max Read Request 1,024(current)
LCTL
0cD0
0x0000
Power Management Disabled
LCAP
0xCC
0x0043F043
Link Rate 8.0G T/S Width X4(Maximum)
LSTS
0xD2
0x1041
Link Rate 2.5G T/S Width X4(Negotiated)

LCAP is the “Link capabilities”, can be displayed correctly with the configuration that I have done to the FPGA
LSTS is the “Link status”, is the actually status of the PCIe, X4 , but only Gen1(2.5G T/S)

xxxxx@sina.com

From: Tim Roberts
Date: 2016-10-29 01:19
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Fw: Why does my PCIe device negotiated speed be 2.5G T/S

xxxxx@sina.com wrote:
>
> I have a Xlinx FPGA demoboard for PCIe. I configure it to PCIe
> 3.0 and X4 and learn to write PCIe driver.
> But after insert into my PC’s motherboard PCIe X16 slot, I saw it
> worked inPCIe Gen1, X4.
>

What is your motherboard? Are you quite sure it supports Gen 3?

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

—
NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:

—
NTDEV is sponsored by OSR

Visit the list online at: http:

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at http:

— NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers! Details at To unsubscribe, visit the List Server section of OSR Online at</http:></http:></http:></http:></http:></http:>

lizzoe,

thanks, the capabilities for 8GT/s are there both ends indeed, and lnkctl2 seems to suggest they aim for it during training.

Strange indeed, but I have seen this with Gen1 vs Gen2 also.

When you tried to set the retrain bit, did you also try to set it on 00:01.0?

Initiating it from the root complex device helped me on a similar device from Altera, now Intel, before.

Also pls. see the application note suggested about further debugging this.

Please keep us updated on your progress.

Regards, Leon

On 30 Oct 2016, at 05:01, “xxxxx@sina.com” wrote:
>
> I’m sorry, the whole result of lspci is too large to exeed the mail list maximum message size.
> # lspci
> 00:00.0 Host bridge: Intel Corporation Sky Lake Host Bridge/DRAM Registers (rev 07)
> 00:01.0 PCI bridge: Intel Corporation Sky Lake PCIe Controller (x16) (rev 07)
> 00:02.0 VGA compatible controller: Intel Corporation Sky Lake Integrated Graphics (rev 06)
> 00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)
> 00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #1 (rev 31)
> 00:17.0 SATA controller: Intel Corporation Device a102 (rev 31)
> 00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #5 (rev f1)
> 00:1c.7 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #8 (rev f1)
> 00:1d.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #9 (rev f1)
> 00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31)
> 00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)
> 00:1f.3 Audio device: Intel Corporation Device a170 (rev 31)
> 00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)
> 01:00.0 Memory controller: Xilinx Corporation Device 8034
> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
>
> #lspci -s “xxxx” -vv
> 00:00.0 Host bridge: Intel Corporation Sky Lake Host Bridge/DRAM Registers (rev 07)
> Subsystem: ASUSTeK Computer Inc. Device 8694
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- SERR- > Latency: 0
> Capabilities: [e0] Vendor Specific Information: Len=10 >
>
> 00:01.0 PCI bridge: Intel Corporation Sky Lake PCIe Controller (x16) (rev 07) (prog-if 00 [Normal decode])
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- > Latency: 0
> Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
> Memory behind bridge: f6000000-f70fffff
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- > BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> Capabilities: [88] Subsystem: ASUSTeK Computer Inc. Device 8694
> Capabilities: [80] Power Management version 3
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
> Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
> Address: fee0f00c Data: 4151
> Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00
> DevCap: MaxPayload 256 bytes, PhantFunc 0
> ExtTag- RBE+
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
> MaxPayload 256 bytes, MaxReadReq 128 bytes
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
> LnkCap: Port #2, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <256ns, L1 <8us
> ClockPM- Surprise- LLActRep- BwNot+
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt+
> SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
> Slot #1, PowerLimit 75.000W; Interlock- NoCompl+
> SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
> Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
> SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
> Changed: MRL- PresDet+ LinkState-
> RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
> RootCap: CRSVisible-
> RootSta: PME ReqID 0000, PMEStatus- PMEPending-
> DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via WAKE# ARIFwd-
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Via WAKE# ARIFwd-
> LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
> Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> Compliance De-emphasis: -6dB
> LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1-
> EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
> Capabilities: [100 v1] Virtual Channel
> Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
> Arb: Fixed- WRR32- WRR64- WRR128-
> Ctrl: ArbSelect=Fixed
> Status: InProgress-
> VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
> Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
> Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
> Status: NegoPending- InProgress-
> Capabilities: [140 v1] Root Complex Link
> Desc: PortNumber=02 ComponentID=01 EltType=Config
> Link0: Desc: TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+
> Addr: 00000000fed19000
> Capabilities: [d94 v1] #19
> Kernel driver in use: pcieport
>
> 01:00.0 Memory controller: Xilinx Corporation Device 8034
> Subsystem: Xilinx Corporation Device 0007
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- > Interrupt: pin A routed to IRQ 11
> Region 0: Memory at f7000000 (32-bit, non-prefetchable) [disabled] [size=2K]
> Region 1: Memory at f6000000 (32-bit, non-prefetchable) [disabled] [size=16M]
> Capabilities: [80] Power Management version 3
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
> Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit+
> Address: 0000000000000000 Data: 0000
> Capabilities: [c0] Express (v2) Endpoint, MSI 00
> DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
> MaxPayload 256 bytes, MaxReadReq 512 bytes
> DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
> LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM not supported, Exit Latency L0s unlimited, L1 unlimited
> ClockPM- Surprise- LLActRep- BwNot-
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
> LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
> Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
> Compliance De-emphasis: -6dB
> LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1-
> EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>
>
>
>
> xxxxx@sina.com
>
> From: Leon Woestenberg
> Date: 2016-10-29 22:40
> To: Windows System Software Devs Interest List
> Subject: Re: Re: [ntdev] Fw: Why does my PCIe device negotiated speed be 2.5G T/S
> Hello Lizzoe,
>
> pls. boot a Ubuntu DVD and run “sudo lspci -vvvn” and copy the full output into an email (not only of your FPGA endpoint, just “all”).
>
> That will allow us to verify that BOTH the RC and EP support Gen3 link under its current configuration.
>
> Regards,
>
> Leon.
>
>
>
>> On Sat, Oct 29, 2016 at 6:41 AM, xxxxx@sina.com wrote:
>> My motherboard is ASUS B150m-K (http://www.asus.com/Motherboards/B150M-K/specifications/)
>>
>> Expansion Slots
>> 1 x PCIe 3.0/2.0 x16 (x16 mode, gray)
>> 2 x PCIe 3.0/2.0 x1
>> It hast only one X16 PCIe slot
>>
>> Maillist can not send mail with attachments, otherwise I could show you my BIOS configuration…
>> In BIOS, the PCIE speed is “Auto”, I used to try to set it to “Gen3”, but it’s still not work.
>>
>> After entering the windows desktop, I use SIV tools to check the actural configuration of my device, it’s 2.5G T/S only.
>>
>> Bus-Numb-Fun
>> IRQ
>> Vendor-Dev-Sub_OEM-Rev
>> Class(4:63)
>> Vendor and Device Description Showing 15 of 15
>> 0-01-0
>> 16
>> 8086-1091-00000000-07
>> PCI Bridge (0-1) X4@1 (X16@3)
>> Intel Core i5 PCIe Controller(X16)
>> 1-00-0
>> 16
>> 10E1-8034-0007100EE-00
>> Memory Controller X4@1
>> Xilinx
>>
>> Device’s detail information:
>> DCAP
>> 0xC4
>> 0x012C8003
>> Max Payload 1,024 (Supported) Power 75W
>> DCTL
>> 0xC8
>> 0x380
>> Max Payload 1,024 Max Read Request 1,024(current)
>> LCTL
>> 0cD0
>> 0x0000
>> Power Management Disabled
>> LCAP
>> 0xCC
>> 0x0043F043
>> Link Rate 8.0G T/S Width X4(Maximum)
>> LSTS
>> 0xD2
>> 0x1041
>> Link Rate 2.5G T/S Width X4(Negotiated)
>>
>> LCAP is the “Link capabilities”, can be displayed correctly with the configuration that I have done to the FPGA
>> LSTS is the “Link status”, is the actually status of the PCIe, X4 , but only Gen1(2.5G T/S)
>> xxxxx@sina.com
>>
>> From: Tim Roberts
>> Date: 2016-10-29 01:19
>> To: Windows System Software Devs Interest List
>> Subject: Re: [ntdev] Fw: Why does my PCIe device negotiated speed be 2.5G T/S
>>
>> xxxxx@sina.com wrote:
>> >
>> > I have a Xlinx FPGA demoboard for PCIe. I configure it to PCIe
>> > 3.0 and X4 and learn to write PCIe driver.
>> > But after insert into my PC’s motherboard PCIe X16 slot, I saw it
>> > worked inPCIe Gen1, X4.
>> >
>>
>> What is your motherboard? Are you quite sure it supports Gen 3?
>>
>> –
>> Tim Roberts, xxxxx@probo.com
>> Providenza & Boekelheide, Inc.
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> Visit the list online at: http:
>>
>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
>> Details at http:
>>
>> To unsubscribe, visit the List Server section of OSR Online at http:
>>
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> Visit the list online at: http:
>>
>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
>> Details at http:
>>
>> To unsubscribe, visit the List Server section of OSR Online at http:
>
> — NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers! Details at To unsubscribe, visit the List Server section of OSR Online at
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at http:</http:></http:></http:></http:></http:></http:></http:></http:></http:>

Leon,
Thank you for your help. You mentioned that I need try to set the retrain bit for 00:01.0, I don’t know how to do that now

I use WdfFdoQueryForInterface to get businterface, and the use DevExt->BusInterface.GetBusData to read my device’s config space
status = WdfFdoQueryForInterface(
DevExt->Device,
&GUID_BUS_INTERFACE_STANDARD,
(PINTERFACE)&DevExt->BusInterface,
sizeof(BUS_INTERFACE_STANDARD),
1, // Version
NULL); // InterfaceSpecificData

I don’t know how to get the bridge’s businterface in driver code.

xxxxx@sina.com

Lizzoe,

I am not able to answer your question about how to access the root complex
(bridge) programmatically under Windows.

However, I did found my Linux script to retrain a i7-2600K with another
FPGA to Gen2 (when it trained to Gen1 initially).

We have developed PCIe IP cores for the major FPGA vendors and we have seen
systems where the BIOS limits the initial training target speeds.

Copy to gen2.sh, then use the command “sudo chmod +x gen2.sh” to make it
executable.
Then run by typing “./gen2.sh”.

For the gen3.sh variant, change the 0x0002:0x000f to 0x0003:0x000f

Regards,

Leon.

==
#!/bin/sh

Lancero Low Latency - development script

Copyright (C) 2011-2016 Sidebranch

Leon Woestenberg
#
# http://www.sidebranch.com/
# http://www.lancero.biz/
#
# Sets the Intel i7-2600 processor PCIe root complex in 5GT/s mode
# Triggers a retrain of the PCI Express link of the Intel i7-2600.
#
# Addresses are system-dependent. Verify using sudo lspci -vvv.
#
# the -s argument must address the root complex port!

# print current LnkCtl2 register
sudo setpci -v -s 00:01.0 0xd0.w
# set LnkCtl2 register; maximum negotiated rate of 5GT/s
# Set LNKCTL2 lower 4 bits to 0x2. (See PCIe Base Spec 3.0 section 7.8.19.)
sudo setpci -s 0:01.0 CAP_EXP+0x30.w=0x0002:0x000f

# print current LnkCtl register
sudo setpci -v -s 00:01.0 0xb0.w
# Set LNKCTL RETRAIN bit (See PCIe Base Spec 3.0 section 7.8.7.):
sudo setpci -s 0:01.0 CAP_EXP+0x10.w=0x0020:0x0020
==

On Sun, Oct 30, 2016 at 11:11 AM, xxxxx@sina.com wrote:

> Leon,
> Thank you for your help. You mentioned that I need try to set the retrain
> bit for 00:01.0, I don’t know how to do that now
>
> I use WdfFdoQueryForInterface to get businterface, and the use D
> evExt->BusInterface.GetBusData to read my device’s config space
> status = WdfFdoQueryForInterface(
> DevExt->Device,
> &GUID_BUS_INTERFACE_STANDARD,
> (PINTERFACE)&DevExt->BusInterface,
> sizeof(BUS_INTERFACE_STANDARD),
> 1,
> // Version
> NULL); //
> InterfaceSpecificData
>
> I don’t know how to get the bridge’s businterface in driver code.
>
> ------------------------------
> xxxxx@sina.com
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:> showlists.cfm?list=ntdev>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:></http:>