[Bug 256264] Devices behind PEX 8664 PCIe Switch not detected since 11.0-RELEASE
Date: Sun, 30 May 2021 13:58:05 UTC
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256264
--- Comment #1 from Christopher Forgeron <chris@acsi.ca> ---
Some lspci commands I used to try and narrow this down:
1) This is the 10.2 System. You can see both the Intel 82599ES and the LSI 2008
cards. Ignore the MegaRAID, that's a built in PERC card that's always detected
(different bus):
lspci -vvnnt
-[0000:00]-+-00.0 Intel Corporation 5500 I/O Hub to ESI Port [8086:3403]
+-01.0-[01]--+-00.0 Broadcom Corporation NetXtreme II BCM5709S
Gigabit Ethernet [14e4:163a]
| \-00.1 Broadcom Corporation NetXtreme II BCM5709S
Gigabit Ethernet [14e4:163a]
+-07.0-[03-09]----00.0-[04-09]--+-00.0-[05]--
| +-01.0-[06]--
| +-02.0-[07]--
| +-04.0-[08]--+-00.0 Intel
Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb]
| | \-00.1 Intel
Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb]
| \-10.0-[09]----00.0 LSI Logic /
Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072]
+-14.0 Intel Corporation 7500/5520/5500/X58 I/O Hub System
Management Registers [8086:342e]
+-14.1 Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and
Scratch Pad Registers [8086:3422]
+-14.2 Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status
and RAS Registers [8086:3423]
+-16.0 Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3430]
+-16.1 Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3431]
+-16.2 Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3432]
+-16.3 Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3433]
+-16.4 Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3429]
+-16.5 Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:342a]
+-16.6 Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:342b]
+-16.7 Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:342c]
+-1a.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#4 [8086:2937]
+-1a.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#5 [8086:2938]
+-1a.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller
#2 [8086:293c]
+-1c.0-[02]----00.0 LSI Logic / Symbios Logic MegaRAID SAS 1078
[1000:0060]
+-1d.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#1 [8086:2934]
+-1d.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#2 [8086:2935]
+-1d.2 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#3 [8086:2936]
+-1d.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller
#1 [8086:293a]
+-1e.0-[0a]----03.0 Matrox Electronics Systems Ltd. MGA G200eW
WPCM450 [102b:0532]
+-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller
[8086:2918]
\-1f.5 Intel Corporation 82801I (ICH9 Family) 2 port SATA
Controller [IDE mode] [8086:2926]
2) This is the same command in 13.0, same hardware:
lspci -vvnnt
-[0000:00]-+-00.0 Intel Corporation 5500 I/O Hub to ESI Port [8086:3403]
+-01.0-[01]--+-00.0 Broadcom Inc. and subsidiaries NetXtreme II
BCM5709S Gigabit Ethernet [14e4:163a]
| \-00.1 Broadcom Inc. and subsidiaries NetXtreme II
BCM5709S Gigabit Ethernet [14e4:163a]
+-07.0-[03-09]----00.0-[04-09]--+-00.0-[05]--
| +-01.0-[06]--
| +-02.0-[07]--
| +-04.0-[08]--
| \-10.0-[09]--
+-14.0 Intel Corporation 7500/5520/5500/X58 I/O Hub System
Management Registers [8086:342e]
+-14.1 Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and
Scratch Pad Registers [8086:3422]
+-14.2 Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status
and RAS Registers [8086:3423]
+-16.0 Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3430]
+-16.1 Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3431]
+-16.2 Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3432]
+-16.3 Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3433]
+-16.4 Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3429]
+-16.5 Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:342a]
+-16.6 Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:342b]
+-16.7 Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:342c]
+-1a.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#4 [8086:2937]
+-1a.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#5 [8086:2938]
+-1a.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller
#2 [8086:293c]
+-1c.0-[02]----00.0 Broadcom / LSI MegaRAID SAS 1078 [1000:0060]
+-1d.0 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#1 [8086:2934]
+-1d.1 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#2 [8086:2935]
+-1d.2 Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#3 [8086:2936]
+-1d.7 Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller
#1 [8086:293a]
+-1e.0-[0a]----03.0 Matrox Electronics Systems Ltd. MGA G200eW
WPCM450 [102b:0532]
+-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller
[8086:2918]
\-1f.5 Intel Corporation 82801I (ICH9 Family) 2 port SATA
Controller [IDE mode] [8086:2926]
3) on 13.0 a quick summary:
lspci -M
00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root
Port 1 (rev 13)
## 00:01.0 is a bridge from 00 to 01-01
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root
Port 7 (rev 13)
## 00:07.0 is a bridge from 00 to 03-09
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management
Registers (rev 13)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad
Registers (rev 13)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and
RAS Registers (rev 13)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device (rev 13)
00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device (rev 13)
00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device (rev 13)
00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device (rev 13)
00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device (rev 13)
00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device (rev 13)
00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device (rev 13)
00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device (rev 13)
00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #4 (rev 02)
00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #5 (rev 02)
00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI
Controller #2 (rev 02)
00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1
(rev 02)
## 00:1c.0 is a bridge from 00 to 02-02
00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #1 (rev 02)
00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #2 (rev 02)
00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #3 (rev 02)
00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI
Controller #1 (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
## 00:1e.0 is a bridge from 00 to 0a-0a
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller
(rev 02)
00:1f.5 IDE interface: Intel Corporation 82801I (ICH9 Family) 2 port SATA
Controller [IDE mode] (rev 02)
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II
BCM5709S Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II
BCM5709S Gigabit Ethernet (rev 20)
02:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 1078 (rev 04)
03:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa)
## 03:00.0 is a bridge from 03 to 04-09
04:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa)
## 04:00.0 is a bridge from 04 to 05-05
04:01.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa)
## 04:01.0 is a bridge from 04 to 06-06
04:02.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa)
## 04:02.0 is a bridge from 04 to 07-07
04:04.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa)
## 04:04.0 is a bridge from 04 to 08-08
04:10.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa)
## 04:10.0 is a bridge from 04 to 09-09
0a:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW
WPCM450 (rev 0a)
Summary of buses:
00: Primary host bus
1e.0 Bridge to 0a-0a
1c.0 Bridge to 02-02
07.0 Bridge to 03-09
01.0 Bridge to 01-01
01: Entered via 00:01.0
02: Entered via 00:1c.0
03: Entered via 00:07.0
00.0 Bridge to 04-09
04: Entered via 03:00.0
10.0 Bridge to 09-09
04.0 Bridge to 08-08
02.0 Bridge to 07-07
01.0 Bridge to 06-06
00.0 Bridge to 05-05
0a: Entered via 00:1e.0
4) Now a dump of the switch I think is in front of my devices in question:
lspci -vv -s :04:10.0
04:10.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 255
Bus: primary=04, secondary=09, subordinate=09, sec-latency=0
I/O behind bridge: 0000e000-0000efff [size=4K]
Memory behind bridge: df000000-df1fffff [size=2M]
Prefetchable memory behind bridge: [disabled]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA+ VGA- VGA16- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
Address: 00000000fee22000 Data: 0030
Masking: 00000000 Pending: 00000000
Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
DevCap: MaxPayload 2048 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq+
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr-
TransPend-
LnkCap: Port #16, Speed 5GT/s, Width x16, ASPM L0s L1, Exit
Latency L0s <2us, L1 <2us
ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp-
LnkCtl: ASPM Disabled; Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s (ok), Width x8 (downgraded)
TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt-
SltCap: AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug+
Surprise-
Slot #240, PowerLimit 25.000W; Interlock- NoCompl-
SltCtl: Enable: AttnBtn+ PwrFlt+ MRL+ PresDet+ CmdCplt+ HPIrq+
LinkChg+
Control: AttnInd Off, PwrInd Off, Power+ Interlock-
SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
Interlock-
Changed: MRL- PresDet- LinkState-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-
NROPrPrP- LTR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt-
EETLPPrefix-
EmergencyPowerReduction Not Supported,
EmergencyPowerReductionInit-
FRS- ARIFwd+
AtomicOpsCap: Routing-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR-
OBFF Disabled, ARIFwd-
AtomicOpsCtl: EgressBlck-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-,
Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-
EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3-
LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8664 64-lane,
16-Port PCI Express Gen 2 (5.0 GT/s) Switch
Capabilities: [100 v1] Device Serial Number aa-86-00-10-b5-df-0e-00
Capabilities: [fb4 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol+
UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout-
AdvNonFatalErr-
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+
AdvNonFatalErr+
AERCap: First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn-
ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Capabilities: [138 v1] Power Budgeting <?>
Capabilities: [148 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=8
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff
Status: NegoPending- InProgress-
Port Arbitration Table <?>
Capabilities: [e00 v1] Multicast
McastCap: MaxGroups 64, ECRCRegen+
McastCtl: NumGroups 1, Enable-
McastBAR: IndexPos 0, BaseAddr 0000000000000000
McastReceiveVec: 0000000000000000
McastBlockAllVec: 0000000000000000
McastBlockUntransVec: 0000000000000000
McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr
0000000000000000
Capabilities: [f24 v1] Access Control Services
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+
EgressCtrl+ DirectTrans+
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd-
EgressCtrl- DirectTrans-
Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0
Len=010 <?>
5) Back to 10.2 to dump all the bridges:
root@pool31:~ # lspci -vv | grep bridge
00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root
Port 1 (rev 13) (prog-if 00 [Normal decode])
Memory behind bridge: da000000-ddffffff
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root
Port 7 (rev 13) (prog-if 00 [Normal decode])
I/O behind bridge: 0000e000-0000efff
Memory behind bridge: df000000-df2fffff
00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1
(rev 02) (prog-if 00 [Normal decode])
I/O behind bridge: 0000f000-0000ffff
Memory behind bridge: df300000-df3fffff
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) (prog-if 01
[Subtractive decode])
Memory behind bridge: de000000-deffffff
Prefetchable memory behind bridge: 00000000d9800000-00000000d9ffffff
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller
(rev 02)
03:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
I/O behind bridge: 0000e000-0000efff
Memory behind bridge: df000000-df1fffff
04:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
04:01.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
04:02.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
04:04.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
04:10.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
I/O behind bridge: 0000e000-0000efff
Memory behind bridge: df000000-df1fffff
After this I think I'm just generating noise, as I'm not sure what info is
needed. I await instruction.
Thank you for reading this far.
--
You are receiving this mail because:
You are the assignee for the bug.