[Bug 256264] Devices behind PEX 8664 PCIe Switch not detected since 11.0-RELEASE

From: <bugzilla-noreply_at_freebsd.org>
Date: Sun, 30 May 2021 13:58:05 +0000
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256264

--- Comment #1 from Christopher Forgeron <chris_at_acsi.ca> ---
Some lspci commands I used to try and narrow this down:


1) This is the 10.2 System. You can see both the Intel 82599ES and the LSI 2008
cards. Ignore the MegaRAID, that's a built in PERC card that's always detected
(different bus):

lspci -vvnnt
-[0000:00]-+-00.0  Intel Corporation 5500 I/O Hub to ESI Port [8086:3403]
           +-01.0-[01]--+-00.0  Broadcom Corporation NetXtreme II BCM5709S
Gigabit Ethernet [14e4:163a]
           |            \-00.1  Broadcom Corporation NetXtreme II BCM5709S
Gigabit Ethernet [14e4:163a]
           +-07.0-[03-09]----00.0-[04-09]--+-00.0-[05]--
           |                               +-01.0-[06]--
           |                               +-02.0-[07]--
           |                               +-04.0-[08]--+-00.0  Intel
Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb]
           |                               |            \-00.1  Intel
Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb]
           |                               \-10.0-[09]----00.0  LSI Logic /
Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072]
           +-14.0  Intel Corporation 7500/5520/5500/X58 I/O Hub System
Management Registers [8086:342e]
           +-14.1  Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and
Scratch Pad Registers [8086:3422]
           +-14.2  Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status
and RAS Registers [8086:3423]
           +-16.0  Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3430]
           +-16.1  Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3431]
           +-16.2  Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3432]
           +-16.3  Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3433]
           +-16.4  Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3429]
           +-16.5  Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:342a]
           +-16.6  Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:342b]
           +-16.7  Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:342c]
           +-1a.0  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#4 [8086:2937]
           +-1a.1  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#5 [8086:2938]
           +-1a.7  Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller
#2 [8086:293c]
           +-1c.0-[02]----00.0  LSI Logic / Symbios Logic MegaRAID SAS 1078
[1000:0060]
           +-1d.0  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#1 [8086:2934]
           +-1d.1  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#2 [8086:2935]
           +-1d.2  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#3 [8086:2936]
           +-1d.7  Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller
#1 [8086:293a]
           +-1e.0-[0a]----03.0  Matrox Electronics Systems Ltd. MGA G200eW
WPCM450 [102b:0532]
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
[8086:2918]
           \-1f.5  Intel Corporation 82801I (ICH9 Family) 2 port SATA
Controller [IDE mode] [8086:2926]

2) This is the same command in 13.0, same hardware:

lspci -vvnnt
-[0000:00]-+-00.0  Intel Corporation 5500 I/O Hub to ESI Port [8086:3403]
           +-01.0-[01]--+-00.0  Broadcom Inc. and subsidiaries NetXtreme II
BCM5709S Gigabit Ethernet [14e4:163a]
           |            \-00.1  Broadcom Inc. and subsidiaries NetXtreme II
BCM5709S Gigabit Ethernet [14e4:163a]
           +-07.0-[03-09]----00.0-[04-09]--+-00.0-[05]--
           |                               +-01.0-[06]--
           |                               +-02.0-[07]--
           |                               +-04.0-[08]--
           |                               \-10.0-[09]--
           +-14.0  Intel Corporation 7500/5520/5500/X58 I/O Hub System
Management Registers [8086:342e]
           +-14.1  Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and
Scratch Pad Registers [8086:3422]
           +-14.2  Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status
and RAS Registers [8086:3423]
           +-16.0  Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3430]
           +-16.1  Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3431]
           +-16.2  Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3432]
           +-16.3  Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3433]
           +-16.4  Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:3429]
           +-16.5  Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:342a]
           +-16.6  Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:342b]
           +-16.7  Intel Corporation 5520/5500/X58 Chipset QuickData Technology
Device [8086:342c]
           +-1a.0  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#4 [8086:2937]
           +-1a.1  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#5 [8086:2938]
           +-1a.7  Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller
#2 [8086:293c]
           +-1c.0-[02]----00.0  Broadcom / LSI MegaRAID SAS 1078 [1000:0060]
           +-1d.0  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#1 [8086:2934]
           +-1d.1  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#2 [8086:2935]
           +-1d.2  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller
#3 [8086:2936]
           +-1d.7  Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller
#1 [8086:293a]
           +-1e.0-[0a]----03.0  Matrox Electronics Systems Ltd. MGA G200eW
WPCM450 [102b:0532]
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
[8086:2918]
           \-1f.5  Intel Corporation 82801I (ICH9 Family) 2 port SATA
Controller [IDE mode] [8086:2926]



3) on 13.0 a quick summary:

lspci -M
00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root
Port 1 (rev 13)
## 00:01.0 is a bridge from 00 to 01-01
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root
Port 7 (rev 13)
## 00:07.0 is a bridge from 00 to 03-09
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management
Registers (rev 13)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad
Registers (rev 13)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and
RAS Registers (rev 13)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device (rev 13)
00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device (rev 13)
00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device (rev 13)
00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device (rev 13)
00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device (rev 13)
00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device (rev 13)
00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device (rev 13)
00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData
Technology Device (rev 13)
00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #4 (rev 02)
00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #5 (rev 02)
00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI
Controller #2 (rev 02)
00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1
(rev 02)
## 00:1c.0 is a bridge from 00 to 02-02
00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #1 (rev 02)
00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #2 (rev 02)
00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI
Controller #3 (rev 02)
00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI
Controller #1 (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
## 00:1e.0 is a bridge from 00 to 0a-0a
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller
(rev 02)
00:1f.5 IDE interface: Intel Corporation 82801I (ICH9 Family) 2 port SATA
Controller [IDE mode] (rev 02)
01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II
BCM5709S Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II
BCM5709S Gigabit Ethernet (rev 20)
02:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 1078 (rev 04)
03:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa)
## 03:00.0 is a bridge from 03 to 04-09
04:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa)
## 04:00.0 is a bridge from 04 to 05-05
04:01.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa)
## 04:01.0 is a bridge from 04 to 06-06
04:02.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa)
## 04:02.0 is a bridge from 04 to 07-07
04:04.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa)
## 04:04.0 is a bridge from 04 to 08-08
04:10.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa)
## 04:10.0 is a bridge from 04 to 09-09
0a:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW
WPCM450 (rev 0a)

Summary of buses:

00: Primary host bus
        1e.0 Bridge to 0a-0a
        1c.0 Bridge to 02-02
        07.0 Bridge to 03-09
        01.0 Bridge to 01-01
01: Entered via 00:01.0
02: Entered via 00:1c.0
03: Entered via 00:07.0
        00.0 Bridge to 04-09
04: Entered via 03:00.0
        10.0 Bridge to 09-09
        04.0 Bridge to 08-08
        02.0 Bridge to 07-07
        01.0 Bridge to 06-06
        00.0 Bridge to 05-05
0a: Entered via 00:1e.0

4) Now a dump of the switch I think is in front of my devices in question:

lspci -vv -s :04:10.0
04:10.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 255
        Bus: primary=04, secondary=09, subordinate=09, sec-latency=0
        I/O behind bridge: 0000e000-0000efff [size=4K]
        Memory behind bridge: df000000-df1fffff [size=2M]
        Prefetchable memory behind bridge: [disabled]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- <SERR- <PERR-
        BridgeCtl: Parity+ SERR+ NoISA+ VGA- VGA16- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
                Address: 00000000fee22000  Data: 0030
                Masking: 00000000  Pending: 00000000
        Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
                DevCap: MaxPayload 2048 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 256 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr-
TransPend-
                LnkCap: Port #16, Speed 5GT/s, Width x16, ASPM L0s L1, Exit
Latency L0s <2us, L1 <2us
                        ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp-
                LnkCtl: ASPM Disabled; Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (ok), Width x8 (downgraded)
                        TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt-
                SltCap: AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug+
Surprise-
                        Slot #240, PowerLimit 25.000W; Interlock- NoCompl-
                SltCtl: Enable: AttnBtn+ PwrFlt+ MRL+ PresDet+ CmdCplt+ HPIrq+
LinkChg+
                        Control: AttnInd Off, PwrInd Off, Power+ Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet-
Interlock-
                        Changed: MRL- PresDet- LinkState-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-
NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt-
EETLPPrefix-
                         EmergencyPowerReduction Not Supported,
EmergencyPowerReductionInit-
                         FRS- ARIFwd+
                         AtomicOpsCap: Routing-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR-
OBFF Disabled, ARIFwd-
                         AtomicOpsCtl: EgressBlck-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-,
Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-
EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3-
LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8664 64-lane,
16-Port PCI Express Gen 2 (5.0 GT/s) Switch
        Capabilities: [100 v1] Device Serial Number aa-86-00-10-b5-df-0e-00
        Capabilities: [fb4 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+
RxOF- MalfTLP- ECRC- UnsupReq- ACSViol+
                UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout-
AdvNonFatalErr-
                CEMsk:  RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+
AdvNonFatalErr+
                AERCap: First Error Pointer: 1f, ECRCGenCap+ ECRCGenEn-
ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [138 v1] Power Budgeting <?>
        Capabilities: [148 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=8
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff
                        Status: NegoPending- InProgress-
                        Port Arbitration Table <?>
        Capabilities: [e00 v1] Multicast
                McastCap: MaxGroups 64, ECRCRegen+
                McastCtl: NumGroups 1, Enable-
                McastBAR: IndexPos 0, BaseAddr 0000000000000000
                McastReceiveVec:      0000000000000000
                McastBlockAllVec:     0000000000000000
                McastBlockUntransVec: 0000000000000000
                McastOverlayBAR: OverlaySize 0 (disabled), BaseAddr
0000000000000000
        Capabilities: [f24 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+
EgressCtrl+ DirectTrans+
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd-
EgressCtrl- DirectTrans-
        Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0
Len=010 <?>

5) Back to 10.2 to dump all the bridges:

root_at_pool31:~ # lspci -vv | grep bridge
00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root
Port 1 (rev 13) (prog-if 00 [Normal decode])
        Memory behind bridge: da000000-ddffffff
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root
Port 7 (rev 13) (prog-if 00 [Normal decode])
        I/O behind bridge: 0000e000-0000efff
        Memory behind bridge: df000000-df2fffff
00:1c.0 PCI bridge: Intel Corporation 82801I (ICH9 Family) PCI Express Port 1
(rev 02) (prog-if 00 [Normal decode])
        I/O behind bridge: 0000f000-0000ffff
        Memory behind bridge: df300000-df3fffff
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) (prog-if 01
[Subtractive decode])
        Memory behind bridge: de000000-deffffff
        Prefetchable memory behind bridge: 00000000d9800000-00000000d9ffffff
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller
(rev 02)
03:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
        I/O behind bridge: 0000e000-0000efff
        Memory behind bridge: df000000-df1fffff
04:00.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
04:01.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
04:02.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
04:04.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
04:10.0 PCI bridge: PLX Technology, Inc. PEX 8664 64-lane, 16-Port PCI Express
Gen 2 (5.0 GT/s) Switch (rev aa) (prog-if 00 [Normal decode])
        I/O behind bridge: 0000e000-0000efff
        Memory behind bridge: df000000-df1fffff  



After this I think I'm just generating noise, as I'm not sure what info is
needed. I await instruction. 

Thank you for reading this far.

-- 
You are receiving this mail because:
You are the assignee for the bug.
Received on Sun May 30 2021 - 13:58:05 UTC

Original text of this message