Devices behind PEX 8664 PCIe Switch not detected since 11.0-RELEASE

From: Christopher Forgeron <csforgeron_at_gmail.com>
Date: Tue, 1 Jun 2021 22:44:39 -0300
Hello,

 I have not been able to upgrade past FreeBSD 10.3 as since versions
11.0 -> 13.0 devices behind my fileserver's PCIe Switch are not
detected. The switch itself is detected, which is a PEX 8664 PCIe
Switch, link below.

 I'm using a Dell M610x blade server, which has a PEX/Broadcom 8664
Switch in it to provide 2 PCIe slots, which I use to run SAS
controllers to a drive array.

 This arrangement has worked perfectly since the early days of 7/8.0,
and I've pretty much ran each version of FreeBSD since then, until
11.0 came out.

 This is when I could no longer see/detect devices plugged into the
PCIe slots. Things were busy, FreeBSD 10.2 worked well, so it was just
left this way for years.

 Now I do need to upgrade software-wise to 13.0, so I've invested some
time tracking down what is happening, with the hopes that a
knowledgeable person can fix this up.

 - I have confirmed with 10.3-RELEASE and a 11.0-RELEASE mini-iso boot
that the problem starts in 11.0, and I can confirm it's still an issue
in 12.0 and 13.0. I also boot into a 10.2-RELEASE because I happen to
have that handy.  Everything works/detects fine in 10.3 or lower.

 - These are GENERIC kernels, fresh from the iso/img I boot from,
nothing custom.

 - I currently have a 10.2 (or 10.3) boot and a 13.0 boot ready to
swap back and forth to try and isolate / probe info. I'll attach a few
lspci dumps after this message.

 - The card in question is a LSI 2008 Chipset, mps0 - But it's nothing
do do with that card/brand/config. I have also installed an Intel X540
Network card in the PCIe slots and find the same issue - There's
nothing behind the PEC 8664 in FreeBSD 13.0, and no problems in 10.2.
I have also confirmed that 13.0 boots up and detects the mps0 without
issue when I use a different system without the PEX 8664

 - At some point in my research I came across a similar issue where
the developer mentioned that for some reason they were not scanning
behind the switch. It was a different chipset and issue, and  I don't
have that thread handy at the moment, I will post if I come across it
again.

Ref:https://www.broadcom.com/products/pcie-switches-bridges/pcie-switches/pex8664

I'm rusty, but fairly knowledgeable and useful if you need me to
compile something, install patches, etc. I could give SSH access, but
sending me a patch to test may be quicker.

lspci dumps are in my bug report here:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256264

Thanks.
Received on Wed Jun 02 2021 - 01:44:39 UTC

Original text of this message