Re: NVMe (U.2) hot-swap support status?

<<On Fri, 9 May 2025 11:57:32 -0600, Warner Losh <imp@bsdimp.com> said:

> I've had access to a couple of hotplug chassis / motherboards. For
> x86, they've just worked for me.  While the controller is bundled onto
> the nvme card, the PCIe bus has protocols to cope with a card being
> removed. FreeBSD has support for the hotplug standards around this.

It definitely did not work in 13.x last summer when I needed to do
this; I had to take an extended[1] outage to swap out 32 NVMe drives
in a file server rather than doing it online, and I believe I asked
about it on this very list and was told about all the things that
didn't yet work or which required extra manual steps for NVMe that
normal hot-swap SAS/SATA backplanes don't.  Some of those things can
be resolved by switching from nvd to nda (not the default in
13-stable) and adding `options PCI_HP` (which may be the default, we
don't run GENERIC).

This was with a standard PCIe-bridge-based chassis; I don't know how
it works with a Broadcom 9500 HBA (we only use SAS devices on those).

-GAWollman

[1] As in, copy all the data off to another server, destroy and
recreate the zpool, rather than having to power-cycle once to swap
each of 32 drives.