Re: NVMe (U.2) hot-swap support status?

> On May 9, 2025, at 2:22 PM, Garrett Wollman <wollman@bimajority.org> wrote:
> 
> <<On Fri, 9 May 2025 11:57:32 -0600, Warner Losh <imp@bsdimp.com> said:
> 
>> I've had access to a couple of hotplug chassis / motherboards. For
>> x86, they've just worked for me.  While the controller is bundled onto
>> the nvme card, the PCIe bus has protocols to cope with a card being
>> removed. FreeBSD has support for the hotplug standards around this.
> 
> It definitely did not work in 13.x last summer when I needed to do
> this; I had to take an extended[1] outage to swap out 32 NVMe drives
> in a file server rather than doing it online, and I believe I asked
> about it on this very list and was told about all the things that
> didn't yet work or which required extra manual steps for NVMe that
> normal hot-swap SAS/SATA backplanes don't.  Some of those things can
> be resolved by switching from nvd to nda (not the default in
> 13-stable) and adding `options PCI_HP` (which may be the default, we
> don't run GENERIC).

I'm curious if you've tried hot swap since then and what happened? And also if you found your original discussion on this anywhere. :)

Can anyone else comment on generally how swapping a drive on a ZFS system might differ from the "old" SATA/SAS/SCSI way? I don't think I've ever encountered PCI hot swap in the wild myself. Is there any documentation I'm missing outside of the manpages that you're aware of that steps through the process?

Thanks,

Charles

> This was with a standard PCIe-bridge-based chassis; I don't know how
> it works with a Broadcom 9500 HBA (we only use SAS devices on those).
> 
> -GAWollman
> 
> [1] As in, copy all the data off to another server, destroy and
> recreate the zpool, rather than having to power-cycle once to swap
> each of 32 drives.
>