Re: NVMe (U.2) hot-swap support status?

> On May 9, 2025, at 1:57 PM, Warner Losh <imp@bsdimp.com> wrote:
> 
> On Tue, May 6, 2025 at 11:31 AM Charles Sprickman <spork@bway.net> wrote:
>> 
>> 
>>> On May 6, 2025, at 1:04 AM, Colin Percival <cperciva@tarsnap.com> wrote:
>>> 
>>> On 5/5/25 21:54, Charles Sprickman wrote:
>>>> Anyhow, we're trying purchase a few servers (likely this Supermicro: https://www.supermicro.com/en/products/system/UP/1U/SYS-111E-WR) and since manufacturers don't seem to validate for FreeBSD these days, I was hoping for some community input.
>>>> Last time I looked into this NVMe (specifically the "U.2" format for servers that includes hot swap) was supported in FreeBSD but hot swap was kind of iffy. I don't think too many FreeBSD users/developers at that time had the hardware, most people seemed to just be using an on-board NVMe drive on desktop or laptop systems, and there's not any real call for hot swap in that segment.
>>>> I have a thread on the forums linked below, but really what I'm looking to find is some larger orgs that are running NVMe drives and can confirm hot swap works as expected, if there's any gotchas to be on the lookout for, etc. Trying not to overcomplicate it, but we don't want to be stuck with a bunch of servers that might be less stable than our current SAS/SATA servers. We exclusively use ZFS, if that's useful info.
>>> 
>>> I can't speak to *physical* hotplug, but I've put a lot of energy over the
>>> past few months into making sure that nvme "hotplug" is 100% functional in
>>> Amazon EC2.  If you run into problems it's probably going to be due to
>>> broken firmware,
>> [...]
>> 
>> Can you explain a bit here if you've got a few minutes?
>> 
>> I'm used to the old world concept of IDE/SCSI/SAS controllers (esp. RAID controllers) where you've got a lot of processing going on in the card to go from the drive interface to whatever kind of bus the controller sits on in the PC. But in the case of NVMe, what even is a "controller"? Is it not just a card that's combining/arbitrating PCI-e lanes from a bunch of drives to the PCI-e bus? Where would be the firmware and what would be the compatibility issues? I admit I don't use any NVMe stuff anywhere, so I'm not very familiar with it, and especially not with this setup where we have a "controller" involved...
> 
> I've had access to a couple of hotplug chassis / motherboards. For
> x86, they've just worked for me.  While the controller is bundled onto
> the nvme card, the PCIe bus has protocols to cope with a card being
> removed. FreeBSD has support for the hotplug standards around this.

So I think it's probably fair to assume that when I see someone talking about "firmware issues" that's really about the "drive", but since the media and the controller are a single entity in this case, I assume that usually would mean an issue with the controller portion? I'm assuming this "controller" is simpler than what I'm used to with a SATA/SAS/SCSI HBA since we don't have specific drivers for say, a Seagate NVMe vs. a WD NVMe vs. an Intel NVMe, etc. correct? Perhaps issues regarding this are more hypothetical than something any sizable number of FreeBSD users might be encountering?

Also found this, which is a nice overview, thank you!

https://freebsdfoundation.org/wp-content/uploads/2018/08/FreeBSD-and-NVM-Express.pdf

(where's the 2025 edition? - I kid!)

Since that is a bit dated, I'm assuming that now nda(4) is the preferred option and the default?

This all has me wishing the Foundation had a bit of an HCL for FreeBSD, fed by places like Netflix and others with committers on staff. That would be so helpful for smaller orgs that can't gamble on hardware. I don't think there's any server vendors currently that still certify for FreeBSD, which is a shame (but also kind of understandable given there's no "LTE" type release that would allow them to certify less often).

Any shopping tips when it comes to the drives? Either from you or from developers like yourself that work with very large deployments at your day jobs?

Thanks,

Charles

> 
> I've had some dodgy firmware on arm64 systems fail, though. I've not
> had the time to puzzle ou why....
> 
> Warner
> 
> 
>> Thanks,
>> 
>> Charles
>> 
>> [...]
>>> so unless anyone has experience with that specific server
>>> I think the best answer you can get is "it should work but you need to test
>>> it and find out".
>>> 
>>> --
>>> Colin Percival
>>> FreeBSD Release Engineering Lead & EC2 platform maintainer
>>> Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid
>>> 
>>> 
>> 
>>