Server with 40 physical cores, 48 NVMe disks, feel free to test it

Warner Losh imp at bsdimp.com
Sun Sep 11 03:34:15 UTC 2016


On Sat, Sep 10, 2016 at 2:58 AM, Christoph Pilka <c.pilka at asconix.com> wrote:
> Hi,
>
> we've just been granted a short-term loan of a server from Supermicro with 40 physical cores (plus HTT) and 48 NVMe drives. After a bit of mucking about, we managed to get 11-RC running. A couple of things are preventing the system from being terribly useful:
>
> - We have to use hw.nvme.force_intx=1 for the server to boot
> If we don't, it panics around the 9th NVMe drive with "panic: couldn't find an APIC vector for IRQ...". Increasing hw.nvme.min_cpus_per_ioq brings it further, but it still panics later in the NVMe enumeration/init. hw.nvme.per_cpu_io_queues=0 causes it to panic later (I suspect during ixl init - the box has 4x10gb ethernet ports).

John Baldwin has patches that help fix this.

> - zfskern seems to be the limiting factor when doing ~40 parallel "dd if=/dev/zer of=<file> bs=1m" on a zpool stripe of all 48 drives. Each drive shows ~30% utilization (gstat), I can do ~14GB/sec write and 16 read.
>
> - direct writing to the NVMe devices (dd from /dev/zero) gives about 550MB/sec and ~91% utilization per device

These are slow drives then if all they can do 600MB/s. The drives
we're looking at do 3.2GB/s read and 1.6GB/s write from the drives
that we're looking at.

48 drives though. Woof. What's the interconnect? There's enough PCIe
lanes for that? 192 lanes? How's that possible?

> Obviously, the first item is the most troublesome. The rest is based on entirely synthetic testing and may have little or no actual impact on the server's usability or fitness for our purposes.
>
> There is nothing but sshd running on the server, and if anyone wants to play around you'll have IPMI access (remote kvm, virtual media, power) and root.

Don't think I have enough time to track this all down...

Warner


More information about the freebsd-fs mailing list