40 cores, 48 NVMe disks, feel free to take over
c.pilka at asconix.com
Sat Sep 10 08:57:18 UTC 2016
the server we got to experiment with is the SuperMicro 2028R-NR48N (https://www.supermicro.nl/products/system/2U/2028/SSG-2028R-NR48N.cfm <https://www.supermicro.nl/products/system/2U/2028/SSG-2028R-NR48N.cfm>), the board itself is a X10DSC+
> On 09 Sep 2016, at 23:14, Dennis Glatting <freebsd at pki2.com> wrote:
> On Fri, 2016-09-09 at 22:51 +0200, Christoph Pilka wrote:
>> we've just been granted a short-term loan of a server from Supermicro
>> with 40 physical cores (plus HTT) and 48 NVMe drives. After a bit of
>> mucking about, we managed to get 11-RC running. A couple of things
>> are preventing the system from being terribly useful:
>> - We have to use hw.nvme.force_intx=1 for the server to boot
>> If we don't, it panics around the 9th NVMe drive with "panic:
>> couldn't find an APIC vector for IRQ...". Increasing
>> hw.nvme.min_cpus_per_ioq brings it further, but it still panics later
>> in the NVMe enumeration/init. hw.nvme.per_cpu_io_queues=0 causes it
>> to panic later (I suspect during ixl init - the box has 4x10gb
>> ethernet ports).
>> - zfskern seems to be the limiting factor when doing ~40 parallel "dd
>> if=/dev/zer of=<file> bs=1m" on a zpool stripe of all 48 drives. Each
>> drive shows ~30% utilization (gstat), I can do ~14GB/sec write and 16
>> - direct writing to the NVMe devices (dd from /dev/zero) gives about
>> 550MB/sec and ~91% utilization per device
>> Obviously, the first item is the most troublesome. The rest is based
>> on entirely synthetic testing and may have little or no actual impact
>> on the server's usability or fitness for our purposes.
>> There is nothing but sshd running on the server, and if anyone wants
>> to play around you'll have IPMI access (remote kvm, virtual media,
>> power) and root.
>> Any takers?
> I'm curious to know what board you have. I have had FreeBSD, including
> release 11 candidates, running on SM boards without any trouble
> although some of them are older boards. I haven't looked at ZFS
> performance because mine are typically low disk use. That said, my
> virtual server (also a SM) IOPs suck but so do its disks.
> I recently found the Intel RAID chip on one SM isn't real RAID, rather
> it's pseudo RAID but for a few dollars more it could be real RAID. :(
> It was killing IOPs so I popped in an old LSI board, routed the cables
> from the Intel chip, and the server is now a happy camper. I then
> replaced 11-RC with Ubuntu 16.10 due to a specific application but I am
> also running RAIDz2 under Ubuntu on three trash 2.5T disks (I didn't do
> this for any reason other than fun).
> root at Tuck3r:/opt/bin# zpool status
> pool: opt
> state: ONLINE
> scan: none requested
> NAME STATE READ WRITE CKSUM
> opt ONLINE 0 0 0
> raidz2-0 ONLINE 0 0 0
> sda ONLINE 0 0 0
> sdb ONLINE 0 0 0
> sdc ONLINE 0 0 0
>> Christoph Pilka
>> Modirum MDpay
>> Sent from my iPhone
>> freebsd-questions at freebsd.org <mailto:freebsd-questions at freebsd.org> mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-questions <https://lists.freebsd.org/mailman/listinfo/freebsd-questions>
>> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freeb
>> sd.org <http://sd.org/>"
> freebsd-questions at freebsd.org <mailto:freebsd-questions at freebsd.org> mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-questions <https://lists.freebsd.org/mailman/listinfo/freebsd-questions>
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org <mailto:freebsd-questions-unsubscribe at freebsd.org>"
More information about the freebsd-questions