NVMe performance 4x slower than expected

[Tobias Oberstein]

> I believe FreeBSD 10.2-RELEASE includes an updated nvme (4) driver. Do these
> changes fix the performance issues described in this thread?

I don't know.

The problems with the machine we've deployed however run deeper than 
"just NVMe":

RAM: 3TB .. the FBSD kernel patches that fixed this might be in 10.2 .. 
dunno

amount of PCIx resources: the box has 8x NVMe, 2x SAS controller, 2x 
dual-port 10GbE .. FBSD freaked out due to this .. there were kernel 
patches also .. again no clue if those are in mainline now

And finally, the bummer: NUMA performance. The box has 4 sockets, 48 
cores, and we are moving to 64 cores.

My unscientific impression was, that FBSD isn't yet able to cope with that.

Anyway, in the meantime, we've moved from SLES 12 to Ubuntu 15.04 ( 
Linux kernel 3.19.3) .. due to recent Linux developments: blkmq and such.

8kB random write performance now is now at >1 million IOPS (over a test 
duration of 10h and a 1TB dataset at block device level - RAID-0 over 8x 
2TB NVMe). That's nearly 40% more than on SLES 12, and the CPU load is 
lower also. So we can saturate the hardware IOPS wise.

Cheers,
/Tobias