Scaling and performance issues with FreeBSD 9 (& 10) on 4 socket systems

[David O'Brien]

Hi all,
$WORK is looking to see what throwing money at a build can do.

We've been building the product on FreeBSD 8.3/amd64 in a 7.1/i386 jail,
on on Supermicro X8DT3
http://www.supermicro.com/products/motherboard/qpi/5500/x8dt3.cfm
w/Xeon X5690 3.47GHz: "2 package(s) x 6 core(s) x 2 SMT threads".
These are nicknamed "Montana-2" or "M2".

The new machines are Supermicro X9QR7-TF+/X9QRi-F+
http://www.supermicro.com/products/motherboard/Xeon/C600/X9QR7-TF_.cfm
w/Xeon E5-4650 2.70GHz: "4 package(s) x 8 core(s) x 2 SMT threads".
These are nicknamed "jbm" (build machine) during their testing phase.
As part of this we also upgraded the host FreeBSD to 9.1 from 8.3.

We've found that a "make -j28" build on the M2 machines is considerably
faster than on the jbm machines.  While somewhat faster might not be
surprising given the faster CPU's, the build is 2x longer on jbm.
(jbm runs are FreeBSD 9.1/amd64 with the same 7.1/i386 jail installation
as the M2)

In addition, we've found the scaling on jbm to be quite bad as additional
jobs or threads or processes are run.

----------%<----------%<----------%<----------%<----------%<----------

In order to better quantify this, a co-worker who's been playing with
Bitcoin suggested the Bitcoin "vanitygen" multi-threaded application
as a good test.  We have results of 1-64 threads on a jbm machine running
(1) FreeBSD 9.1/amd64, (2) FreeBSD 10-current/amd64 [no Witness],
(3) FreeBSD 8.4/amd64, (4) FreeBSD 10-current/amd64 SCHED_4BSD, &
(5) Fedora 18/amd64.  The results are graphed at
http://people.freebsd.org/~obrien/jbm/vanitygen/vanity-perf-graph.png

We found FreeBSD 8.4 to perform better than FreeBSD 9.1, and Linux
considerably better than both on the same machine.

----------%<----------%<----------%<----------%<----------%<----------

To get more data, I ran /usr/ports/benchmarks/sysbench/ and see
similar concerning results between the M2 machines and jbm machines
(on the native amd64 host, not within the i386 jail).
This is a multi-threaded application. The graph for that is
http://people.freebsd.org/~obrien/jbm/sysbench/sysbench.png

----------%<----------%<----------%<----------%<----------%<----------

Simon also did some benchmarking using an integrity file signing server.
This one is a multi-process benchmark doing 2k RSA+SHA1 calculations.
The graph for that is
http://people.freebsd.org/~obrien/jbm/sigs/sigs.png

----------%<----------%<----------%<----------%<----------%<----------

We've tried various things and haven't been able to explain why FreeBSD
isn't scaling on the new hardware.  Nor why it performs so much worse
than FreeBSD on the older "M2" machines.

Thoughts?
-- 
-- David  (obrien at FreeBSD.org)

P.S. vanitygen is from 'git clone git://github.com/samr7/vanitygen.git'
and builds with this patch:
--- Makefile.ORIG	2013-06-06 21:39:10.000000000 -0700
+++ Makefile	2013-06-06 21:38:45.000000000 -0700
@@ -1,2 +1,2 @@
-LIBS=-lpcre -lcrypto -lm -lpthread
-CFLAGS=-ggdb -O3 -Wall
+LIBS=-L/usr/local/lib -lpcre -lcrypto -lm -lpthread
+CFLAGS=-ggdb -O3 -Wall -I/usr/local/include