Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server

[Garrett Cooper]

My personal thoughts on all of the complaints that FreeBSD isn't fast
enough and the Phoronix benchmarks aren't representative of true
FreeBSD performance.

Disclaimer: I don't know if the Phoronix benchmarks do tuning out of
the box or not on Linux, so if they do, please correct me Matthew.

The ultimate problem from my perspective is multifold:

- There's an unknown -- seemingly unquantifiable -- SCHED_ULE
scheduler performance issue that people are trying to root cause.

ZFS:
- Out of the box requires extensive tuning in order to get reasonable
performance if you're doing something intensive on the machine,
depending on what features you enable as well (hint: don't ever use
dedup if you want performance).
- FreeBSD's ZFS assumes that it owns all of the hardware except in
certain low memory scenarios (VMs) from what I've seen.
- It's also hardware hungry (RAM, SSD or better L2ARC and ZIL, etc),
unlike certain filesystems like ext*, and UFS SU+J. Depending on what
operations you perform (asynch/synch writes, etc), throwing hardware
at the problem will fix things.

In general:
- FreeBSD is currently 'tuned' for the L.C.D. compute machine, which
means my Soekris box (if I had one) or an AMD/Intel 48+ core machine
can run FreeBSD out of the box with GENERIC (ex: `cpu I486_CPU',
etc').
- GENERIC's also full of a bunch of drivers and options that are
potentially useless for standard users depending on my machine and
what I'm trying to achieve (ex: `device fdc', `makeoptions DEBUG=-g',
`options FLOWTABLE', etc).
- Some of the default tuning parameters don't work for compute
intensive or interactive scenarios, and in many cases they're mutually
exclusive (kern.sched.preempt_thresh comes to mind).
- Our timecounter mechanism has been noted as slow compared to Linux
because of its precision (clock_gettime and friends), which also
affects other dependent syscalls if I'm not mistaken (epoll, select,
etc?) which are used extensively in some tests and applications. IMHO
we should have a less precise timecounter on by default (if it's good
enough for the system and the user doesn't see an issue, it should be
good enough for the masses).
- There are a lot of other fancy features in Linux that help
performance that don't exist on FreeBSD (ex: a true tickless kernel,
better NUMA architecture support, etc).

And the list goes on and on.

The point is that while some of the suggestions have been good on how
to write good benchmarks (someone suggested a medium math and a worker
benchmark set of tests, which I think was on the right track), a lot
of the shouting on the list has been missing the key point based on
Matthew's statement a while back about using *untuned* FreeBSD in his
benchmarks:

- In order to use FreeBSD [well] on more recent hardware, I need to
tune the OS in order for it to perform reasonably close to what Linux
performs at.

My personal suggestion based on personal observations to Matthew:

- Don't touch ZFS unless you have the hardware to make it go faster,
i.e. at least an enterprise SSD for your ZIL and L2ARC, and at least
12GB of RAM. It's slow otherwise if you slam the box, compared to UFS
SU+J.
- If you run X11 on the box you need to set an rtprio for Xorg, but I
really advise not using X11 in your benchmarks because it introduces
noise in the benchmarks -- in part because the featuresets between
Linux and FreeBSD as far as what's enabled and how things are coded up
can differ a *lot* (ex: many developers code things up with inotify,
not kqueues or even libevent, and can sprinkle little architecture
specific #ifdef's around the code), but also because interaction at
some level with the syscalls differs enough on a micro scale that it
can affect the macro picture.

Thanks,
-Garrett