Hints for precision benchmarking...

[Robert Watson]

On Mon, 26 Jan 2004, Poul-Henning Kamp wrote:

> *   Run in single user mode.  Cron(8) and and other daemons
>     only add noise.

Of particular interest here is sshd -- either disable its SSHv1 key
regeneration, or kill the parent sshd (or don't run sshd) during the
tests. 

A few obvious ones just worth remembering:

(1) Don't use bgfsck unless you want to benchmark with bgfsck.  Also,
    disable background_fsck in rc.conf unless you plan not to start the
    benchmark for at least 60+fudge seconds after the boot, as rc wakes up
    and checks to see if anything needs fscking if it is enabled.
    Likewise, make sure you don't have snapshots lying around unless you
    mean to teset with snapshots. 

(2) If you must leave the system connected to a public network, watch out
    for spikes of broadcast traffic.  Even though you don't notice it, it
    will eat your CPU.  Multicast has similar caveats.

(3) If you see unexpectedly bad performance, check for things like
    high interrupt volume from an unexpected source.  I've seen several
    reports of ACPI "misbehaving" and generating excess interrupts under
    some of the ACPI code drops.  If you're seeing odd results, take a few
    snapshots of vmstat -i and look for anything unusual.

(4) Make sure to be careful about optimization parameters for kernel and
    userspace, likewise debugging.  It's easy to let something slip
    through and realize later you weren't comparing the same thing.

(5) And finally, do not ever benchmark with WITNESS and INVARIANTS unless
    you are really just interested in benchmarking those features.  :-)
    I've seen 400%+ drops in performance from running with WITNESS.
    Likewise, userspace malloc paramters default differently in -CURRENT
    from the way they'd ship in production. 

Also, it would be very helpful if, while benchmarking, people would do the
same benchmark with SCHED_ULE and SCHED_4BSD, and see what impact it has. 
Scheduler changes are very influential to performance, and we want to make
sure we catch edge cases and glitches that will now be being seen with
more broad testing.

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert at fledge.watson.org      Senior Research Scientist, McAfee Research