Call for regression and performance testing - 8.0
Robert Watson
rwatson at FreeBSD.org
Wed Aug 12 12:29:36 UTC 2009
Dear all:
As we're approaching 8.0 BETA3, now would be a really good time to start
identifying functional and performance regressions from the 7.x series. We've
done a mixed job at this in the past, but when it comes to "I'll do better some
day", there's no time like the present :-). Some notes:
Functional testing
We actually have a sizable regression test suite in src/tools/regression --
some of these tools will have broken as a result of 8-CURRENT development, so
the first task may be to fix them. The next is to run them on 7-STABLE and
8-CURRENT and decide if things have gotten worse -- or maybe we have a bug to
fix in both. Pick the tool of your choice, and give it a spin. More than one
person per tool is fine, because that way we get more diverse testing.
Performance testing
On the performance front, life is always a bit more tricky -- performance
testing is a subtle art. However, the most clear lessons are that (a) testing
with diverse workloads and diverse environments is extremely important, (b) you
should do multiple runs and use ministat(1) to analyze results, and (c) that
you want to compare apples with apples -- use the same hardware/configuration
wherever possible. Watch out for annoying nits such as partition layout
affecting I/O throughput for two different installs on the same disk.
Pick something you think is important to you: bytes/sec over TCP on loopback,
web hits/sec, NFS ops/sec, disk I/O transactions/sec, and do some comparison
between 7.2 and 8.0. If you find improvement -- great! If you find a
regression, please start a thread on current@ to help get it diagnosed. And if
you want help doing performance measurement for a particular workload that
isn't well studied, send some e-mail to performance@ to ask for advice on how
best to measure it.
Thanks,
Robert N M Watson
Computer Laboratory
University of Cambridge
More information about the freebsd-current
mailing list