Possible evidence of performance regression for 8.1-S (vs. 7.1)

Fri Oct 22 23:29:08 UTC 2010

On Wed, Oct 20, 2010 at 04:50:10PM -0700, Julian Elischer wrote:
> ...
> try the 7.x machine but running the 8.x kernel.. i.e. change nothing, 
> but boot the new kernel.

I just started that test, to run over the weekend.

I did run another test (3 iterations) with the 7.1 OS (kernel+userland)
on the physical machine that normally runs 8.x.  Here is a summary
of the earlier results, along with the new ones:

     start        stop      real      user       sys  host     os
1287525363  1287546846  21483.13  82628.20  21703.09  ref-7.x  7.1-R+
1287548005  1287569100  21094.63  82853.19  22185.02  ref-7.x  7.1-R+
1287570300  1287591371  21071.33  82756.81  21943.22  ref-7.x  7.1-R+
1287592592  1287614103  21511.23  82637.30  21849.90  ref-7.x  7.1-R+
1287615323  1287636770  21446.42  82715.81  21708.97  ref-7.x  7.1-R+

1287436357  1287461948  25590.99  81502.22  18115.07  ref-8.x  8.1-S at r214029
1287462797  1287488766  25969.26  81452.14  17920.14  ref-8.x  8.1-S at r214029
1287489641  1287515287  25645.84  81548.40  18256.52  ref-8.x  8.1-S at r214029
1287516151  1287541481  25329.64  81546.23  18294.10  ref-8.x  8.1-S at r214029
1287542355  1287568599  26244.59  81431.47  17902.39  ref-8.x  8.1-S at r214029

1287710312  1287732046  21733.20  82688.01  22108.95  ref-8.x  7.1-R+
1287733360  1287754549  21188.88  82869.09  21890.83  ref-8.x  7.1-R+
1287755881  1287777566  21684.09  82772.50  21933.74  ref-8.x  7.1-R+

I only ran 3 tests, as the purpose was merely a reality check --
to verify that we weren't actually seeing differences in hardware
between the two machines.

I believe that the above results provide adequate reason to believe
that hardware is not the underlying issue: the 7.1 results on the
ref-8.x machine are just about identical to the 7.1 results on the
ref-7.x machine, and both are noticably different from the 8.1
results.

> >FWIW, the workload is fairly CPU intensive during most of the run; the
> >I/O done during (most of) the test appears to be very light, and the
> >memory used is fairly modest.  In each of the test machines, I have
> >turned off HTT (HyperThreading Technology); hw.ncpu reports 8 for each.
> 
> try with HTT on for modern hardware..
> ....

OK, I'll try that experiment at some point when I have the time and
other resources, but what I'm trying to point out is that I believe
I'm seeing something that appears to be a performance regression,
comparing 7.1 (+ a few patches that were later committed to stable/7)
to stable/8 @r214029, on identically configured hardware running
the same workload (that happens to be critical for my employer).

FWIW, I tried comparing the 7.1 (as above) vs. stable/7 a couple
of months ago, and found no statistically significant difference
in performance.  Thus, I believe it would be possible for someone
else to reproduce my results, comparing a fasirly recent stable/7
to a fairly recent stable/8.  If someone does try that (with a
workload primarily composed of building software), and gets results
that differ significantly from what I'm reporting, I would appreciate
an opportunity to compare notes, as it would then be apparent that
I've managed to damage my stable/8 environment in some way.

On the other hand, if someone else tries that, and gets results
that corroborate what I'm seeing, we may have some work to do to
get stable/8 whipped into shape before 8.2-RELEASE (for which the
code freeze is currently scheduled for 28 Nov).

Granted, performance isn't the only thing that drives releases, but I
know that for the developers I support, having an 18% increase in
elapsed time to perform a critical task, merely because of an "upgrade"
to the OS, would be ... poorly received.  And there's no way I could
recommend doing that, unless the alternatives were demonstrably worse.

I'm a bit concerned, here.

Peace,
david
-- 
David H. Wolfskill				david at catwhisker.org
Depriving a girl or boy of an opportunity for education is evil.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-performance/attachments/20101022/65fd4cf6/attachment.pgp