CoC does not help in benchmarks

[Edward Tomasz Napierała]

It's not just that - part of the problem is that Phoronix Test Suite
seems be focused more on producing pretty graphs than obtaining any
meaningful results.  It fails at basic statistics - error bars are
nowhere to be found.  Even if we assume the differences in numbers
are not just measurement errors, they might come from the compiler
differences, or just comparing completely different pieces of software:
the "gzip compression", which shows an order of magnitude of difference
between FreeBSD and Clear Linux, mentions "patches its carrying for
threading"; the "git benchmark" is at least clear that it compares
different git versions - ranging from 1.7.1 to 2.18.0.  Then there's
the PHPBench, where there's 4x difference between Linux versions
(Scientific Linux and Clear Linux).  There are also differences
in compiler options and mount flags.

Then there's the fact that when they actually try to compare the
same piece of software, they build it from vanilla sources instead
of using official ports/packages.  And they use the default
configuration, ignoring basic guidelines that are generally known
to people who use the software - disabling update_process_titles,
for example.  Which means they run something different from what
people actually use.

To put it bluntly: Phoronix benchmarks are utterly useless for
anything apart from generating page views.

On 0715T1258, CL Moonriver wrote:
> Can you clarify what you mean? Performance benchmarks like that are, 
> after all, often of limited value when it comes to why one might choose 
> one OS over another. A lot of other things factor into the decision. 
> Security, stability, consistency, ease of configuration and 
> administration, etc.
> 
> 
> On 07/13/18 17:44, Erich Dollansky wrote:
> > Hi,
> >
> > here are the consequences of putting a CoC up high on the priority list:
> >
> > https://www.phoronix.com/scan.php?page=article&item=windows-freebsd112-8linux&num=1
> >
> > Focusing on software would have made FreeBSD do better.
> >
> > Erich