Puzzling change in performance

[Robert Watson]

On Fri, 30 Jan 2009, Borja Marcos wrote:

> The attached graphs are from a server running FreeBSD 7.1-i386 (now) with 
> the typical Apache2+MySQL with forums, Joomla...
>
> I just cannot explain this. Disk I/O bandwidth was suffering a lot, and 
> after the update the disks are almost idle.
>
> Any ideas? I cannot imagine a change between versions to justify this. The 
> latest update to 7-STABLE had been done in August.
>
> Maybe it was just degradation due to not being rebooted since August? But 
> it's strange anyway, and I had restarted MySQL and Apache some times during 
> this period.

There are basically three ways to go about exploring this, none particularly 
good:

(1) Do a more formal before and after analysis of performance on the box,
     perhaps using tools like kernel profiling, hwpmc, dtrace, etc.

(2) Do a binary search to narrow down the date of the change that improved
     things until it becomes clear which mattered.

(3) Hope someone annecdotally remembers something that might or might not be
     it and assume they're right.

Of these, I'd guess (2) is actually the most effective way to go about it, but 
is potentially time-consuming.  As you point out, the most interesting 
question is whether, when you go back to 7.0, things suddenly get slower 
again, or not.  Typically long uptimes don't lead to performance problems on 
FreeBSD (in my experience) so I think that's unlikely to be the source.  There 
are a lot of improvements in 7.1 relating to performance, but none 
particularly stands out for me as having the effect you describe.  If you're 
really curious, I would try to pin it down with a binary search.

Robert N M Watson
Computer Laboratory
University of Cambridge