Performance Tracker project update

[Erik Cederstrand]

Kris Kennaway wrote:
> 
> This is coming along very nicely indeed!
> 
> One suggestion I have is that as more metrics are added it becomes 
> important for an "at a glance" overview of changes so we can monitor for 
> performance improvements and regressions among many workloads.
 >
> One way to do this would be a matrix of each metric with its change 
> compared to recent samples.  e.g. you could do a student's T comparison 
> of today's numbers with those from yesterday, or from a week ago, and 
> colour-code those that show a significant deviation from "no change". 
> This might be a bit noisy on short timescales, so you could aggregrate 
> data into larger bins and compare e.g. moving 1-week aggregates. 
> Fluctuations on short timescales won't stand out, but if there is a real 
> change then it will show up less than a week later.

I agree that there's a need for an overview and some sort of 
notification. I've been collecting historical data to get a baseline for 
the statistics and I'll try to see what I can do over the next weeks.

> These significant events could also be graphed themselves and/or a 
> history log maintained (or automatically annotated on the individual 
> graphs) so historical changes can also be pinpointed.
> 
> At some point the ability to annotate the data will become important 
> (e.g. "We understand the cause of this, it was r1.123 of foo.c, which 
> was corrected in r1.124.  The developer responsible has been shot.")

There's a field in the database for this sort of thing. I just think it 
needs some sort of authentication. That'll have to wait a bit.

> P.S. If I understand correctly, the float test shows a regression?  The 
> metric is calculations/second, so higher = better?

The documentation on Unixbench is scarce, but I would think so.

BTW if anyone's interested my SVN repo is online at:

svn://littlebit.dk/website/trunk    (Pylons project)
svn://littlebit.dk/tracker/trunk    (sh/Python scripts for runnning the 
server and slaves)

Be careful with your eyes - this is my first attempt at both shell 
scripting and Python :-)

Erik