Performance Tracker project update

[gnn at freebsd.org]

At Wed, 23 Jan 2008 14:49:14 +0100,
Erik Cederstrand wrote:
> 
> Kris Kennaway wrote:
> > 
> > This is coming along very nicely indeed!
> > 
> > One suggestion I have is that as more metrics are added it becomes 
> > important for an "at a glance" overview of changes so we can monitor for 
> > performance improvements and regressions among many workloads.
>  >
> > One way to do this would be a matrix of each metric with its change 
> > compared to recent samples.  e.g. you could do a student's T comparison 
> > of today's numbers with those from yesterday, or from a week ago, and 
> > colour-code those that show a significant deviation from "no change". 
> > This might be a bit noisy on short timescales, so you could aggregrate 
> > data into larger bins and compare e.g. moving 1-week aggregates. 
> > Fluctuations on short timescales won't stand out, but if there is a real 
> > change then it will show up less than a week later.
> 
> I agree that there's a need for an overview and some sort of 
> notification. I've been collecting historical data to get a baseline for 
> the statistics and I'll try to see what I can do over the next weeks.
> 
> > These significant events could also be graphed themselves and/or a 
> > history log maintained (or automatically annotated on the individual 
> > graphs) so historical changes can also be pinpointed.
> > 
> > At some point the ability to annotate the data will become important 
> > (e.g. "We understand the cause of this, it was r1.123 of foo.c, which 
> > was corrected in r1.124.  The developer responsible has been shot.")
> 
> There's a field in the database for this sort of thing. I just think it 
> needs some sort of authentication. That'll have to wait a bit.
> 
> > P.S. If I understand correctly, the float test shows a regression?  The 
> > metric is calculations/second, so higher = better?
> 
> The documentation on Unixbench is scarce, but I would think so.
> 
> BTW if anyone's interested my SVN repo is online at:
> 
> svn://littlebit.dk/website/trunk    (Pylons project)
> svn://littlebit.dk/tracker/trunk    (sh/Python scripts for runnning the 
> server and slaves)
> 
> Be careful with your eyes - this is my first attempt at both shell 
> scripting and Python :-)
> 

BTW This is excellent work and, if I get the time, I'll be using some
of this at my day job.

Thanks,
George