collecting statistics / metrics

[Andriy Gapon]

on 03/04/2013 20:36 Alfred Perlstein said the following:
> Hey folks, sorry for the top post here, but I just came into this thread.
> 
> Here at iXsystems we've just developed a set of scripts to scrape the various
> FreeBSD user land utilities (sysctl, netstat, nfsstat, vmstat, etc, etc) and put
> them into graphs based on time.
> 
> The goal is to be able to line up all these metrics with whatever benchmark we
> are currently running and be able to see what may be causing issues.
> 
> Potentially you should be able to scroll through the graphs and see things like
> "ran out of mbufs @time", "vm system began paging at @time", "buffer deaemon
> went nuts @time"
> 
> Then we can take the information back and leverage it to make tuning decisions,
> or potentially change kernel algorithms.

This is very very useful!

> The only problem we have is that every user land tool has its own format, so
> along with my team we have written some shell to coerce the output from the
> various programs into pseudo-CSV (key/value pair) which can then be post
> processed by tools to convert to CSV which can then be put into something like
> open office, or put through an R program to graph it.
> 
> I'm hoping to have something shortly.
> 
> What I was hoping to do over the next few days was discuss with people how we
> can (or should we even) fix the user land statistics tools to output machine
> readable output that can be easily parsed.
> 
> Example: netstat -m  (hard to parse) versus 'vmstat -z | grep mbuf' easy to parse.
> 
> The idea of outputting xml is good, CSV is OK, however CSV is problematic as in
> the case of sysctl, if new nodes appear, then we can't begin to emit them, we
> must either ignore them, or abort, or log them to auxiliary files.  Anything
> that makes life easier is good.
> 
> I should be able to share our scripts within the next couple of days.

Just an alternative idea...
I think gathering all this information via plugins to e.g. collectd could be
more flexible and less processing / parsing intensive.  That would allow to
avoid unnecessary formatting and re-parsing and to store the data in a
convenient format. Ideally it would be great to have an umbrella library on top
of sysctl, devstat, etc that would expose various stats in a convenient form.
Another thing of convenience would be an ability to know which sysctls are
actually stats.  I think that you have already done work towards this goal.
There are certain heuristics that may help to distinguish stats from knobs,
constants, etc, but the explicit "this is a metric" should be used.  Of course,
it would take a lot of work to properly mark all the sysctls.

Just thinking out loud.

-- 
Andriy Gapon