I like iostat, but...

[Stefan Parvu]

> ... I rather wish I could get the same information via sysctl.  (Well,
> something seems to be available via the "opaque" kern.devstat.all
> sysctl(8) variable, but sysctl(8) doesn't display all of it, and parsing
> it seems as if that would require knowledge about the internals of the
> system where the data were acquired.)

I gave up parsing sysctl via Perl for disks and network devices. It would be
nice to have devstat properly working via sysctl for disk devices. Similar way
kern.cp_times does. Currently there is no simple way to extract per disk stats from
sysctl as a Perl or Sh consumer, unless we build a C module to do that. 

> If iostat(8) could be taught to (optionally) provide a timestamp, that
> might suffice.

In fact all performance userland tools should be able to nicely produce timestamp
CSV records which can be used for capacity planning and feed them to an
analytic product. Something like:

1411445589:4.01:16.03:383.97:1.93:0.29:1.68:0.11:95.99:0.00:0.00:0.00:0.00:123.00:229516.00:570992.00

or something like iostat:

1411445733:ada0:0.00:0.00:0.00:0.00:0.00:0.00:0.00

where the first field would be always the timestmp, unix time. It is not that complicated
but it does not exist.
 
> The problem I'm trying to solve is this: I need to be able to acquire
> various resource counters (along with timestamps), so I can post-process
> the acquired data (generally, on a system other than the one where the
> data were gathered) in order to be able to see how the
> resource-consumption changes over the duration of a moderately
> long-running task (typically. 0.5 - 8 hrs.).

My idea of having standard data recorders: sysrec, cpurec, nicrec diskrec 
which can extract: overall system consumption, per device statistics.
http://www.systemdatarecorder.org/recording/agents.html 

First place to start with will be sysrec, the main recorder which will report overall
system consumption:

  * cpu utilization across all CPUs
  * memory utilization
  * disk IO across all disk devices
  * network IO across all NICs
  * LA
http://www.systemdatarecorder.org/recording/sysrec_freebsd.html


> I believe that having an ability to correlate the "iostat -x"
> information with the CPU, load average, and memory utilization would be
> useful -- but I don't see a reasonable way to go about it.  If anyone
> has suggestions, I'm listening. :-}

How about sysrec, like describe above. I dont like this version because it is using 
iostat, netstat for disks and NICs but I dont have a better solution right now. 

There is another way for NIC stats to use libstatgrab and a Perl module for it. I got some
troubles in using it under FreeBSD 10 and having some considerable memory 
usage over time, and I did not have time to test it further. But I will a bit later. 
There are some limitations about libstatgrab: does not know per CPU stats for
example.


http://www.i-scream.org/libstatgrab/

-- 
Stefan Parvu <sparvu at systemdatarecorder.org>