Continual benchmarking / regression testing?

[Alan Somers]

On Tue, Jan 7, 2014 at 10:49 AM, Ivan Voras <ivoras at freebsd.org> wrote:
> On 7 January 2014 18:01, Alan Somers <asomers at freebsd.org> wrote:
>> Unfortunately, my framework is extremely incomplete.  It's not even
>> good enough for internal use, much less a wider audience.  And I fear
>> that my bosses won't give me any more time to work on it.  It's also
>> written in Ruby and uses STAF to command slave nodes, which the
>> FreeBSD community might not be excited about.  However, if there is
>> any interest, I can ask for permission to share my design as a
>> starting point for a more general framework.
>
> I don't think I have the time+spare brainpower to learn Ruby (I'm a
> Pythonista) but it would be interested to see the design you've
> created.
>
> I've looked at STAF before and I'm curious why you picked it? It
> seemed a combination of being antiquated and an overkill to me...

I picked Ruby and STAF for the same reason: we're already using them
at $JOB.  In the case of STAF, we were already using it as part of our
continuous install and test system.  We aren't using very many of its
capabilities, though.  In particular, we've long since switched from
STAX to ATF/Kyua.  Basically, we only use STAF to execute commands on
a remote machine.  It offers a little more control than using SSH.  We
also use the STAF semaphore service to ensure exclusive access to both
test machines and network slaves.