Google Summer of Code 2014 question

[Wojciech A. Koszek]

On Thu, Mar 20, 2014 at 01:54:53PM -0700, Craig Rodrigues wrote:
> On Thu, Mar 20, 2014 at 8:24 AM, John Baldwin <jhb at freebsd.org> wrote:
> >
> > This isn't a bad place to ask this question.  I think a person you might
> > want to talk to is Craig Rodrigues (cc'd) who has recently setup Jenkins
> > with the goal of doing automated testing using virtual machines.  I'm not
> > sure exactly what design he is considering, but he is the person to ask.
> 
> Last year I signed up to be a GSoC mentor, but this year I unfortunately
> do not have time, so haven't done so.  Unfortunately, since I did not
> sign up as a mentor, I can't view your proposal on the google-melange web site.
> 
> If you look at the following link, you can see the March 13
> presentation which describes
> some work I did setting up Jenkins and BHyve for continuous
> integration and testing on FreeBSD:
> 
> https://wiki.freebsd.org/Jenkins#Presentations_and_Working_Groups
> 
> If you look at this link, item 22:
> 
> https://wiki.freebsd.org/Jenkins#Jenkins_for_FreeBSD_status
> 
> you will see that we are trying to work at integrating Jenkins with the Kyua
> framework, so that Jenkins can run Kyua, and report the results in the
> Jenkins web UI.
> 
> Jenkins is used by hundreds of companies, not just in the FreeBSD world.
> Kyua is gaining popularity in the FreeBSD community, and is already used by
> major companies (EMC/Isilon and Spectralogic are the ones that I know of).
> 
> There are already many systems out there for doing automation,
> starting/stopping VM's,
> and running tests.  I'm focusing on integrating as many existing
> technologies and
> getting them to run under FreeBSD to put together a more modern
> continuous integration and testing
> pipeline for the FreeBSD project.
> 
> I cannot comment on the specifics your proposal without reading it.
> However, I don't see a lot of value in coming up with a new Domain
> Specific Language
> for starting/stopping VM's and sending commands to VM's.  There are
> already many people
> doing this already with other frameworks and automation API's.  For
> example, libvirt ( http://libvirt.org )
> is one popular API specifically for controlling VM's.
> 
> In terms of frameworks that can send remote commands to VM's, there
> are already plenty
> of automation systems to choose from:  Puppet, Chef, SaltStack,
> CFEngine are some examples.
> 
> Creating a new Domain Specific Language for VM control sounds like a
> more academic/research exercise,
> so if you can get someone to mentor you for such a project, maybe you
> can target the results
> of your work for publication, if that is of interest to you.

John, Craig,

When I initially proposed the idea:

https://wiki.freebsd.org/SummerOfCode2014#TEST-o-steron_for_FreeBSD_.28node.js.2FLua.29

it was meant to be mainly a regression testing tool for FreeBSD.  Something
that would let me build and run continuous builds for FreeBSD ARM (ZEDBOARD
kernel), and then make a SD image out of it, which I can boot in QEMU/real
hardware.

It would be nice if we payed attention to BHyVE in this project, but not
limit ourselves to it. E.g.: ARM testing was my original drive towards
proposing this.

For such a system I'd like to be able to recognize what is being printed on
UART at the boot time. Then register the output in a file and later compare
it. And I'd like enough expressiveness to say: "if there's no login screen
after 30s of bot time, fail and let me know what the screen content at the
time of failure was".

DSL isn't a must here, I agree, but you'll be limited in testing otherwise.
Unless some technology already has what I want, I suggested getting this
functionality:

	vm0 = vm.start("vm0");
	vm1 = vm.start("vm1");

	vm0.power(1);
	vm1.power(1);

	vm0.login();
	vm1.login();

	vm0.cmd("ifconfig .....");
	vm1.cmd("ifconfig ....");

	vm0.cmd("ping vm1");

	vm0.register(function(data) {
		if (data =~ /*Not*/) {
			error();
		}
	});

Once this would work for VMs, I'd be interesting in proposing the creation
of the physical infrastructure for continuous FreeBSD testing. We could
start this from Atom-based PC and Raspberry PI netbooting latest FreeBSD.

If John sees it in a similar way and he'd like to explore this path, I'd be
very interested in mentoring this idea. But once again: only if there isn't
something of a similar functionality.

Thanks,

-- 
Wojciech A. Koszek
wkoszek at FreeBSD.czest.pl
http://FreeBSD.czest.pl/~wkoszek/