Migrating tools/regression/ tests (and dealing with TAP)
Julio Merino
julio at meroh.net
Mon Nov 25 18:29:57 UTC 2013
Hello all,
Here come sample patches to move existing tests in tools/regression/bin
to the new layout ('tests' subdirectories within the corresponding tools)
and to hook these tests into the test suite.
The patches are quite straightforward: I just did an 'svn mv' of the
existing shell scripts files, wrote a Makefile using plain.test.mk (here
is where things get interesting) and applied minor changes to get all
things sorted out (e.g. mtree directory registration).
Consider this a non-working proof of concept. Again: this does NOT work
(well, it does, but yields invalid results). The problem is that
TAP-based test programs (like the ones to be run by prove) do NOT report
1 on failure and thus the plain Kyua interface is unsuitable because it
always reports everything as passed.
There are two solutions:
1) Patch every existing TAP-based test to propagate a 1 error code if
any of its tests fails. Doable, yes, but it will take a while.
(Mind you: this is the approach we tried in NetBSD and not all old
tests are gone yet, years later... it's a significant amount of work
to manually tweak all test programs!)
I'd rather keep existing code as is *for now*; it simplifies the
review of the changes and it doesn't immediately break people's
knowledge of existing test code. (If there is a need for a rewrite,
I'd suggest to go and use the ATF interface instead. But that's even
more work.)
2) Add a TAP-compliant backend to Kyua. This would allow us to plug
pretty much all existing tests in tools/regression/ without changes
to the code. I believe this is a worthy goal for the transitional
step and an easy first cut.
But this means I've gotta implement this backend (fun!). I've been
reading the protocol and seems reasonably simple, so I expect to have
something workable this week.
I'm choosing option 2 for now while leaving open the option of rewriting
tests using ATF *only after* they have been hooked into the build.
(What I'm shooting for, if you haven't guessed yet, is having a single
tests tree instead of 3 in src ASAP... Keeping multiple trees is not a
pretty situation for long.
Once the new Kyua backend is ready, the proposed patches in this mail
should change minimally: basically do s/plain/prove/ in the Makefiles.
This is why I'm sending this mail out already: you can get a pretty
good idea of what I'd like to do and your feedback would be useful!
There is a "tricky" side change to be made to bsd.progs.mk so that it is
possible to have a directory with SCRIPTS but no PROGS. make tinderbox
is happy.
Comments?
Thanks!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: handle-scripts.diff
Type: text/x-diff
Size: 1642 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-testing/attachments/20131125/86608484/attachment.diff>
More information about the freebsd-testing
mailing list