Running tests as a developer prior to commit

[Craig Rodrigues]

On Mon, Feb 23, 2015 at 7:20 PM, Ed Maste <emaste at freebsd.org> wrote:

>
> This is precisely why I'd like to find a way to make it easy to run a
> (rather small) subset of tests, if a developer desires to do so. The
> alternative is that they decide it's too awkward to run the tests, and
> just don't bother. Even in that case we're still better off today
> (since the Jenkins task will catch it) than we were some time ago
> before consistent test infrastructure existed. But it seems we ought
> to avoid the latency, mailing list churn, and masking effect of a
> broken test run if we can facilitate the developer running a set of
> tests that they've identified.
>

If I could do:
   cd /usr/src/bin/ls/
   make test

and have it run the unit tests associated with /bin/ls, that would be a
good start.

It wouldn't be perfect, because if a change to /bin/ls, affects some other
test
in /usr/src/usr.bin/foobar/ , this wouldn't catch it.  You would have to do
something
more complicated by building an expert system which can compute the
dependencies
in the tree and run the appropriate tests.  But having that is better than
nothing.

As far as I understand things, the way the unit tests are integrated in the
Makefile
infrastructure, the tests need to be staged to /usr/tests/ in order to be
run.
As part of the staging process, Kyuafiles are generated which are used as
input
to the "kyua" binary.  I don't have any make-fu to know how to simplify all
of this stuff.

Well, on a positive note, at least we *are* running the unit tests every
day as people
commit code, and reporting the results.  That's a huge deal.
I've seen large companies who don't do as good a job with this stuff, so we
are not doing too badly in FreeBSD.

Over time, I am sure we will improve things with better automation.

Julio Merino accomplished a huge milestone by getting *any* type of unit
tests into the FreeBSD source
tree, and integrated with the build.  Regularly running the tests under
automation was the next milestone.

The next milestone will be making it easier for developers to use.  I think
that the solution
which will be realistically achievable, which people will actually use and
not rebel over
is some type of test automation hooked into Github pull requests against
the FreeBSD src tree.

--
Craig

--
Craig