Running tests as a developer prior to commit

[Ed Maste]

On 23 February 2015 at 21:37, Craig Rodrigues <rodrigc at freebsd.org> wrote:
>
> If someone touches an arbitrary piece of code in FreeBSD,
> what are the subset of tests that need to be run to validate the change?
> Some changes are very simple, and it is quite easy to extrapolate which
> tests need to be run.
>
> However, some changes are more subtle, and it is not so obvious which
> tests are affected.

Absolutely, but I'm not suggesting that we change the current setup,
or avoid proceeding with other ideas you raise below. We'll still have
the full test run from Jenkins if someone commits after running a
chosen subset of tests.

The use case I'm considering is someone making a change to a single
binary under one of the bin directories, which has Kyua tests in its
own subdirectory.

My goal is to be able to:
- run the test against the not-yet-installed tool built in $OBJDIR
- report overall pass/failure and 0/non-0 return code (in addition to
other output)
- easily obtain stdout / stderr for failing tests to track down issues

> The other suggestion I have seen in other companies is to require developers
> to run all unit tests for every change before they commit.  This works to
> some
> extent, because in a company you can force this behavior.
>
> For FreeBSD, I think this will fail.  We already have over 3000 tests under
> /usr/tests.
> It's not that hard to run them with "cd /usr/tests && kyua test".
> However, requiring FreeBSD developers to run these tests
> for every change they do is a big change in how they submit code to FreeBSD.
> This will probably result in rebellion.

This is precisely why I'd like to find a way to make it easy to run a
(rather small) subset of tests, if a developer desires to do so. The
alternative is that they decide it's too awkward to run the tests, and
just don't bother. Even in that case we're still better off today
(since the Jenkins task will catch it) than we were some time ago
before consistent test infrastructure existed. But it seems we ought
to avoid the latency, mailing list churn, and masking effect of a
broken test run if we can facilitate the developer running a set of
tests that they've identified.

> I would like to see a compromise solution.  I would like an optional
> workflow,
> where someone does a pull request on the FreeBSD repository in GitHub, which
> then
> kicks off automation which builds an image, runs the kyua tests, and reports
> the results.

I think a project like this would be excellent to have, and I would
personally make use of it. As you say it will be a lot of work, and it
still introduces a great deal of latency over "make test" for a simple
userland binary change.