Running tests as a developer prior to commit

[Garrett Cooper]

On Feb 26, 2015, at 19:17, Craig Rodrigues <rodrigc at FreeBSD.org> wrote:

> On Mon, Feb 23, 2015 at 7:20 PM, Ed Maste <emaste at freebsd.org> wrote:
> 
>> 
>> This is precisely why I'd like to find a way to make it easy to run a
>> (rather small) subset of tests, if a developer desires to do so. The
>> alternative is that they decide it's too awkward to run the tests, and
>> just don't bother. Even in that case we're still better off today
>> (since the Jenkins task will catch it) than we were some time ago
>> before consistent test infrastructure existed. But it seems we ought
>> to avoid the latency, mailing list churn, and masking effect of a
>> broken test run if we can facilitate the developer running a set of
>> tests that they've identified.
>> 
> 
> If I could do:
>   cd /usr/src/bin/ls/
>   make test
> 
> and have it run the unit tests associated with /bin/ls, that would be a
> good start.
> 
> It wouldn't be perfect, because if a change to /bin/ls, affects some other
> test
> in /usr/src/usr.bin/foobar/ , this wouldn't catch it.  You would have to do
> something
> more complicated by building an expert system which can compute the
> dependencies
> in the tree and run the appropriate tests.  But having that is better than
> nothing.
> 
> As far as I understand things, the way the unit tests are integrated in the
> Makefile
> infrastructure, the tests need to be staged to /usr/tests/ in order to be
> run.
> As part of the staging process, Kyuafiles are generated which are used as
> input
> to the "kyua" binary.  I don't have any make-fu to know how to simplify all
> of this stuff.
> 
> Well, on a positive note, at least we *are* running the unit tests every
> day as people
> commit code, and reporting the results.  That's a huge deal.
> I've seen large companies who don't do as good a job with this stuff, so we
> are not doing too badly in FreeBSD.
> 
> Over time, I am sure we will improve things with better automation.
> 
> Julio Merino accomplished a huge milestone by getting *any* type of unit
> tests into the FreeBSD source
> tree, and integrated with the build.  Regularly running the tests under
> automation was the next milestone.
> 
> The next milestone will be making it easier for developers to use.  I think
> that the solution
> which will be realistically achievable, which people will actually use and
> not rebel over
> is some type of test automation hooked into Github pull requests against
> the FreeBSD src tree.

Hi,

This hook (running tests from a subdir) already exists, but is incomplete:

$ make test
*** Using this test does not preclude you from running the tests
*** installed in /usr/tests.  This test run may raise false
*** positives and/or false negatives.

*** WARNING: make test is experimental
***
*** Using this test does not preclude you from running the tests
*** installed in /usr/tests.  This test run may raise false
*** positives and/or false negatives.

kyua: E: Load of '/usr/tests/bin/pkill/Kyuafile' failed: File '/usr/tests/bin/pkill/Kyuafile' not found.
*** Error code 2

Stop.
make: stopped in /usr/src.svn/bin/pkill/tests

I have code on github which expands make test to install everything to ${.OBJDIR}, then run the tests.

make testworld doesn’t exist either, so I’ll work on plumbing that into the build system, because having to cd into every directory to run tests is silly.

Thanks for the reminder :).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-testing/attachments/20150226/d3683246/attachment.sig>