PATCH: add ATF tests in sys

[Julio Merino]

On Jan 21, 2014, at 19:23, Garrett Cooper <yaneurabeya at gmail.com> wrote:

>> 
>> On Jan 21, 2014, at 16:02, Alan Somers <asomers at freebsd.org> wrote:
>> 
>>> On Tue, Jan 21, 2014 at 4:56 PM, Garrett Cooper <yaneurabeya at gmail.com> wrote:
>>> 
>>>> On Jan 21, 2014, at 14:45, Alan Somers <asomers at freebsd.org> wrote:
>>>> 
>>>> I rewrote the unix seqpacket tests in ATF.  The hard part is adding
>>>> them to the build.  The linked patch adds them in the directory
>>>> sys/kern/tests and builds them as part of buildworld, not buildkernel.
>>>> They get installed to /usr/tests/sys/kern, but the intermediate
>>>> objects are stored in /usr/obj/sys/kern/tests.  That means that you
>>>> can't have different tests associated with different kernel configs.
>>>> I think that this is desirable, because you wouldn't be able to
>>>> install tests for different kernel configs anyway, given our chosen
>>>> layout of /usr/tests.
>>>> 
>>>> Please comment on the parts of this patch that deal with Makefiles.
>>>> Is this the appropriate way to add sys tests to the build?  Shouldn't
>>>> I be building them in buildkernel instead of buildworld?  I couldn't
>>>> find a good way to do that.
>>> 
>>> I bypassed that for simplicity and placed the tests in tests/sys/kern/... etc. Shoehorning things into our overly complex kernel build system just seems like a really bad idea... Just make it load as a driver, loadable, and dependent on kern.features to run..?
>>> Cheers!
>>> -Garrett
>> 
>> I tried to place the tests as closely as possible to the source code
>> that they're testing, according to
>> https://wiki.freebsd.org/TestSuite/Structure#Makefiles_for_test_programs
>> .  I don't feel strongly about it either way, but I think Simon and
>> jmmv do.

Yes in general, but... see below.

>> 
>> FWIW, the tests run as a userland program, not in kernel mode.
> 
> Fwiw I spent months mulling over various details (this was just one point) with gnn, marcel, and mfleming. The consensus was that that was a really bad idea because it wasn't terribly clear that tests covered a particular area, and some sys/ tests could cover multiple areas (pjdfstest for instance).
> 
> I mirrored the exact structure in the source tree, but avoided tainting things in sys/... Because integrating sys/ into buildworld and starting to mix kernel and world build logic together will irritate a large number of kernel devs and userland devs. This was the path of least resistance...

I think Garrett has a good point here and, if the build within src/sys/ is significantly different from the build of userspace tools and has different restrictions, it is probably a good idea to treat the kernel tests as "special".

After all, these tests are userland programs as you well said and they are a kind of "integration" tests because there is no way currently to unit-test kernel code.  Such programs will just do "stuff" hoping to validate kernel code, but there are so many levels of indirection that they cannot be though of as testing things in very fine-grain detail.

(If we had rump-like features as NetBSD, things may be a bit different though because in that case you really are writing unit tests for very specific kernel code.)

Using src/tests/sys/ and replicating the layout of src/sys/ as much as possible within it may be a reasonable and easy compromise for now.