Plugging ATF tests into the build and other cleanups

[Simon J. Gerraty]

On Tue, 29 Oct 2013 17:46:19 -0400, Julio Merino writes:
>> You can't even do OLD_FILES+= ${KYUAFILES:H}/*
>
>I'm confused... Is that "can't" supposed to be "can"?  Or in other
>words, do you mean that this would or would not work if we assumed
>that ATF/Kyua are the only thing populating /usr/tests ?

No it wouldn't work - the existing delete-old-* targets are very careful
about only deleting what was explicitly listed.
That's probably a good thing.

>That's part of the rationale for the next patch: to make /usr/tests/
>conditional on a single knob (WITH_TESTS) so that such assumptions can
>be made: either you install the full test suite or you don't. I don't

You could make that argument yes, and further, if folk want their own
test suites they could be told to put them somewhere else.

Since this isn't critical one way or the other yet, I'd leave some time
for folk to ponder/discuss - ie. just ignore cleaning /usr/tests for now.

>> freebsd-testing/recurse-subdir.diff
>>
>> I'm not so crazy about.
>> What's wrong with requiring folk to set TESTS_SUBDIR[S] ?
>
>Because SUBDIR and TESTS_SUBDIRS serve different purposes.  With

This patch seems to blur that distinction no?

>In the current patchset, this is useful in, for example,
>src/lib/atf/Makefile (see last patch) : this file needs to generate a
>Kyuafile that registers its 3 subdirectories (libatf-c, libatf-c++ and
>test-programs) so that the recursion can be performed at runtime.
>There is no need to list those as TESTS_SUBDIRS because they are
>already in SUBDIR.

But for this corner case, listing those dirs in TESTS_SUBDIRS (despite
also being in SUBDIR) would do no great harm, and would avoid the need
to blur the distiction noted above.

>There are some alternatives to this behavior:
>
>1. Move the generation of the Kyuafile to src/lib/atf/tests/Makefile.
>But if you do that, you cannot get the automatic generation to work
>because that tests/ directory does not have any children (and thus the
>automatic generation code has no idea about what to do).

I'm not sure I fully understand the issue (given what we are doing - see
below).

>2. Manually set TESTS_SUBDIR in src/lib/atf/Makefile to SUBDIR...
>which would be suboptimal because this is something that could be done
>automatically (this patch).

Optimizing for corner cases isn't necessarily a good idea.
If it turns out that this is a common issue, we can always revisit.
So this (or the next) would be my suggestion - for now.

>3. Make src/lib/atf/tests/Makefile install a generic Kyuafile that
>discovers its directories, just like /usr/tests/Kyuafile and
>/usr/tests/lib/Kyuafile do.  This is a reasonable optional and would
>follow your comments below of keeping stuff in tests/

This is what we actually do.
Before packaging tests, a generic Atffile is dropped into each dir under
${STAGE_OBJTOP}/usr/tests/ that does not already have one, this seems to
work ok.

>> freebsd-testing/move-kyuafiles.diff
>>
>> I'm also not crazy about this one (the change to lib/Makefile)
>> What is the expected use?
>
>To populate /usr/tests/lib accordingly.  You need:
>
>/usr/tests/Kyuafile - auto-recurse into directories
>/usr/tests/lib/Kyuafile - auto-recurse into directories
>/usr/tests/lib/libc/Kyuafile - list of tests to run and possibly other
>subdirectories
>and that /usr/tests/lib/Kyuafile has to come from somewhere.  I think

Sure, but if Kyuafile works anything like Atffile, a generic boilerplate 
one would do? and this could be handled from almost anywhere.
Since we (junos - and same in projects/bmake) ignore intermediate
makefiles like lib/ etc, that wouldn't be my choice.
Doing it in a leaf dir, works just as well and is then independent of
the orchestration of the build.

>We could do this from src/lib/tests/ as easily; didn't think about it.
> Gave this a try and updated the patches accordingly; check them out.

Will do.

>Still open is the question if we should do the same for
>src/lib/atf/Makefile et. al.  I suspect the answer is yes, but then
>there is the issue mentioned above of the Kyuafile auto-generation.
>(Haven't done it yet as I don't have the time now and would appreciate
>your input.)

All of the Atffiles in our stage tree are autogenerated.
The leaf dirs usr/tests/bin/cat/Atffile are done by atf.test.mk:

$ cat ../obj/stage/i386/usr/tests/bin/cat/Atffile
Content-Type: application/X-atf-atffile; version="1"

# Automatically generated by atf-test.mk.

prop: test-suite = "FreeBSD"

tp: t_cat
$

and the intermediate dirs by the method mentioned above

$ cat ../obj/stage/i386/usr/tests/bin/Atffile    
Content-Type: application/X-atf-atffile; version="1"

prop: test-suite = "FreeBSD"
tp-glob: *
$ ls -li ../obj/stage/i386/usr/tests/bin/Atffile
93925694 -rw-r--r--  26 sjg  wheel  94 Sep 17 14:27 ../obj/stage/i386/usr/tests/bin/Atffile
$ cat ../obj/stage/i386/usr/tests/bin/Atffile.dirdep
juniper/pkgs/os-tests/finish.i386
$

that last let's you know who put it there.

>Indeed, having a SRCTOP would have made things easier... but I didn't
>find that available.

No, not yet.  It is needed for dirdeps.mk though.

>Note: I have applied some of your suggestions to the existing patches
>but some things still remain open so don't bother doing an in-depth
>review again because they are broken.  However, I gotta go now and I'd
>prefer to send this out for further comments while I have your
>attention ;)

Sure - thanks very much for your help.

--sjg