Need input on preference on location of 3rd party tests vs FreeBSD tests

[Garrett Cooper]

On Thu, Jul 24, 2014 at 12:24 PM, Julio Merino <jmmv at freebsd.org> wrote:
> On Fri, Jul 18, 2014 at 5:52 PM, Garrett Cooper <yanegomi at gmail.com> wrote:
>> On Jul 18, 2014, at 7:45 AM, Alan Somers <asomers at freebsd.org> wrote:
>>
>>> On Fri, Jul 18, 2014 at 1:11 AM, Garrett Cooper <yaneurabeya at gmail.com> wrote:
>>>> Hi all,
>>>>   One of the things that I've done on my fork of FreeBSD is I've imported ATF test suites from NetBSD and I have integrated existing test suites from freebsd's tools/regression tree into Kyua as well. Due to the size and difference in test content/coverage, I pulled lib/libc and lib/msun from bother sources and integrated them into Kyua. What I did was I put the netbsd testcases into the tests/ subdirectory and put the FreeBSD test suites into a tests/legacy subdirectory. The goal was that the legacy directory would eventually be converted over to atf testcases and then could be removed once the conversion was complete.
>>>>   I'm not sure if this scheme makes sense though. Does anyone have a preference as to whether or not this makes sense?
>>>> Thanks!
>>>> -Garrett
>>>
>>>
>>> I don't understand.  What did you put in tests/legacy?
>>
>> The tests from tools/regression. tests/ contains the tests from NetBSD.
>
> I do not think adding tests under src/tests/ is a good idea. We have
> chosen to put tests under the 'tests' subdirectories of the affected
> components and we should stick to this rule except for very specific
> cases (see src/tests/sys/). Creating an artificial barrier between
> tests imported from NetBSD and native tests to FreeBSD is not
> beneficial (e.g. why should users/developers care where the tests came
> from?).

Sorry if this was a bit misleading in my original email...

Basically I'm trying to solve the problem of keeping the FreeBSD
testcases isolated from the NetBSD testcases. This only impacted two
components (so far) -- libc and msun. What I did was placed the libc
testcases from NetBSD in lib/libc/tests and the libc testcases from
FreeBSD in lib/libc/tests/legacy (kind of matching the model that we
have for TAP/plain scripts with the "legacy" name, but in name only).
Example:

- https://github.com/yaneurabeya/freebsd/tree/isilon-atf/lib/libc/tests
- the NetBSD testcases (mostly)
- https://github.com/yaneurabeya/freebsd/tree/isilon-atf/lib/libc/tests/legacy
- the testcases from tools/regression on FreeBSD head

> "Reusing the NetBSD Makefiles" is not a strong enough reason because
> those are usually trivial -- and if they aren't, that's an indication
> that they are wrong!

Well, they're really trivial, yes, and apart from small caveats with
bsd.progs.mk in HEAD today, there isn't a reason for the testcase
Makefiles to be divergent, except when dealing with FreeBSD vs NetBSD
MK_ knobs.

> We have two options:
>
> 1) We treat the NetBSD tests as upstream. In this case, we ought to do
> the right thing, which is to put them in a vendor branch, merge into
> contrib, and add reachover Makefiles where necessary.

This is what I was thinking of doing because I don't have a compelling
argument and it doesn't really seem to make sense to fork [at this
point]. The modifications that Peter and I have been making to the
isilon-atf branch are relatively trivial/documented with #if
defined(__FreeBSD__) or #if defined(__NetBSD__) (still not happy with
the __arraycount vs nitems divergence, but it's a simple sed
one-liner). I've pushed a number of patches out to NetBSD GNATS to fix
issues with the testcases (mostly header discrepancies, but also some
valid testcase bugs with NetBSD's copy of the tests).

> OR
>
> 2) We just copy the code (effectively a fork), and when doing so we
> can do as we wish with the structure.
>
> So...
>
> Option 1 may sound nice, but it will be problematic long-term without
> collaboration from NetBSD: I don't think we want to treat NetBSD's
> tests as upstream if NetBSD themselves don't consider the tests as a
> "first class project" (and they don't currently). Not to mention that
> this will require a significant amount of local patching with the
> corresponding maintenance headaches.
>
> Option 2 is definitely easier to handle on our side, but with the
> obvious disadvantage that we won't be able to easily import tests from
> NetBSD. How big of a problem is that today? Honestly the fact that our
> codebases are separate for things like libc is more of a problem than
> having diverging tests.
>
> Option 3 (yes, another one!) would be to start a third-party
> repository holding "generic Unix tests" (a POSIX test suite?) that
> both NetBSD and FreeBSD import as a third-party product. Then, each
> system would just extend these suites with local patches to verify
> their system-specific behavior. A very nice option in my opinion, but
> this is a major effort.  (And this can be tackled later once we the
> FreeBSD Test Suite is more complete and we can easily spot how much it
> overlaps to the NetBSD one.)

This is an alternative as well, but the tricky thing is that it moves
away from the whole idea of "having the tests in the tree" which was
part of the point behind doing things this way (to get developers in
the habit of writing tests that match the test code), unless both
FreeBSD and NetBSD collaboratively pull test code from the "BSD Test
Project".

Thanks!
-Garrett