Need input on preference on location of 3rd party tests vs FreeBSD tests
jmmv at freebsd.org
Thu Jul 24 19:24:34 UTC 2014
On Fri, Jul 18, 2014 at 5:52 PM, Garrett Cooper <yanegomi at gmail.com> wrote:
> On Jul 18, 2014, at 7:45 AM, Alan Somers <asomers at freebsd.org> wrote:
>> On Fri, Jul 18, 2014 at 1:11 AM, Garrett Cooper <yaneurabeya at gmail.com> wrote:
>>> Hi all,
>>> One of the things that I've done on my fork of FreeBSD is I've imported ATF test suites from NetBSD and I have integrated existing test suites from freebsd's tools/regression tree into Kyua as well. Due to the size and difference in test content/coverage, I pulled lib/libc and lib/msun from bother sources and integrated them into Kyua. What I did was I put the netbsd testcases into the tests/ subdirectory and put the FreeBSD test suites into a tests/legacy subdirectory. The goal was that the legacy directory would eventually be converted over to atf testcases and then could be removed once the conversion was complete.
>>> I'm not sure if this scheme makes sense though. Does anyone have a preference as to whether or not this makes sense?
>> I don't understand. What did you put in tests/legacy?
> The tests from tools/regression. tests/ contains the tests from NetBSD.
I do not think adding tests under src/tests/ is a good idea. We have
chosen to put tests under the 'tests' subdirectories of the affected
components and we should stick to this rule except for very specific
cases (see src/tests/sys/). Creating an artificial barrier between
tests imported from NetBSD and native tests to FreeBSD is not
beneficial (e.g. why should users/developers care where the tests came
"Reusing the NetBSD Makefiles" is not a strong enough reason because
those are usually trivial -- and if they aren't, that's an indication
that they are wrong!
We have two options:
1) We treat the NetBSD tests as upstream. In this case, we ought to do
the right thing, which is to put them in a vendor branch, merge into
contrib, and add reachover Makefiles where necessary.
2) We just copy the code (effectively a fork), and when doing so we
can do as we wish with the structure.
Option 1 may sound nice, but it will be problematic long-term without
collaboration from NetBSD: I don't think we want to treat NetBSD's
tests as upstream if NetBSD themselves don't consider the tests as a
"first class project" (and they don't currently). Not to mention that
this will require a significant amount of local patching with the
corresponding maintenance headaches.
Option 2 is definitely easier to handle on our side, but with the
obvious disadvantage that we won't be able to easily import tests from
NetBSD. How big of a problem is that today? Honestly the fact that our
codebases are separate for things like libc is more of a problem than
having diverging tests.
Option 3 (yes, another one!) would be to start a third-party
repository holding "generic Unix tests" (a POSIX test suite?) that
both NetBSD and FreeBSD import as a third-party product. Then, each
system would just extend these suites with local patches to verify
their system-specific behavior. A very nice option in my opinion, but
this is a major effort. (And this can be tackled later once we the
FreeBSD Test Suite is more complete and we can easily spot how much it
overlaps to the NetBSD one.)
More information about the freebsd-testing