Need input on preference on location of 3rd party tests vs FreeBSD tests

Julio Merino jmmv at
Fri Jul 25 01:23:05 UTC 2014

On Thu, Jul 24, 2014 at 4:39 PM, Garrett Cooper <yaneurabeya at> wrote:
> On Thu, Jul 24, 2014 at 12:24 PM, Julio Merino <jmmv at> wrote:
>> On Fri, Jul 18, 2014 at 5:52 PM, Garrett Cooper <yanegomi at> wrote:
>>> On Jul 18, 2014, at 7:45 AM, Alan Somers <asomers at> wrote:
>>>> On Fri, Jul 18, 2014 at 1:11 AM, Garrett Cooper <yaneurabeya at> wrote:
>>>>> Hi all,
>>>>>   One of the things that I've done on my fork of FreeBSD is I've imported ATF test suites from NetBSD and I have integrated existing test suites from freebsd's tools/regression tree into Kyua as well. Due to the size and difference in test content/coverage, I pulled lib/libc and lib/msun from bother sources and integrated them into Kyua. What I did was I put the netbsd testcases into the tests/ subdirectory and put the FreeBSD test suites into a tests/legacy subdirectory. The goal was that the legacy directory would eventually be converted over to atf testcases and then could be removed once the conversion was complete.
>>>>>   I'm not sure if this scheme makes sense though. Does anyone have a preference as to whether or not this makes sense?
>>>>> Thanks!
>>>>> -Garrett
>>>> I don't understand.  What did you put in tests/legacy?
>>> The tests from tools/regression. tests/ contains the tests from NetBSD.
>> I do not think adding tests under src/tests/ is a good idea. We have
>> chosen to put tests under the 'tests' subdirectories of the affected
>> components and we should stick to this rule except for very specific
>> cases (see src/tests/sys/). Creating an artificial barrier between
>> tests imported from NetBSD and native tests to FreeBSD is not
>> beneficial (e.g. why should users/developers care where the tests came
>> from?).
> Sorry if this was a bit misleading in my original email...
> Basically I'm trying to solve the problem of keeping the FreeBSD
> testcases isolated from the NetBSD testcases. This only impacted two
> components (so far) -- libc and msun. What I did was placed the libc
> testcases from NetBSD in lib/libc/tests and the libc testcases from
> FreeBSD in lib/libc/tests/legacy (kind of matching the model that we
> have for TAP/plain scripts with the "legacy" name, but in name only).

Right... but I'm not sure if that's what FreeBSD should be doing.

I can see how, in the context of Isilon, this was the best thing to
do: keep yourself out of the business of writing tests for the base
system where possible and just reuse what NetBSD had already done.

However, we are now working towards the test suite that ships with
FreeBSD itself, and in this case we really should be fully on-board
with maintaining the tests themselves within the project. Compare this
to the development of libc itself, for example.

My biggest fear is that we will perform the initial import of tests
from NetBSD (as a bunch of files littered throughout the tree, with no
links to where they came from), then hack on the code to make it work
for FreeBSD, then add a bunch of new tests by various users... and
then never import updates from NetBSD again. There are plenty of
examples of code that was shared across projects and that never
suffered further reimports due to the difficulty of it; see the truly
obsolete smbfs in NetBSD, the pf in FreeBSD, or the way tmpfs has gone
different paths in NetBSD and FreeBSD.

This is why I wouldn't introduce any intentional divergence between
NetBSD and FreeBSD tests in the tree, in particular to prevent any
internal fears about "this code is not ours so better not touch it
much". Just import the code and, if we want to pull in new tests from
NetBSD, do it on a case-by-case basis.

(Note that I don't object to keeping the legacy tests in a
subdirectory though as that sounds reasonable. But I feel that this is
orthogonal to whether we (re)import tests from NetBSD or we write our
shiny new test programs!)

More information about the freebsd-testing mailing list