ATF config variables for FreeBSD tests

[Alan Somers]

On Mon, Mar 17, 2014 at 9:14 AM, Alan Somers <asomers at freebsd.org> wrote:
> On Sun, Mar 16, 2014 at 10:53 AM, Garrett Cooper <yaneurabeya at gmail.com> wrote:
>>
>>> On Mar 16, 2014, at 8:56, Alan Somers <asomers at freebsd.org> wrote:
>>>
>>>> On Sat, Mar 15, 2014 at 6:49 PM, Julio Merino <jmmv at freebsd.org> wrote:
>>>>> On Sun, Mar 16, 2014 at 5:33 AM, Alan Somers <asomers at freebsd.org> wrote:
>>>>> On Fri, Mar 14, 2014 at 10:21 PM, Julio Merino <jmmv at freebsd.org> wrote:
>>>> [...]
>>>>>> I think these all sound reasonable.
>>>>>>
>>>>>> Can these expected side-effects be reversed in the test cleanup?
>>>>>
>>>>> For the first two, yes.  I doubt that Kyua could do it automatically,
>>>>> but each test case certainly can.
>>>>
>>>> Right. I do not think Kyua should get into these tricky details
>>>> either, if only because they are too OS-specific. But we can certainly
>>>> offer additional scripts/functions (where it makes sense) in the
>>>> FreeBSD test suite to simplify the test cases that might need this.
>>>
>>> Definitely.  This is a good case for a library of reusable test code.
>>> For example, I have two separate test programs that would like to
>>> share setup and cleanup code for FIBs.  Should I put it in
>>> /usr/tests/include/fibs.sh or /usr/tests/lib/fibs.sh ?  I don't like
>>> those options, because it looks like fibs.sh is a test program
>>> designed to test stuff in /include or /lib.  Perhaps
>>> /usr/tests/tests/sh/include/fibs.sh?  I'm really not sure where the
>>> best place would be.  What do you think?
>>
>> /usr/tests/libexec seems logical..
>
> But that's where tests for the stuff in /libexec go.  If anything, I
> think /usr/libexec/tests would be more appropriate.  Unfortunately,
> such files would share the same source location as the tests for the
> stuff in /libexec.  But I think it's preferable to have a confusing
> source layout to a confusing installed layout.
>
>>
>>>>
>>>>> Perhaps even for the fourth, though
>>>>> it would be tricky.  But there's no way to reverse the effects for the
>>>>> third.  The ZFS tests, for example, create and destroy zpools.  How
>>>>> would you reverse that?  You can't.  Whatever was previously on the
>>>>> disk is gone.
>>>>
>>>> Ah, I guess I missed that detail.
>>>>
>>>> Couldn't those tests run on top of vnd devices though?
>>>
>>> FreeBSD doesn't have vnd(4), though it has md(4), which is similar.
>>> md(4) devices could be used for some ZFS tests.  as it happens, ZFS
>>> can also use file-backed vdevs.  But these workarounds don't work in
>>> all cases.  Sometimes you need real disks:
>>> 1) Using file-backed vdevs doesn't exercise vdev_geom.c, where I've
>>> one a lot of work.
>>> 2) Neither file-backed vdevs nor md(4) devices have physical path
>>> information, which is needed to test some hotspare functionality.
>>> Only da(4) devices that are attached to ses(4) expanders have that.
>>> 3) There is no way to remove an active file-backed vdev, and I don't
>>> think that you can destroy an in-use md(4) device either.  Therefore,
>>> to test how ZFS handles drive removals requires real drives.
>>> 4) gibbs has made many tweaks to ZFS to better support 4K and 8K
>>> sector drives.  There is no way to emulate those with file-backed
>>> vdevs.  Perhaps it could be done with md(4) and gnop(8), but I've
>>> never tried.
>>> 5) pjd and Steve Hartland have been working on TRIM support.  That's
>>> not supported by file-backed vdevs, and I doubt that md(4) supports it
>>> either.
>>> 6) Copy-on-write on top of copy-on-write is very slow.  If your system
>>> uses ZFS root, then both file-backed vdevs and file-backed md(4)
>>> devices are doing COW-on-COW.  Using physical disks for the tests
>>> greatly speeds the tests' runtimes.
>>>
>>> Ideally, the ZFS tests would use test_suites.FreeBSD.disks if it is
>>> defined, and use file-backed vdevs otherwise.  Tests that absolutely
>>> require physical disks would be skipped if test_suites.FreeBSD.disks
>>> isn't defined.  It would take some work, but it's doable.
>>>
>>>>
>>>> [...]
>>>>>> And lastly, we'd just need a simple "filtering" feature in the kyua
>>>>>> cli to allow specifying which size of tests to run (or to filter by
>>>>>> any other metadata property, for that matter).
>>>>>
>>>>> I don't like this idea.
>>>>
>>>> I hope you are objecting to the filtering by test sizes, not the
>>>> filtering itself!  See below.
>>>>
>>>>> We tried it at work, and it didn't work out
>>>>> very well.  Basically, I automatically assigned sizes (short, medium,
>>>>> long) to all of our tests based on their runtimes, so users could
>>>>> select to only run the short or medium tests on the bench.  The
>>>>> problem is that the classification didn't make sense.  For an
>>>>> expedited test run, you don't want the shortest tests; what you want
>>>>> are the tests with the most value per unit time.  Unit tests have a
>>>>> high value and a very short runtime.  Stress tests have a very long
>>>>> runtime and arguably low value since they don't always have consistent
>>>>> results.
>>>>>
>>>>> At my previous job there was a large department whose duties included
>>>>> curating test suites and deciding which tests would be included in the
>>>>> short runs.  FreeBSD isn't going to have that, but we could still ask
>>>>> test authors to classify tests along the lines that I suggested.
>>>>
>>>> That's a good point. If I understand you correctly, what is important
>>>> is to manually curate the set of "smoke tests" that run very quickly
>>>> and that provide a reasonably high assurance that major subsystems
>>>> haven't broken.
>>>>
>>>> We can do this too pretty easily, but configuration variables are the
>>>> wrong mechanism. We can already define this information in metadata
>>>> properties as long as you prefix them with X-. So all we'd need is a
>>>> way to tell Kyua to only run tests marked with such property (i.e.
>>>> with the filtering feature described above).
>>>
>>> I was assuming that a test case could put in its header
>>> 'require.config "stress_tests"', or something like that.  Then runtime
>>> filtering would work by which variables the user defines.  That way,
>>> no new work would be required in Kyua.  But it sounds like you're
>>> talking about something different.  IIUC, you're suggesting that a
>>> test case would say in its header 'atf_set "X-runtime" "stress_test"'
>>> and then some TBD filtering mechanism in Kyua selects on it.  That
>>> would certainly be more intuitive.  Is that what you had in mind?
>>>
>>> -Alan

I committed the man page change for the four variables we agreed on.
But I didn't add the runtime selection variables.  It sounds like
Julio has bigger plans for those, and they'll require upstream work on
Kyua.

-Alan