ATF config variables for FreeBSD tests

[Alan Somers]

On Sun, Mar 16, 2014 at 10:53 AM, Garrett Cooper <yaneurabeya at gmail.com> wrote:
>
>> On Mar 16, 2014, at 8:56, Alan Somers <asomers at freebsd.org> wrote:
>>
>>> On Sat, Mar 15, 2014 at 6:49 PM, Julio Merino <jmmv at freebsd.org> wrote:
>>>> On Sun, Mar 16, 2014 at 5:33 AM, Alan Somers <asomers at freebsd.org> wrote:
>>>> On Fri, Mar 14, 2014 at 10:21 PM, Julio Merino <jmmv at freebsd.org> wrote:
>>> [...]
>>>>> I think these all sound reasonable.
>>>>>
>>>>> Can these expected side-effects be reversed in the test cleanup?
>>>>
>>>> For the first two, yes.  I doubt that Kyua could do it automatically,
>>>> but each test case certainly can.
>>>
>>> Right. I do not think Kyua should get into these tricky details
>>> either, if only because they are too OS-specific. But we can certainly
>>> offer additional scripts/functions (where it makes sense) in the
>>> FreeBSD test suite to simplify the test cases that might need this.
>>
>> Definitely.  This is a good case for a library of reusable test code.
>> For example, I have two separate test programs that would like to
>> share setup and cleanup code for FIBs.  Should I put it in
>> /usr/tests/include/fibs.sh or /usr/tests/lib/fibs.sh ?  I don't like
>> those options, because it looks like fibs.sh is a test program
>> designed to test stuff in /include or /lib.  Perhaps
>> /usr/tests/tests/sh/include/fibs.sh?  I'm really not sure where the
>> best place would be.  What do you think?
>
> /usr/tests/libexec seems logical..

But that's where tests for the stuff in /libexec go.  If anything, I
think /usr/libexec/tests would be more appropriate.  Unfortunately,
such files would share the same source location as the tests for the
stuff in /libexec.  But I think it's preferable to have a confusing
source layout to a confusing installed layout.

>
>>>
>>>> Perhaps even for the fourth, though
>>>> it would be tricky.  But there's no way to reverse the effects for the
>>>> third.  The ZFS tests, for example, create and destroy zpools.  How
>>>> would you reverse that?  You can't.  Whatever was previously on the
>>>> disk is gone.
>>>
>>> Ah, I guess I missed that detail.
>>>
>>> Couldn't those tests run on top of vnd devices though?
>>
>> FreeBSD doesn't have vnd(4), though it has md(4), which is similar.
>> md(4) devices could be used for some ZFS tests.  as it happens, ZFS
>> can also use file-backed vdevs.  But these workarounds don't work in
>> all cases.  Sometimes you need real disks:
>> 1) Using file-backed vdevs doesn't exercise vdev_geom.c, where I've
>> one a lot of work.
>> 2) Neither file-backed vdevs nor md(4) devices have physical path
>> information, which is needed to test some hotspare functionality.
>> Only da(4) devices that are attached to ses(4) expanders have that.
>> 3) There is no way to remove an active file-backed vdev, and I don't
>> think that you can destroy an in-use md(4) device either.  Therefore,
>> to test how ZFS handles drive removals requires real drives.
>> 4) gibbs has made many tweaks to ZFS to better support 4K and 8K
>> sector drives.  There is no way to emulate those with file-backed
>> vdevs.  Perhaps it could be done with md(4) and gnop(8), but I've
>> never tried.
>> 5) pjd and Steve Hartland have been working on TRIM support.  That's
>> not supported by file-backed vdevs, and I doubt that md(4) supports it
>> either.
>> 6) Copy-on-write on top of copy-on-write is very slow.  If your system
>> uses ZFS root, then both file-backed vdevs and file-backed md(4)
>> devices are doing COW-on-COW.  Using physical disks for the tests
>> greatly speeds the tests' runtimes.
>>
>> Ideally, the ZFS tests would use test_suites.FreeBSD.disks if it is
>> defined, and use file-backed vdevs otherwise.  Tests that absolutely
>> require physical disks would be skipped if test_suites.FreeBSD.disks
>> isn't defined.  It would take some work, but it's doable.
>>
>>>
>>> [...]
>>>>> And lastly, we'd just need a simple "filtering" feature in the kyua
>>>>> cli to allow specifying which size of tests to run (or to filter by
>>>>> any other metadata property, for that matter).
>>>>
>>>> I don't like this idea.
>>>
>>> I hope you are objecting to the filtering by test sizes, not the
>>> filtering itself!  See below.
>>>
>>>> We tried it at work, and it didn't work out
>>>> very well.  Basically, I automatically assigned sizes (short, medium,
>>>> long) to all of our tests based on their runtimes, so users could
>>>> select to only run the short or medium tests on the bench.  The
>>>> problem is that the classification didn't make sense.  For an
>>>> expedited test run, you don't want the shortest tests; what you want
>>>> are the tests with the most value per unit time.  Unit tests have a
>>>> high value and a very short runtime.  Stress tests have a very long
>>>> runtime and arguably low value since they don't always have consistent
>>>> results.
>>>>
>>>> At my previous job there was a large department whose duties included
>>>> curating test suites and deciding which tests would be included in the
>>>> short runs.  FreeBSD isn't going to have that, but we could still ask
>>>> test authors to classify tests along the lines that I suggested.
>>>
>>> That's a good point. If I understand you correctly, what is important
>>> is to manually curate the set of "smoke tests" that run very quickly
>>> and that provide a reasonably high assurance that major subsystems
>>> haven't broken.
>>>
>>> We can do this too pretty easily, but configuration variables are the
>>> wrong mechanism. We can already define this information in metadata
>>> properties as long as you prefix them with X-. So all we'd need is a
>>> way to tell Kyua to only run tests marked with such property (i.e.
>>> with the filtering feature described above).
>>
>> I was assuming that a test case could put in its header
>> 'require.config "stress_tests"', or something like that.  Then runtime
>> filtering would work by which variables the user defines.  That way,
>> no new work would be required in Kyua.  But it sounds like you're
>> talking about something different.  IIUC, you're suggesting that a
>> test case would say in its header 'atf_set "X-runtime" "stress_test"'
>> and then some TBD filtering mechanism in Kyua selects on it.  That
>> would certainly be more intuitive.  Is that what you had in mind?
>>
>> -Alan
>> _______________________________________________
>> freebsd-testing at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-testing
>> To unsubscribe, send any mail to "freebsd-testing-unsubscribe at freebsd.org"