ATF config variables for FreeBSD tests

Garrett Cooper yaneurabeya at
Sun Mar 16 16:53:44 UTC 2014

> On Mar 16, 2014, at 8:56, Alan Somers <asomers at> wrote:
>> On Sat, Mar 15, 2014 at 6:49 PM, Julio Merino <jmmv at> wrote:
>>> On Sun, Mar 16, 2014 at 5:33 AM, Alan Somers <asomers at> wrote:
>>> On Fri, Mar 14, 2014 at 10:21 PM, Julio Merino <jmmv at> wrote:
>> [...]
>>>> I think these all sound reasonable.
>>>> Can these expected side-effects be reversed in the test cleanup?
>>> For the first two, yes.  I doubt that Kyua could do it automatically,
>>> but each test case certainly can.
>> Right. I do not think Kyua should get into these tricky details
>> either, if only because they are too OS-specific. But we can certainly
>> offer additional scripts/functions (where it makes sense) in the
>> FreeBSD test suite to simplify the test cases that might need this.
> Definitely.  This is a good case for a library of reusable test code.
> For example, I have two separate test programs that would like to
> share setup and cleanup code for FIBs.  Should I put it in
> /usr/tests/include/ or /usr/tests/lib/ ?  I don't like
> those options, because it looks like is a test program
> designed to test stuff in /include or /lib.  Perhaps
> /usr/tests/tests/sh/include/  I'm really not sure where the
> best place would be.  What do you think?

/usr/tests/libexec seems logical..

>>> Perhaps even for the fourth, though
>>> it would be tricky.  But there's no way to reverse the effects for the
>>> third.  The ZFS tests, for example, create and destroy zpools.  How
>>> would you reverse that?  You can't.  Whatever was previously on the
>>> disk is gone.
>> Ah, I guess I missed that detail.
>> Couldn't those tests run on top of vnd devices though?
> FreeBSD doesn't have vnd(4), though it has md(4), which is similar.
> md(4) devices could be used for some ZFS tests.  as it happens, ZFS
> can also use file-backed vdevs.  But these workarounds don't work in
> all cases.  Sometimes you need real disks:
> 1) Using file-backed vdevs doesn't exercise vdev_geom.c, where I've
> one a lot of work.
> 2) Neither file-backed vdevs nor md(4) devices have physical path
> information, which is needed to test some hotspare functionality.
> Only da(4) devices that are attached to ses(4) expanders have that.
> 3) There is no way to remove an active file-backed vdev, and I don't
> think that you can destroy an in-use md(4) device either.  Therefore,
> to test how ZFS handles drive removals requires real drives.
> 4) gibbs has made many tweaks to ZFS to better support 4K and 8K
> sector drives.  There is no way to emulate those with file-backed
> vdevs.  Perhaps it could be done with md(4) and gnop(8), but I've
> never tried.
> 5) pjd and Steve Hartland have been working on TRIM support.  That's
> not supported by file-backed vdevs, and I doubt that md(4) supports it
> either.
> 6) Copy-on-write on top of copy-on-write is very slow.  If your system
> uses ZFS root, then both file-backed vdevs and file-backed md(4)
> devices are doing COW-on-COW.  Using physical disks for the tests
> greatly speeds the tests' runtimes.
> Ideally, the ZFS tests would use test_suites.FreeBSD.disks if it is
> defined, and use file-backed vdevs otherwise.  Tests that absolutely
> require physical disks would be skipped if test_suites.FreeBSD.disks
> isn't defined.  It would take some work, but it's doable.
>> [...]
>>>> And lastly, we'd just need a simple "filtering" feature in the kyua
>>>> cli to allow specifying which size of tests to run (or to filter by
>>>> any other metadata property, for that matter).
>>> I don't like this idea.
>> I hope you are objecting to the filtering by test sizes, not the
>> filtering itself!  See below.
>>> We tried it at work, and it didn't work out
>>> very well.  Basically, I automatically assigned sizes (short, medium,
>>> long) to all of our tests based on their runtimes, so users could
>>> select to only run the short or medium tests on the bench.  The
>>> problem is that the classification didn't make sense.  For an
>>> expedited test run, you don't want the shortest tests; what you want
>>> are the tests with the most value per unit time.  Unit tests have a
>>> high value and a very short runtime.  Stress tests have a very long
>>> runtime and arguably low value since they don't always have consistent
>>> results.
>>> At my previous job there was a large department whose duties included
>>> curating test suites and deciding which tests would be included in the
>>> short runs.  FreeBSD isn't going to have that, but we could still ask
>>> test authors to classify tests along the lines that I suggested.
>> That's a good point. If I understand you correctly, what is important
>> is to manually curate the set of "smoke tests" that run very quickly
>> and that provide a reasonably high assurance that major subsystems
>> haven't broken.
>> We can do this too pretty easily, but configuration variables are the
>> wrong mechanism. We can already define this information in metadata
>> properties as long as you prefix them with X-. So all we'd need is a
>> way to tell Kyua to only run tests marked with such property (i.e.
>> with the filtering feature described above).
> I was assuming that a test case could put in its header
> 'require.config "stress_tests"', or something like that.  Then runtime
> filtering would work by which variables the user defines.  That way,
> no new work would be required in Kyua.  But it sounds like you're
> talking about something different.  IIUC, you're suggesting that a
> test case would say in its header 'atf_set "X-runtime" "stress_test"'
> and then some TBD filtering mechanism in Kyua selects on it.  That
> would certainly be more intuitive.  Is that what you had in mind?
> -Alan
> _______________________________________________
> freebsd-testing at mailing list
> To unsubscribe, send any mail to "freebsd-testing-unsubscribe at"

More information about the freebsd-testing mailing list