Structure of ATF testcases (was "Test scenario for sysctl kern.maxfiles")

[Garrett Cooper]

On Thu, Mar 6, 2014 at 10:02 AM, Julio Merino <jmmv at freebsd.org> wrote:
> On Thu, Mar 6, 2014 at 12:45 PM, Alan Somers <asomers at freebsd.org> wrote:
>> I suggest the more specific name "allow_sysctl_side_effects".  As I
>> continue to add tests, config variables like this will become more
>> important.  In particular, I have some tests that create and destroy
>> zpools, some that can cause panics, some that load modules, etc.
>> Before I can upstream all of them, we'll need to agree on a consistent
>> set of config variables that control test case execution.  I haven't
>> put much thought into it yet.
>
> Sure; more granularity++.  I haven't put any thought on this though so
> please propose something :)
>
> Whatever we decide, 1) these variables need to be documented in
> tests(7), 2) a ready-to-enable template in
> ports/devel/kyua-cli/files/kyua.conf.in about them would be beneficial
> and 3) src/projects/jmmv/autotest/ should switch these on.
>
>>  How does NetBSD handle tests that have
>> potentially harmful side effects?
>
> They don't AFAIK. The thing is that not that many tests have
> side-effects because the majority of kernel-level tests use rump to
> avoid this problem and be more reliable/deterministic. The few that
> have side-effects have ad-hoc cleanup routines.

Seems like another standalone tool should be developed to cleanup
variables after exit per test and per test suite.

Setup/cleanup in ATF doesn't seem to follow the model that many other
testing frameworks do when it comes to executing testcases and
managing setup/cleanup. I really wish things were like the
JUnit/unittest model, i.e.

- Test suite setup
- For each testcase:
    - Run the testcase setup
    - If the setup passed, continue
    - Run the testcase
    - Run the testcase cleanup
- Test suite teardown

Thanks :)!
-Garrett