Structure of ATF testcases (was "Test scenario for sysctl kern.maxfiles")
yaneurabeya at gmail.com
Thu Mar 6 18:28:14 UTC 2014
On Thu, Mar 6, 2014 at 10:02 AM, Julio Merino <jmmv at freebsd.org> wrote:
> On Thu, Mar 6, 2014 at 12:45 PM, Alan Somers <asomers at freebsd.org> wrote:
>> I suggest the more specific name "allow_sysctl_side_effects". As I
>> continue to add tests, config variables like this will become more
>> important. In particular, I have some tests that create and destroy
>> zpools, some that can cause panics, some that load modules, etc.
>> Before I can upstream all of them, we'll need to agree on a consistent
>> set of config variables that control test case execution. I haven't
>> put much thought into it yet.
> Sure; more granularity++. I haven't put any thought on this though so
> please propose something :)
> Whatever we decide, 1) these variables need to be documented in
> tests(7), 2) a ready-to-enable template in
> ports/devel/kyua-cli/files/kyua.conf.in about them would be beneficial
> and 3) src/projects/jmmv/autotest/ should switch these on.
>> How does NetBSD handle tests that have
>> potentially harmful side effects?
> They don't AFAIK. The thing is that not that many tests have
> side-effects because the majority of kernel-level tests use rump to
> avoid this problem and be more reliable/deterministic. The few that
> have side-effects have ad-hoc cleanup routines.
Seems like another standalone tool should be developed to cleanup
variables after exit per test and per test suite.
Setup/cleanup in ATF doesn't seem to follow the model that many other
testing frameworks do when it comes to executing testcases and
managing setup/cleanup. I really wish things were like the
JUnit/unittest model, i.e.
- Test suite setup
- For each testcase:
- Run the testcase setup
- If the setup passed, continue
- Run the testcase
- Run the testcase cleanup
- Test suite teardown
More information about the freebsd-testing