Structure of ATF testcases (was "Test scenario for sysctl kern.maxfiles")

[Garrett Cooper]

On Thu, Mar 6, 2014 at 11:54 AM, Julio Merino <jmmv at freebsd.org> wrote:
> On Thu, Mar 6, 2014 at 1:28 PM, Garrett Cooper <yaneurabeya at gmail.com> wrote:
>>
>> Seems like another standalone tool should be developed to cleanup
>> variables after exit per test and per test suite.
>
> Which variables?

sysctls in this case. Isilon would probably love this and other folks
would too no doubt.

> I can see some benefit in providing helper functions to store and load
> arbitrary data structures to/from disk with one-liners, but is that
> what you mean?

Sorta kinda.

>> Setup/cleanup in ATF doesn't seem to follow the model that many other
>> testing frameworks do when it comes to executing testcases and
>> managing setup/cleanup. I really wish things were like the
>> JUnit/unittest model, i.e.
>>
>> - Test suite setup
>> - For each testcase:
>>     - Run the testcase setup
>>     - If the setup passed, continue
>>     - Run the testcase
>>     - Run the testcase cleanup
>> - Test suite teardown
>
> Sure... but the systems that implement this setup assume that all the
> steps above run under one process. That's fine in the vast majority of
> unit tests but the model fails when tests modify system-wide state and
> crash before they get to the cleanup step.

Agreed. Hence the need for tools that restore system state and careful
code reviews and code design.

> Keep in mind that one of the first design decisions for ATF was to
> support test cases like the above where system-wide state may be
> modified and you need to revert the changes.  Because test cases can
> crash at any point, having an out-of-process cleanup routine was a
> must.  This comes with its oddities though.

Yes, but it falls on the developer to handle global state outside of
the atf test directory sandbox. ATF shouldn't worry about protecting
someone from doing something like fiddling around with sysctls and
files in /etc (for example) -- it's the test developer's job to not do
things like this.

> All said, I'd also like a more JUnit-like model because it's easier to
> understand and satisfies 99% of the cases (assuming you have rump).

Yup yup!

-Garrett