Structure of ATF testcases (was "Test scenario for sysctl kern.maxfiles")

[Julio Merino]

On Thu, Mar 6, 2014 at 1:28 PM, Garrett Cooper <yaneurabeya at gmail.com> wrote:
>
> Seems like another standalone tool should be developed to cleanup
> variables after exit per test and per test suite.

Which variables?

I can see some benefit in providing helper functions to store and load
arbitrary data structures to/from disk with one-liners, but is that
what you mean?

> Setup/cleanup in ATF doesn't seem to follow the model that many other
> testing frameworks do when it comes to executing testcases and
> managing setup/cleanup. I really wish things were like the
> JUnit/unittest model, i.e.
>
> - Test suite setup
> - For each testcase:
>     - Run the testcase setup
>     - If the setup passed, continue
>     - Run the testcase
>     - Run the testcase cleanup
> - Test suite teardown

Sure... but the systems that implement this setup assume that all the
steps above run under one process. That's fine in the vast majority of
unit tests but the model fails when tests modify system-wide state and
crash before they get to the cleanup step.

Keep in mind that one of the first design decisions for ATF was to
support test cases like the above where system-wide state may be
modified and you need to revert the changes.  Because test cases can
crash at any point, having an out-of-process cleanup routine was a
must.  This comes with its oddities though.

All said, I'd also like a more JUnit-like model because it's easier to
understand and satisfies 99% of the cases (assuming you have rump).