ATF Test Cases

Alan Somers asomers at
Thu Mar 27 04:00:30 UTC 2014

On Wed, Mar 26, 2014 at 7:47 PM, Julio Merino <jmmv at> wrote:
> On Thu, Mar 27, 2014 at 7:26 AM, Kilner, Peter <Peter.Kilner at> wrote:
>> Hello,
> Hi Peter,
> Alan already answered this pretty well, but let me throw in my 2 cents:
>> I hope I am posting this question to the correct list.
>> Is there a way to have a single ATF test case report multiple pass/fail results?  Currently I have seen that one can include a number of test conditions in a single test case (for example many aft_checks).  However the test case will only report one pass/fail result for that case.
> That's by design: one test case should represent one (and only one)
> scenario. Therefore, there can only be one meaningful result.
>> I would like to build a test case that will iterate though many configurations and would like to see a pass/fail for each config.  However this is difficult to implement with multiply test cases because of the for loops that I am using.
> It sounds like you actually should split your test case into smaller
> ones and name each test case accordingly.  Your email doesn't provide
> a lot of details, but I guess that if you are testing more than one
> configuration file is because each configuration file is exposing a
> specific behavior of the program under test.
> Consider this: "test_configuration" sounds like a really broad test
> case name and provides no information about what is wrong when the
> test fails. But if you consider
> "test_configuration__missing_hostname",
> "test_configuration__invalid_syntax",
> "test_configuration__duplicate_host_stanzas" as separate test cases,
> then when one fails it becomes obvious what specific case is broken.
> Doing this needn't be hard.
> If you are using atf-sh, my suggestion is to put all of the test case
> body in a separate function.  Then, with a loop, you dynamically
> define small test case bodies (using eval) that call the auxiliary
> function with the right parameters.

I never thought about using eval to dynamically define testcases.  Can
you please give an example?

> If you are using atf-c, you can do something similar, maybe even abusing macros.
> But basically: just move the code to an auxiliary function and have
> all the individual test cases call it as a one-liner with the right
> arguments.
> See the following for some more ideas:
> Please let me know if anything is not clear or lacks detail.
> _______________________________________________
> freebsd-testing at mailing list
> To unsubscribe, send any mail to "freebsd-testing-unsubscribe at"

More information about the freebsd-testing mailing list