ATF Test Cases

[Alan Somers]

On Wed, Mar 26, 2014 at 7:47 PM, Julio Merino <jmmv at freebsd.org> wrote:
> On Thu, Mar 27, 2014 at 7:26 AM, Kilner, Peter <Peter.Kilner at emc.com> wrote:
>> Hello,
>
> Hi Peter,
>
> Alan already answered this pretty well, but let me throw in my 2 cents:
>
>> I hope I am posting this question to the correct list.
>>
>> Is there a way to have a single ATF test case report multiple pass/fail results?  Currently I have seen that one can include a number of test conditions in a single test case (for example many aft_checks).  However the test case will only report one pass/fail result for that case.
>
> That's by design: one test case should represent one (and only one)
> scenario. Therefore, there can only be one meaningful result.
>
>> I would like to build a test case that will iterate though many configurations and would like to see a pass/fail for each config.  However this is difficult to implement with multiply test cases because of the for loops that I am using.
>
> It sounds like you actually should split your test case into smaller
> ones and name each test case accordingly.  Your email doesn't provide
> a lot of details, but I guess that if you are testing more than one
> configuration file is because each configuration file is exposing a
> specific behavior of the program under test.
>
> Consider this: "test_configuration" sounds like a really broad test
> case name and provides no information about what is wrong when the
> test fails. But if you consider
> "test_configuration__missing_hostname",
> "test_configuration__invalid_syntax",
> "test_configuration__duplicate_host_stanzas" as separate test cases,
> then when one fails it becomes obvious what specific case is broken.
>
> Doing this needn't be hard.
>
> If you are using atf-sh, my suggestion is to put all of the test case
> body in a separate function.  Then, with a loop, you dynamically
> define small test case bodies (using eval) that call the auxiliary
> function with the right parameters.

I never thought about using eval to dynamically define testcases.  Can
you please give an example?

>
> If you are using atf-c, you can do something similar, maybe even abusing macros.
>
> But basically: just move the code to an auxiliary function and have
> all the individual test cases call it as a one-liner with the right
> arguments.
>
> See the following for some more ideas:
>
>    https://wiki.freebsd.org/TestSuite/Structure#Test_programs_vs._test_cases
>
> Please let me know if anything is not clear or lacks detail.
> _______________________________________________
> freebsd-testing at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-testing
> To unsubscribe, send any mail to "freebsd-testing-unsubscribe at freebsd.org"