ATF Test Cases

[Julio Merino]

On Thu, Mar 27, 2014 at 7:26 AM, Kilner, Peter <Peter.Kilner at emc.com> wrote:
> Hello,

Hi Peter,

Alan already answered this pretty well, but let me throw in my 2 cents:

> I hope I am posting this question to the correct list.
>
> Is there a way to have a single ATF test case report multiple pass/fail results?  Currently I have seen that one can include a number of test conditions in a single test case (for example many aft_checks).  However the test case will only report one pass/fail result for that case.

That's by design: one test case should represent one (and only one)
scenario. Therefore, there can only be one meaningful result.

> I would like to build a test case that will iterate though many configurations and would like to see a pass/fail for each config.  However this is difficult to implement with multiply test cases because of the for loops that I am using.

It sounds like you actually should split your test case into smaller
ones and name each test case accordingly.  Your email doesn't provide
a lot of details, but I guess that if you are testing more than one
configuration file is because each configuration file is exposing a
specific behavior of the program under test.

Consider this: "test_configuration" sounds like a really broad test
case name and provides no information about what is wrong when the
test fails. But if you consider
"test_configuration__missing_hostname",
"test_configuration__invalid_syntax",
"test_configuration__duplicate_host_stanzas" as separate test cases,
then when one fails it becomes obvious what specific case is broken.

Doing this needn't be hard.

If you are using atf-sh, my suggestion is to put all of the test case
body in a separate function.  Then, with a loop, you dynamically
define small test case bodies (using eval) that call the auxiliary
function with the right parameters.

If you are using atf-c, you can do something similar, maybe even abusing macros.

But basically: just move the code to an auxiliary function and have
all the individual test cases call it as a one-liner with the right
arguments.

See the following for some more ideas:

   https://wiki.freebsd.org/TestSuite/Structure#Test_programs_vs._test_cases

Please let me know if anything is not clear or lacks detail.