[GSOC17] Smoke testing of all base utilities - Week 7

[Shivansh Rai]

Hello all,

My apologies for a late progress report (~3 days), I was involved in
shifting my belongings to a new residence as final year is about to start.
A brief summary of the updates made to the tooling in the past week is
available here [1].

The tool can now generate annotations based on failed testcases during a
test runtime. The annotation file contains the name of testcases which we
don’t want to be added in the tests generated in future runs. An example
for these kind of testcases are the ones produced previously in date(1) -
since most testcases were time/timezone dependent, one would prefer not to
include them in the generated test script. Hence, the annotation file for
date [2] contains all those testcase names. The newly generated test for
date(1) only has the successful testcases [3].

@brooks suggested that maintaining annotations in a separate file has an
advantage of not disrupting the existing test scripts.

For each generated test script in “generated_tests/*”, we maintain a
corresponding annotation file in “annotations/“ (in case there are failing
testcases). For e.g. the annotation file for “generated_tests/date_test.sh”
is “annotations/date_test.annot”.

A brief pseudocode for generating annotations is as follows -

for test_script in generated_tests/*
  run $test_script via `kyua test`
  # The above step is yet to be implemented as a
  # few things have to be kept in mind such as:
  #   - placing the test_script in a correct destination
  #   - if the makefile exists, updating it; if not, then generating it
  #   - appropriately updating "etc/mtree/BSD.tests.dist"

  extract failing testcases via `kyua report`
  generate annotations for $test_script based on failed testcases
  if annotation file exists
    append newly found annotations
  end if

I was unable to find a good interface for extracting exit statuses of
individual testcases from a test run (need to search more), hence extracted
the same from the output of kyua report. The current method will be updated
if I come across a cleaner approach.

Refactoring of the codebase is in progress alongside as I read C++11
guidelines and good practices.

On a different note, my PR for adding the atf_check_not_equal function to
atf-sh was merged [4]. Once the new version of atf(7) is released,
corresponding updates will be made which led to the creation of this PR [5].

The next task is to populate the annotation files pertaining to different
architectures and test environments (will be automated). If a particular
testcase fails in any case, we add a corresponding annotation as the
generated tests should be architecture/environment independent.
Also, the first step in the above pseudocode will be implemented.

[1]: https://github.com/shivrai/smoketestsuite/compare/2bf499...12d88b
[2]:
https://github.com/shivrai/smoketestsuite/blob/master/tool/annotations/date_test.annot
[3]:
https://github.com/shivrai/smoketestsuite/blob/master/tool/generated_tests/date_test.sh
[4]:
https://github.com/jmmv/atf/commit/0824235a3bcb45f16e1dfae7dce347e2d8b10d26
[5]: https://reviews.freebsd.org/D11084?id=29301#inline-65206

Thank you.
With best regards,
Shivansh Rai
​