[GSOC17] Smoke testing of all base utilities - Final Report

Shivansh Rai shivansh at freebsd.org
Wed Sep 6 17:46:40 UTC 2017


Hello all,

This is the final progress report for all the tasks that have been
accomplished for the project - Smoke testing of all base utilities.
Targets met in the timeline

   -

   The automation tool [1] can now generate tests which can check whether a
   utility is properly linked. A basic structure of testcases in a generated
   test is as follows -
   - invalid_usage : Verifies the behavior of a utility when an invalid
      usage is performed for a supported option.
      - ($option)_flag: Verifies the behavior of a utility when a valid
      usage is performed for a supported option - $option.
      - no_arguments : Verifies the behavior of a utility when it is used
      without any supported options.
   -

   The generated tests are maintained at a separate location [2] and
   revised with updated tests when the tool is updated.
   -

   The final differential [3] which lays the basic layout of the
   auto-generated tests will be soon merged. Once it is approved, similar
   batches of tests will be sent for review.

I have prepared a small diagram [4] which shows how different components of
the tool fit in with the testcase generation component. I am planning to
add more details to it.
Current and future scope for work

   -

   I am currently working on trying to resolve an interesting issue which I
   have recently noticed while generating the tests. The description of the
   issue is as follows -
   The functionality to stop generation of tests during runtime was
   successfully integrated recently [5] (after a failed previous attempt
   summarized here -
   https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221882). The
   requirement of this functionality arose out of the fact that there are
   utilities with options which prompt the user for input, and halt at this
   point until some action is taken. However, currently the tool does not
   concern itself with any form of “external” intervention while generating
   tests, so we put a timeout of 1 second for each testcase to complete (It is
   assumed that 1 second is sufficient to successfully process the groff
   script and generate the test ; a thread is spawned for individually
   handling testcase generation for each utility). It so happens that on first
   encounter of such a utility (let’s say it is ‘u1’), all the future
   instances seem to take more than 1 second to complete and are not generated
   by the tool. Interestingly, if the groff script of ‘u1’ is removed (the
   tool loads groff scripts from the directory src/groff before generating
   tests), then everything seems to work fine until another utility like ‘u1’
   is encountered.

   This is an example trace demonstrating the above scenario -

   ... (Previous log) ...
   Generating test for: ypcat(1) ...Successful
   Generating test for: printenv(1) ...Successful
   Generating test for: uniq(1) ...Failed!
   Generating test for: ktrace(1) ...Failed!
   Generating test for: revoke(1) ...Failed!

   The utility for which first failure was encountered while test
   generation is uniq(1) ;  the test generation for ktrace(1) and revoke(1)
   also fails, which should not happen. Let’s remove the groff script for
   uniq(1) from `src/groff` and try again -

   ... (Previous log) ...
   Generating test for: ypcat(1) ...Successful
   Generating test for: printenv(1) ...Successful
   Generating test for: ktrace(1) ...Successful
   Generating test for: revoke(1) ...Successful

   As it can be observed, the utilities which failed to produce tests
   earlier (ktrace(1), revoke(1)) are now successful.
   I am suspecting this behavior is due to having an improper thread
   cleanup routine for the failing instances (there is no cleanup being done
   yet apart from a call to timed_join() of boost::thread). Although I have
   not been able to reproduce this behavior on my Linux machine. Looking into
   this currently! (interestingly we are studying about pthreads currently in
   the OS course :P )
   -

   I am yet to finish the test for getfacl(1) that I started in D11315 [6]
   (my deepest apologies for being so much late in this even after a reminder
   from asomers@, am overloaded with deadlines at the moment :/ But will
   complete it by this weekend, and this deadline is final and not subjected
   to change! ).
   -

   The tool currently handles processing of groff scripts for section 1
   utilities. However, brooks@ pointed out that the way in which things are
   currently done might not work for section 8 utilities. So, the next task
   will be to expand the tool to cover all the new cases.

The wiki page [7] is fairly up to date now, and I will continue to add
updates to it.

Finally, I would like to thank my mentors for all their help and prompt
responses (and awesome code reviews!) during the entire duration, and hope
to stay in touch with them. I would also like to thank the FreeBSD
community for giving me this great opportunity to learn and hope to be an
integral part of it in times to come.

[1]: https://github.com/shivansh/smoketestsuite/
[2]: https://github.com/shivansh/smoketests/
[3]: https://reviews.freebsd.org/D12036
[4]: https://github.com/shivansh/smoketestsuite/#automation-tool
[5]:
https://github.com/shivansh/smoketestsuite/commit/267540d534fab4a3bea183518d64d0b23cee19d7
[6]: https://reviews.freebsd.org/D11315
[7]: https://wiki.freebsd.org/SummerOfCode2017/SmokeTestingOfBaseUtilities

Thank you.
With best regards,
Shivansh Rai
​


More information about the soc-status mailing list