[GSOC17] Smoke testing of all base utilities - Week 4

[Shivansh Rai]

Hello all,

A little (too much) late (~4 days) for the progress report of week 4, my
deepest apologies. This is the progress made in the last week, finishing
the milestones which I promised in the previous report.

A very brief gist of how the smoke testing tool works as of now -
If a "known option" is encountered while looking in the hashmap containing
pre-defined option definitions, a testcase to check the validity of the
result of that option is produced.
*NOTE:* Pre-defined option definitions try to relate option names with the
most relevant keywords which might occur in their descriptions (for e.g.
'-h' might contain "help", '-v' might contain "version" etc.)
In case no such option was encountered, testcases to verify the correct
usage message for all of the valid (accepted by the utility under test)
options are produced when they are used incorrectly (since we couldn't
guess their correct usage).
The code is fairly readable (I guess, refactoring in progress), and it
might give a better idea.

For those wondering as this is extremely trivial, this mail from @brooks
[1] will provide a very good context and the big picture. The state of the
tool now is a small step forward as to what is finally expected.

The tool [2] is ready for basic experimentation. It can be run via the
following commands from <project_root>/tool -
```
cd <project_root>/tool
make && make run
```

The expected behavior for this is that all the options accepted by ln(1)
will fail (since we cannot guess (yet) how they are supposed to be used by
just reading the man-page) and produce a usage message. An atf-sh test file
`ln_test.sh` will be produced which will contain test-cases for all these
options which basically runs them and check the validity of the generated
usage message.
Since ln(1) already has a test now, you can copy this test file to
<local_freebsd_src>/bin/ln/tests and do the regular steps to run a kyua
test -

(BEWARE! Your source tree should have some form of version controlling to
revert it back to the original state.)

```
cd <local_freebsd_src>/bin/ln/tests
make install
cd /usr/tests/bin/ln
kyua test
```

This should produce 10 successful testcases.

I also prepared a differential [4] for a test of getfacl(1) utility (I am
yet to update the requested changes, will be doing them soon).

*---(End of progress report, meta information ahead!)---*

While I was writing the basic functionality today, I had an idea as to how
to work upon generating sane option identifiers. Currently they are very
simple (like "version", "help"). But since the tool will be going through
(mostly) all the utilities' man pages in the base system, some sort of
relation can be mapped between option names and the most frequent words
which "happen to occur" in their descriptions based on the history of
previously visited option definitions. This way, our comparison will not be
limited to a simple `string.find()` for a single word while looking in a
utility's option description, and we can check for the presence of these
frequently occurring words in option descriptions and "try to guess" an
appropriate usage for that specific option.
A priority queue (heap?) can be a way to go, but a lot of thinking has to
be put in for several other aspects. It all seems a bit far-fetched at the
moment when the thought of a sane running time complexity surfaces, but
hopefully some way can be figured out. Ideas and suggestions are most
welcome!
The above will be worked out before sending the next report, and also the
description of how the tool works [5] will be updated shortly.

Updates to the tool since last report:
https://github.com/shivrai/smoketestsuite/compare/c2b5c7...4969ae
[1]: https://shivrai.github.io/assets/tmp/smoke_testing_mail.pdf
[2]: https://github.com/shivrai/smoketestsuite/tree/master/tool
[4]: https://reviews.freebsd.org/D11315
[5]: https://shivrai.github.io/assets/tmp/GSoC17Automation.pdf

PS. On a side note, I wasn't willing to merge the progress report for weeks
4 and 5 together, hence will be sending the next report for week 5 (current
week) in a few days' duration.

Thanks!
With best regards,
Shivansh Rai