Simple framework for regression tests in src/tools/regression

John Baldwin jhb at
Thu Jan 17 14:31:21 PST 2008

When I converted the test program for the posix shm stuff over to a regression 
test I added some framework bits to manage the output format prove(1) 
expects.  You can find it in src/tools/regression/posixshm/test.[ch].  I was 
curious if we wanted to perhaps make it a library in the base system 
(libbsdtest.a or some such) that our regression tools could use.

The basic organization of the current test.[ch] is that you have a linker set 
of tests which it uses to number the tests and determine the test count.  The 
framework provides routines for the test to indicate its result like so:

pass()			- indicates test passed
fail()			- indicates test failed with no error message
fail_err(fmt, ...)	- failure with printf(3) error message
fail_errno(fmt, ...)	- failure with error message followed by strerror(3)
			  output similar to err(3)
skip(const char *)	- test skipped with reason for skip
todo(const char *)	- test marked as TODO

If a given test doesn't call any of the status indicators above, then the main 
test loop will fail the test with an error of "unknown status".  No effort is 
made to handle multiple status indications for a given test.  I assume 
prove(1) would handle complaining about such anomalies in the test output.

Tests consist of a void foo(void) function along with a description and each 
test is created via the TEST() macro like so:

	struct passwd *pwd;

	pwd = getpwuid(getuid());
	if (strcmp(pwd->pw_name, "jhb") == 0) {
		fail_err("great evil detected");
TEST(mytest, "check for evil users");

To invoke the tests, one executes the 'run_tests()' method provided by the 
framework.  It outputs the header expected by prove(1) and then executes each 
test in succession.  I specifically didn't put main() in the framework in 
case a test utility wants to take optional arguments, etc.  A simple test 
utility (like posixshm.c) just invokes run_tests() in its main().

One design decision that I'm sure will be debated is that I went with the 
functions to indicate results (fail, etc.) rather than having the test method 
return a value because it becomes more complicated to handle error messages 
otherwise.  If this were C++ the tests could return a std::pair<> of a status 
code and some sort of object with a toString() method or something (or if it 
was python I would return a tuple).  However, this is C. :)  It's also nicer 
to be able to just pass the va args to vprintf() directly w/o having to print 
them into strings and print them out later, etc.

Anyways, comments, etc.?

John Baldwin

More information about the freebsd-arch mailing list