Simple framework for regression tests in src/tools/regression

[Alfred Perlstein]

* John Baldwin <jhb at freebsd.org> [080117 14:28] wrote:
> When I converted the test program for the posix shm stuff over to a regression 
> test I added some framework bits to manage the output format prove(1) 
> expects.  You can find it in src/tools/regression/posixshm/test.[ch].  I was 
> curious if we wanted to perhaps make it a library in the base system 
> (libbsdtest.a or some such) that our regression tools could use.
> 
> The basic organization of the current test.[ch] is that you have a linker set 
> of tests which it uses to number the tests and determine the test count.  The 
> framework provides routines for the test to indicate its result like so:
> 
> pass()			- indicates test passed
> fail()			- indicates test failed with no error message
> fail_err(fmt, ...)	- failure with printf(3) error message
> fail_errno(fmt, ...)	- failure with error message followed by strerror(3)
> 			  output similar to err(3)
> skip(const char *)	- test skipped with reason for skip
> todo(const char *)	- test marked as TODO
> 
> If a given test doesn't call any of the status indicators above, then the main 
> test loop will fail the test with an error of "unknown status".  No effort is 
> made to handle multiple status indications for a given test.  I assume 
> prove(1) would handle complaining about such anomalies in the test output.
> 
> Tests consist of a void foo(void) function along with a description and each 
> test is created via the TEST() macro like so:
> 
> void
> mytest(void)
> {
> 	struct passwd *pwd;
> 
> 	pwd = getpwuid(getuid());
> 	if (strcmp(pwd->pw_name, "jhb") == 0) {
> 		fail_err("great evil detected");
> 		return;
> 	}
> 	pass();
> }
> TEST(mytest, "check for evil users");
> 
> To invoke the tests, one executes the 'run_tests()' method provided by the 
> framework.  It outputs the header expected by prove(1) and then executes each 
> test in succession.  I specifically didn't put main() in the framework in 
> case a test utility wants to take optional arguments, etc.  A simple test 
> utility (like posixshm.c) just invokes run_tests() in its main().
> 
> One design decision that I'm sure will be debated is that I went with the 
> functions to indicate results (fail, etc.) rather than having the test method 
> return a value because it becomes more complicated to handle error messages 
> otherwise.  If this were C++ the tests could return a std::pair<> of a status 
> code and some sort of object with a toString() method or something (or if it 
> was python I would return a tuple).  However, this is C. :)  It's also nicer 
> to be able to just pass the va args to vprintf() directly w/o having to print 
> them into strings and print them out later, etc.
> 
> Anyways, comments, etc.?

There is C++ in the base system.

You can return a pair by returning a structure which is supported in
C.

-Alfred