Testing the build infrastructure

[Nik Clayton]

Guys,

Given the comments that have been bandied about recently relating to
breakage of various parts of the build infrastructure, and functionality
that used to work no longer working, I'd like to make some concrete
progress towards fixing things.

To my mind, step 1 is describing, given a particular set of inputs
(probably arguments given to make(1)), what we expect the output to be.

This means we need to write some tests.  Automated tests.  The kinds of
tests that can get run automatically every 24 hours or so, to rapidly
pin-point any commits that have caused a regressions in build
functionality.

Thinking out loud for the moment, I think we need 

    doc/regression/{articles, books}/

Below each one of those directories we have one or more subdirectories,
one for each type of document that we're testing.  Each document has all
the usual content that we would normally provide.  DocBook markup,
images, use of marked sections, a Makefile, and so on.

In addition, we create multiple test files.  By convention, these will
have a .t extension, and follow the protocol described in
src/tools/regression/README.  Each .t file implements one or more
logically related tests.  Each test can be quite simple (there's no such
thing as a useless test) or complex as necessary.

For example, we might have a test file that makes sure that the HTML
format works.

    #!/bin/sh

    # Test: Make sure that FORMATS=html-split is understood
    test="\"html-split\" is a valid format"

    make -V ALL_FORMATS | xargs -n 1 echo | grep '^html-split$'
    if [ $? -eq 0 ]; then
    	echo "ok 1 - $test"
    else
    	echo "not ok 1 - $test"
    fi

    # Test: Does 'make FORMATS=html-split' complete, and produce 
    # sensible output
    test="make FORMATS=html-split"

    make FORMATS=html-split
    if [ $? -eq 0 ]; then
    	echo "ok 2 - $test"
    else
    	echo "not ok 2 - $test"
    fi

    test="Generated .html file exists"
    if [ -e index.html ]; then
    	echo "ok 3 - $test"
    else
    	echo "not ok 3 - $test"
    fi

    test="and it's a sensible size"
    if [ -s index.html ]; then
    	echo "ok 4 - $test"
    else
    	echo "not ok 4 - $test"
    fi

... and so on -- additional tests to verify the DOCTYPE, that all the
internal links work, that sort of thing.  There should be tests to make
sure that things like building with a r/o doc/ tree works, and so forth.

Thoughts?

N
-- 
FreeBSD: The Power to Serve      http://www.freebsd.org/               (__)
FreeBSD Documentation Project    http://www.freebsd.org/docproj/    \\\'',)
                                                                      \/  \ ^
   --- 15B8 3FFC DDB4 34B0 AA5F  94B7 93A8 0764 2C37 E375 ---         .\._/_)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-doc/attachments/20041207/940af7ef/attachment.sig>