198752Sjmallett# $FreeBSD$
298752Sjmallett
398755Sjmallettdnl A library of routines for doing regression tests for userland utilities.
498755Sjmallett
598755Sjmallettdnl Start up.  We initialise the exit status to 0 (no failure) and change
698755Sjmallettdnl into the directory specified by our first argument, which is the
798755Sjmallettdnl directory to run the tests inside.
898752Sjmallettdefine(`REGRESSION_START',
998752SjmallettTESTDIR=$1
1098752Sjmallettif [ -z "$TESTDIR" ]; then
1198752Sjmallett  TESTDIR=.
1298752Sjmallettfi
1398752Sjmallettcd $TESTDIR
1498752Sjmallett
1598752SjmallettSTATUS=0)
1698752Sjmallett
1798812Sjmallettdnl Check $? to see if we passed or failed.  The first parameter is the test
1898812Sjmallettdnl which passed or failed.  It may be nil.
1998812Sjmallettdefine(`REGRESSION_PASSFAIL',
2098812Sjmallettif [ $? -eq 0 ]; then
21137587Snik  echo "ok - $1 # Test detected no regression. (in $TESTDIR)"
2298812Sjmallettelse
2398812Sjmallett  STATUS=$?
24137587Snik  echo "not ok - $1 # Test failed: regression detected.  See above. (in $TESTDIR)"
2598812Sjmallettfi)
2698812Sjmallett
2798755Sjmallettdnl An actual test.  The first parameter is the test name.  The second is the
2898755Sjmallettdnl command/commands to execute for the actual test.  Their exit status is
2998755Sjmallettdnl checked.  It is assumed that the test will output to stdout, and that the
3098755Sjmallettdnl output to be used to check for regression will be in regress.TESTNAME.out.
3198752Sjmallettdefine(`REGRESSION_TEST',
32263227Sjmmv$2 | diff -u ${SRCDIR:-.}/regress.$1.out -
3398812SjmallettREGRESSION_PASSFAIL($1))
3498752Sjmallett
3598758Sjmallettdnl A freeform regression test.  Only exit status is checked.
3698758Sjmallettdefine(`REGRESSION_TEST_FREEFORM',
3798758Sjmallett$2
3898812SjmallettREGRESSION_PASSFAIL($1))
3998758Sjmallett
4098758Sjmallettdnl A regression test like REGRESSION_TEST, except only regress.out is used
4198758Sjmallettdnl for checking output differences.  The first argument is the command, the
4298758Sjmallettdnl second argument (which may be empty) is the test name.
4398758Sjmallettdefine(`REGRESSION_TEST_ONE',
44263227Sjmmv$1 | diff -u ${SRCDIR:-.}/regress.out -
4598812SjmallettREGRESSION_PASSFAIL($2))
4698758Sjmallett
4798758Sjmallettdnl A fatal error.  This will exit with the given status (first argument) and
4898758Sjmallettdnl print the message (second argument) prefixed with the string "FATAL :" to
4998758Sjmallettdnl the error stream.
5098758Sjmallettdefine(`REGRESSION_FATAL',
51137587Snikecho "Bail out! $2 (in $TESTDIR)" > /dev/stderr
5298758Sjmallettexit $1)
5398758Sjmallett
5498755Sjmallettdnl Cleanup.  Exit with the status code of the last failure.  Should probably
5598755Sjmallettdnl be the number of failed tests, but hey presto, this is what it does.  This
5698755Sjmallettdnl could also clean up potential droppings, if some forms of regression tests
5798755Sjmallettdnl end up using mktemp(1) or such.
5898752Sjmallettdefine(`REGRESSION_END',
5998752Sjmallettexit $STATUS)
60