MFC r295643:

r295643 (by bdrewery):

Test directories can build in parallel fine.

MFC r285119,r292502,r295380:

r285119 (by jmmv):

Add support for TEST_METADATA

Allow Makefiles to define generic metadata settings that apply to all test
programs defined by a Makefile. The generic TEST_METADATA variable extends
the per-test program settings already supported via TEST_METADATA.<program>.

This feature will be useful to easily apply some settings to all programs
in a directory. In particular, Kyua 0.12 will support parallel execution
of test programs and a bunch of them will need to be tagged as is_exclusive
to indicate that they cannot be run in parallel with anything else due to
their side-effects. It will be reasonable to set this setting on whole
directories.

r292502:

Always expose LOCALBASE, not just when CROSS_TOOLCHAIN is defined

Instead of using which(1) to look for doxygen, look for it in <LOCALBASE>/bin .
$PATH gets mangled by make buildenv, etc so it's better to just be explicit
about the path if someone uses that for instance.

r295380:

Simplify running the FreeBSD test suite

Replace `make regress` (legacy test make target) and `make test` (incomplete
test make target added with the FreeBSD test suite) with make check as it's
consistent with other open source projects.

`make check` defaults to running tests from `.OBJDIR`, but can be overridden
with the `CHECKDIR` variable.

Add `make checkworld` target to simplify running the FreeBSD test suite from
`TESTSBASE` (i.e. the top-level tests directory), similar to buildworld.

Document `make check` and `make checkworld` in build(7).

Other minor changes:

- Rename intermediate file (`Kyuafile.auto`) to `Kyuafile` to simplify
`make check`.
- Remove terse warnings attached to `beforetest`/`aftertest`.
- Add kyua binary check to check target in suite.test.mk; error out if it's
not found

The MFC is [partly] contingent on other build related changes being MFCed.

X-MFC to: stable/10
Relnotes: yes

MFC r296554:

Remove things set already by bsd.progs.mk.

MFC r296122:

(partial) Move PROGS logic to proper place and remove redundant and unneeded
logic.

MFC r292507,r292508:

r292507:

- Use LOCALBASE instead of KYUA_PREFIX for the --prefix to kyua(1)
- Use LOCALBASE instead of hardcoding /usr/local for perl

Differential Revision: https://reviews.freebsd.org/D4406 (part of a larger diff)
Reviewed by: emaste, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division

r292508:

Document LOCALBASE in the bsd.test.mk section

Differential Revision: https://reviews.freebsd.org/D4406 (part of a larger diff)
Reviewed by: emaste, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division

MFC r284408,r289151,r289158:

r284408:

Ensure TESTSDIR is defined before bsd.test.mk is .include'd

r289151:

Simplify netbsd-tests.test.mk

- projects/bmake and subsequent commits provide SRCTOP; there's no need to
manually specify it now.
- Compute a sane default for OBJTOP based on .OBJDIR and RELDIR. Manually
specifying this is probably no longer needed, but it persists just in case
(supporting commits will need to be made to move it out of some of the meta
.mk files).
- Compute a sane default for TESTSRC. Error out if the path cannot be found.

Sponsored by: EMC / Isilon Storage Division

r289158:

Default TESTSDIR to /usr/tests/${RELDIR:H}

When run from bin/ls/tests, for example, the value of TESTSDIR would be
${TESTSBASE}/${RELDIR:H} -> /usr/tests/bin/ls/tests/.. ->
/usr/tests/bin/ls

Document the new behavior in bsd.README.

While here, also document TESTSBASE

Relnotes: yes
Differential Revision: D1022
Sponsored by: EMC / Isilon Storage Division

Revert r289043.

r284408 was marked for MFC but is not safe for stable/10 yet due to failing
in bin/sh/tests.

MFC r288158:

Fix most cases of bsd.progs.mk running duplicate or missing commands.

Direct commit to fix usage with fmake.

fmake does not have :tW, so use some clever :Q tricks to achieve the
same result. This won't work if PATH actually contains spaces, but
it's better than not working at all.

MFC r284408:

Ensure TESTSDIR is defined before bsd.test.mk is .include'd

MFC r264400,r265836:

r264400:

NO_MAN= has been deprecated in favor of MAN= for some time, go ahead
and finish the job. ncurses is now the only Makefile in the tree that
uses it since it wasn't a simple mechanical change, and will be
addressed in a future commit.

r265836:

Remove last two NO_MAN= in the tree. In both of these cases, MAN= is
what is needed.

MFC r267276:

Ensure files are created during the build when using bsd.subdir.mk.

When FILES is defined in a Makefile that _also_ includes bsd.subdir.mk, the
build of the files (if any) was not properly triggered during the build
stage. This was because bsd.files.mk did not define the buildfiles target
if it was already defined... and bsd.subdir.mk defined this target on its
own, thus causing a conflict.

Fix this by unconditionally defining buildfiles from bsd.files.mk; this is
safe because nothing else in the tree needs to redefine this and because the
target itself contains no commands: all it does is define dependencies.
Also ensure that bsd.files.mk is always pulled in by bsd.test.mk regardless
of what bsd.prog.mk does.

These fixes allow "make installworld" to run cleanly on a system with
read-only src and obj trees.

This is "make tinderbox" clean.

Reviewed by: imp
Obtained from: jilles

MFC r273803,r273810:

r273803:

Filter out TESTS_SUBDIRS already added to SUBDIR instead of blindly
appending the TESTS_SUBDIRS variable to SUBDIR

Duplicate directory entries can cause unexpected side effects, like
installing the same files multiple times. This can be easily
reproduced via the following testcase prior to this commit:

SUBDIR= dir
TESTS_SUBDIRS+= dir

.include <bsd.test.mk>

Sponsored by: EMC / Isilon Storage Division

r273810:

Fix the logic inversion in the previous commit by ensuring that the matched
expression (:M) is empty, not the not matched (:N) is empty. The former case
means we have not found the TEST_SUBDIR value in SUBDIR

Reported by: rodrigc
Pointyhat to: me (did not use a clean install root)
Sponsored by: EMC / Isilon Storage Division

MFC r266473,267331,267511:

Use an intermediate target to associate with _SUBDIR which is marked .MAKE
this allows make -n to do tree walks as expected without
doing anything else (as intended).
Use prefix _sub. to help avoid conflict with any real target.

Put the test suite in its own tests.txz distribution file.
Force all the contents of /usr/tests to go into a separate distribution
file so that users of binary releases can easily choose to not install

Create a mechanism for providing fine-grained build order dependencies
during SUBDIR_PARALLEL builds. This augments the coarse .WAIT mechanism,
which is still useful if you've got a situation such as "almost everything
depends on A and B".

MFC refactoring of the *.test.mk files.

- r263161 Make bsd.test.mk the only public mk fragment for the building of tests.
- r263172 Move FreeBSD Test Suite-specific code to a suite.test.mk file.
- r263204 Add some documentation for bsd.test.mk.
- r263217 Document support for TAP-compliant Perl test programs.

This is "make tinderbox" clean.

MFC a couple of improvements to the test suite mk files.

- r260632 Support defining test program metadata from the Makefiles.
- r260633 Support perl-based TAP-compliant test programs.

MFC various fixes for the ATF tests.

- r260505 Allow tests to provide a Kyuafile when they relied on auto-generation.
- r260525 Respect the original layout of the atf-{c,c++} tests.
- r260526 Fix path to the process_helpers for the libatf-c++ tests.
- r260576 Generate and install pkg-config files for atf.
- r260577 Add atf pkg-config files from the vendor branch.
- r260584 Prevent misc_helpers from running as a test.

Split and extend bsd.test.mk into {atf,plain,tap}.test.mk.

This is a MFC of:

- r256761 Clearly split the logic to build ATF and plain tests apart.
- r256762 Add the automatic generation of Atffile files.
- r256763 Add the automatic generation of Kyuafile files.
- r256764 Plug atf-run into the 'test' target.
- r256765 Plug kyua into the 'test' target.
- r257096 Move the TESTSBASE definition to bsd.own.mk.
- r257099 Add missing plain.test.mk.
- r258297 Remove registration of C++ test programs into PROGS.
- r258298 Fix the build of plain test programs.
- r258551 Install plain.test.mk.
- r259208 Add tap.test.mk.

Approved by: rpaulo (mentor)

MFC r264400,r265836:

r264400:

NO_MAN= has been deprecated in favor of MAN= for some time, go ahead
and finish the job. ncurses is now the only Makefile in the tree that
uses it since it wasn't a simple mechanical change, and will be
addressed in a future commit.

r265836:

Remove last two NO_MAN= in the tree. In both of these cases, MAN= is
what is needed.

MFC r267276:

Ensure files are created during the build when using bsd.subdir.mk.

When FILES is defined in a Makefile that _also_ includes bsd.subdir.mk, the
build of the files (if any) was not properly triggered during the build
stage. This was because bsd.files.mk did not define the buildfiles target
if it was already defined... and bsd.subdir.mk defined this target on its
own, thus causing a conflict.

Fix this by unconditionally defining buildfiles from bsd.files.mk; this is
safe because nothing else in the tree needs to redefine this and because the
target itself contains no commands: all it does is define dependencies.
Also ensure that bsd.files.mk is always pulled in by bsd.test.mk regardless
of what bsd.prog.mk does.

These fixes allow "make installworld" to run cleanly on a system with
read-only src and obj trees.

This is "make tinderbox" clean.

Reviewed by: imp
Obtained from: jilles

MFC r273803,r273810:

r273803:

Filter out TESTS_SUBDIRS already added to SUBDIR instead of blindly
appending the TESTS_SUBDIRS variable to SUBDIR

Duplicate directory entries can cause unexpected side effects, like
installing the same files multiple times. This can be easily
reproduced via the following testcase prior to this commit:

SUBDIR= dir
TESTS_SUBDIRS+= dir

.include <bsd.test.mk>

Sponsored by: EMC / Isilon Storage Division

r273810:

Fix the logic inversion in the previous commit by ensuring that the matched
expression (:M) is empty, not the not matched (:N) is empty. The former case
means we have not found the TEST_SUBDIR value in SUBDIR

Reported by: rodrigc
Pointyhat to: me (did not use a clean install root)
Sponsored by: EMC / Isilon Storage Division

MFC r266473,267331,267511:

Use an intermediate target to associate with _SUBDIR which is marked .MAKE
this allows make -n to do tree walks as expected without
doing anything else (as intended).
Use prefix _sub. to help avoid conflict with any real target.

Put the test suite in its own tests.txz distribution file.
Force all the contents of /usr/tests to go into a separate distribution
file so that users of binary releases can easily choose to not install

Create a mechanism for providing fine-grained build order dependencies
during SUBDIR_PARALLEL builds. This augments the coarse .WAIT mechanism,
which is still useful if you've got a situation such as "almost everything
depends on A and B".

MFC refactoring of the *.test.mk files.

- r263161 Make bsd.test.mk the only public mk fragment for the building of tests.
- r263172 Move FreeBSD Test Suite-specific code to a suite.test.mk file.
- r263204 Add some documentation for bsd.test.mk.
- r263217 Document support for TAP-compliant Perl test programs.

This is "make tinderbox" clean.

MFC a couple of improvements to the test suite mk files.

- r260632 Support defining test program metadata from the Makefiles.
- r260633 Support perl-based TAP-compliant test programs.

MFC various fixes for the ATF tests.

- r260505 Allow tests to provide a Kyuafile when they relied on auto-generation.
- r260525 Respect the original layout of the atf-{c,c++} tests.
- r260526 Fix path to the process_helpers for the libatf-c++ tests.
- r260576 Generate and install pkg-config files for atf.
- r260577 Add atf pkg-config files from the vendor branch.
- r260584 Prevent misc_helpers from running as a test.

Split and extend bsd.test.mk into {atf,plain,tap}.test.mk.

This is a MFC of:

- r256761 Clearly split the logic to build ATF and plain tests apart.
- r256762 Add the automatic generation of Atffile files.
- r256763 Add the automatic generation of Kyuafile files.
- r256764 Plug atf-run into the 'test' target.
- r256765 Plug kyua into the 'test' target.
- r257096 Move the TESTSBASE definition to bsd.own.mk.
- r257099 Add missing plain.test.mk.
- r258297 Remove registration of C++ test programs into PROGS.
- r258298 Fix the build of plain test programs.
- r258551 Install plain.test.mk.
- r259208 Add tap.test.mk.

Approved by: rpaulo (mentor)