Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

make check: suppress echo of kyua binary location

986deea5b518ee5bf6b8b1486056a21819bd8bd2 inadvertently removed this; fix it.

build: `make check`: use a PATH search instead for Kyua

which(1) accepts both relative/absolute paths as well as lone binary
names. Set KYUA to kyua and use which(1) to confirm that it can find one;
if it cannot, just advise the user to set KYUA directly to the kyua binary
rather than assuming a relative location from LOCALBASE.

This allows `make check` to be operated with the version of kyua in base
without losing the flexibility of specifying another one.

ngie@ notes that the original intention was to avoid redundant $PATH lookups
and improve the determinism of the target. A future change will likely push
us back to this state, perhaps in the form of reverting this entirely and
just switching to using kyua in base. Accepting any in $PATH should be
considered a transitional move, at least until it's declared otherwise,
since kyua was only semi-recently added to base.

Reviewed-by: brooks, emaste, lwhsu, ngie
Differential-Revision: https://reviews.freebsd.org/D28045

In suite.test.mk, test if ${DESTDIR} exists before attempting to run
chflags -R on it, otherwise the command will error out. (Note that
adding -f to the chflags invocation does not help, unlike with rm.)

MFC after: 3 days

Clean up all directories created by `make hier`

The logic I introduced in r322511 unfortunately left chflags schg'ed
directories behind created by `make hier` (in the stock /etc/mtree
files, this is limited to /var/empty).

The proposed change calls `chflags -R 0` and `rm -Rf ...` to clean all
of the directories that could not be removed by `${MAKE} clean`.
`${MAKE} clean` in bsd.obj.mk calls `cleandir`/`cleanobj`, which handles
the first directory tree walk/removal.

Approved by: emaste (mentor)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D18641

Avoid trying to remove a fullpath directory in CLEANDIRS.

Let CLEANDIRS work relative since make is already in .OBJDIR.

MFC after: 2 weeks
X-MFC-With: r322511
Sponsored by: Dell EMC Isilon

Checkpoint work to integrate coverage and make check together

Add limited sandbox capability to "make check"

== Rationale ==

r295380 introduced "make check" and consolidated means for running
test code in an attempt to simplify running tests. One could either
install files/libraries/programs and run "make check", or run "make check"
with an explicit CHECKDIR, e.g., `make check CHECKDIR=$(make -V.OBJDIR)``.

One criticism that was received is that "make check" should be run with
the intent of making dev->test->commit easier, which means that the target
audience's workflow should be developers. One developer pattern available
in other opensource projects is to run test code from a developer sandbox,
instead of installing to a system.

== Method ==

This approach is slightly different from the standard approach, in the sense
that it builds and installs into a deterministic directory under .OBJDIR (as I call it,
the "sandbox"), then runs "make check" against that. In the event the test
run is successful, the deterministic directory is removed to save space.

== Approach ==

bsd.lib.mk, bsd.prog.mk:

To support this functionality, a new variable `HAS_TESTS` is being added.

HAS_TESTS enables appropriate behavior with bsd.lib.mk and bsd.prog.mk, as
follows:
- Add "make check" as an available target from the directory.
- Pass down appropriate variables via ${TESTS_ENV}, i.e.,
${TESTS_LD_LIBRARY_PATH} and ${TESTS_PATH}.

One should add "HAS_TESTS" to directories containing tests in them, e.g. from
bin/sh/Makefile,

HAS_TESTS=
SUBDIR.${MK_TESTS}+= tests

HAS_TESTS doesn't automatically add the tests subdirectory for flexibility
reasons.

bsd.opts.mk, src.opts.mk:
- The knob ${MK_MAKE_CHECK_USE_SANDBOX} has been added, both to explicitly
direct (internally) when to set a deterministic ${DESTDIR} and to also allow
users to disable this behavior globally, i.e., via src.conf.
- MK_TESTS has been promoted from src.opts.mk to bsd.opts.mk to leverage
syntactic sugar for having MK_TESTS be a dependency for
MK_MAKE_CHECK_USE_SANDBOX, but to also ensure that src.opts.mk isn't required
to use suite.test.mk (which is a dependency of bsd.test.mk).

suite.test.mk:
- beforecheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified
from a no-op to:
-- Build.
-- Run "make hierarchy" on the sandbox dir.
-- Install the tests/files to the sandbox dir.
- aftercheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified
from a no-op to:
-- Remove the sandbox dir.

Again, because the dependency order set in bsd.test.mk is
beforecheck -> check -> aftercheck, "make check" will not be run unless
"beforecheck" completes successfully, and "aftercheck" will not be run unless
"beforecheck" and "check" complete successfully.

== Caveats ==

- This target must either be run with MK_INSTALL_AS_USER or as root. Otherwise
it will fail when running "make install" as the default user/group for many
makefiles when calling INSTALL is root/wheel.
- This target must be run from a suitable top-level directory. For example,
running tests from `tests/sys/fs/tmpfs` won't work, but `tests/sys/fs` will,
because `tests/sys/fs/tmpfs` relies on files installed by `tests/sys/fs`.
- Running MK_INSTALL_AS_USER may introduce determinism issues. However, using
it could identify deficiences in tests in terms of needing to be run as
root, which are not properly articulated in the test requirements.
- The doesn't negate the need for running "make installworld" and
"make checkworld", etc. Again, this just is intended to simplify the
dev->test->commit workflow.

== Cleanup done ==
- CHECKDIR is removed; one can use "MK_MAKE_CHECK_USE_SANDBOX=no" to enable
"legacy" (r295380) behavior.

MFC after: 2 months
Relnotes: yes (CHECKDIR removed; "make check" behavior changed)
Requested by: jhb
Reviewed by: arch (silence), testing (silence)
Differential Revision: D11905

Always expose DESTDIR as MAKE_CHECK_SANDBOX_DIR and always add it to CLEANDIRS
This removes a need for passing CLEANDIRS in to "make clean" in a slightly
convoluted way.

Also, remove the ${MAKE_CHECK_SANDBOX_DIR} is already present -> bail logic.
It was incredibly annoying dealing with false positives/failures (of which
there are a handful in the tree with tests). It's better to just assume that
the user is smart enough to do the right thing and not invoke "make check"
(sandboxed) multiple times, keeping ${MAKE_CHECK_SANDBOX_DIR}.

Fix last minute change I made for checking ${MK_MAKE_CHECK_USE_SANDBOX}

It needs to be compared against == "yes", not evaluated for its existence,
i.e., as a boolean sentinel.

Expose _TESTS_USE_OBJDIR as MK_MAKE_CHECK_USE_SANDBOX and anchor the default
value on MK_TESTS.

Use bsd.opts.mk in suite.test.mk to toggle the behavior.

Only tweak DESTDIR, etc if actively invoking "make check"

This fixes "make installworld"/"make install" (standard use).

Clean up DESTDIR, not CHECKDIR

While here, make the variable immediate to avoid silly mistakes that might
result in wiping out / by accident if the variable is evaluated in a delayed
manner due to poor use of $DESTDIR.

Make "make check" smarter when run from a root directory

If the user decides to specify HAS_TESTS in the upper directory, "make check"
will now iterate down the "test" directory, running the tests with an assumably
sane default value for $LD_LIBRARY_PATH, $PATH, etc.

The purpose of this work is to simplify "make check" -- in particular, to
increase dev-test velocity and get me (and others who pay attention, like
me) out of the business of paying attention to Jenkins runs and get other
developers to (hopefully) test their code more often prior to commit.

Huge caveat:

Obviously, this will fail miserably when the host can't run the target
architecture/code. Necessary compat layer or qemu usermode will be required
to make this possible, which is outside the scope of this work (batteries
not included).

Merge ^/user/ngie/release-pkg-fix-tests to unbreak how test files are installed
after r298107

Summary of changes:

- Replace all instances of FILES/TESTS with ${PACKAGE}FILES. This ensures that
namespacing is kept with FILES appropriately, and that this shouldn't need
to be repeated if the namespace changes -- only the definition of PACKAGE
needs to be changed
- Allow PACKAGE to be overridden by callers instead of forcing it to always be
`tests`. In the event we get to the point where things can be split up
enough in the base system, it would make more sense to group the tests
with the blocks they're a part of, e.g. byacc with byacc-tests, etc
- Remove PACKAGE definitions where possible, i.e. where FILES wasn't used
previously.
- Remove unnecessary TESTSPACKAGE definitions; this has been elided into
bsd.tests.mk
- Remove unnecessary BINDIRs used previously with ${PACKAGE}FILES;
${PACKAGE}FILESDIR is now automatically defined in bsd.test.mk.
- Fix installation of files under data/ subdirectories in lib/libc/tests/hash
and lib/libc/tests/net/getaddrinfo
- Remove unnecessary .include <bsd.own.mk>s (some opportunistic cleanup)

Document the proposed changes in share/examples/tests/tests/... via examples
so it's clear that ${PACKAGES}FILES is the suggested way forward in terms of
replacing FILES. share/mk/bsd.README didn't seem like the appropriate method
of communicating that info.

MFC after: never probably
X-MFC with: r298107
PR: 209114
Relnotes: yes
Tested with: buildworld, installworld, checkworld; buildworld, packageworld
Sponsored by: EMC / Isilon Storage Division

Simplify running the FreeBSD test suite

Replace `make regress` (legacy test make target) and `make test` (incomplete
test make target added with the FreeBSD test suite) with make check as it's
consistent with other open source projects.

`make check` defaults to running tests from `.OBJDIR`, but can be overridden
with the `CHECKDIR` variable.

Add `make checkworld` target to simplify running the FreeBSD test suite from
`TESTSBASE` (i.e. the top-level tests directory), similar to buildworld.

Document `make check` and `make checkworld` in build(7).

Other minor changes:

- Rename intermediate file (`Kyuafile.auto`) to `Kyuafile` to simplify
`make check`.
- Remove terse warnings attached to `beforetest`/`aftertest`.
- Add kyua binary check to check target in suite.test.mk; error out if it's
not found

The MFC is [partly] contingent on other build related changes being MFCed.

Differential Revision: https://reviews.freebsd.org/D4406
MFC after: 2 months
X-MFC to: stable/10
Relnotes: yes
Reviewed by: bdrewery, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division

Clean up Kyuafile.tmp, not Kyuafile.auto.tmp

MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D4406 (part of a larger diff)
Reviewed by: emaste, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division

- Use LOCALBASE instead of KYUA_PREFIX for the --prefix to kyua(1)
- Use LOCALBASE instead of hardcoding /usr/local for perl

MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D4406 (part of a larger diff)
Reviewed by: emaste, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division

Deal with another hardcoded reference to Kyuafile in the KYUAFILE == auto
case

Differential Revision: https://reviews.freebsd.org/D4406 (part of a larger diff)
Reviewed by: emaste, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division

Fix typo in r292500 by adding missing conditional statement

MFC after: 1 week
X-MFC with: r292500
Pointyhat to: ngie
Differential Revision: https://reviews.freebsd.org/D4406 (part of a larger diff)
Sponsored by: EMC / Isilon Storage Division

Simplify Kyuafile generation logic with KYUAFILE == auto and
related complexity with variables

MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D4406 (part of a larger diff)
Reviewed by: emaste, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division

Remove redundant default TESTSDIR that is already defined in bsd.test.mk
after r289158

MFC after: 1 week
X-MFC with: r289158
Sponsored by: EMC / Isilon Storage Division

Remove 'set -e' that are no longer needed as it is already default.

When bmake was initially imported at r241298 shell commands were no longer
ran with 'set -e' as they were before. This was fixed in r254980 so they
again always use 'set -e'.

Sponsored by: EMC / Isilon Storage Division

Add support for TEST_METADATA

Allow Makefiles to define generic metadata settings that apply to all test
programs defined by a Makefile. The generic TEST_METADATA variable extends
the per-test program settings already supported via TEST_METADATA.<program>.

This feature will be useful to easily apply some settings to all programs
in a directory. In particular, Kyua 0.12 will support parallel execution
of test programs and a bunch of them will need to be tagged as is_exclusive
to indicate that they cannot be run in parallel with anything else due to
their side-effects. It will be reasonable to set this setting on whole
directories.

MFC after: 1 week

Move FreeBSD Test Suite-specific code to a suite.test.mk file.

The new suite.test.mk file contains all the logic needed to install test
programs under /usr/tests/ and to support Kyua as the run-time engine.
This file is included by default by bsd.test.mk so Makefiles do not need
to care about its existence.

Specific Makefiles can define NOT_FOR_TEST_SUITE to indicate that whatever
test programs they are building are not supposed to be installed under
/usr/tests/ nor run by Kyua. (The effect of passing this setting is that
suite.test.mk is simply not included.)

NOT_FOR_TEST_SUITE should never be used by Makefiles in the base system.
This functionality is provided so that third-parties can hook in their
own test code, with different semantics, if they wish. This was asked
for by sjg@.