tools/build/make.py: Avoid Python 3.7+ subprocess.run capture_output

This is just a convenient alias for setting stdout and stderr to PIPE,
so substitute it for that to be compatible with Python 3.6.

Fixes: 69cfdc81ea7b ("tools/build/make.py: Keep bootstrapped bmake binary up-to-date")

tools/build/make.py: Add missing comma to fix tinderbox and worlds

The missing comma meant this was interpreted as a single target called
"tinderboxworlds", and so neither tinderbox nor worlds were recognised
as being MI targets (i.e. still required TARGET(_ARCH) to be given).

Fixes: 5157b451c654 ("tools/build/make.py: Grow the list of MI targets")

tools/build/make.py: Make --with-default-sys-path mirror usr.bin/bmake

The top-level Makefile passes -m to its sub-makes in order to ensure
they use the in-tree mk files in share/mk, but the top-level make itself
has to rely on whatever environment the bmake used has. For FreeBSD, we
configure the system bmake with .../share/mk:/usr/share/mk, which means
it will pick up src's share/mk whenever run from within the src tree,
but currently for non-FreeBSD we configure our bootstrap bmake only with
bmake's own mk files. This is mostly compatible, with two exceptions:

1. "targets" runs at the top level, but needs TARGET_MACHINE_LIST and
the corresponding MACHINE_ARCH_LIST_${target}, otherwise it will just
print an empty list.

2. "universe" and "universe-toolchain", when run at the top level (i.e.
not via the various wrappers around universe like tinderbox), end up
failing in universe-toolchain itself with:

bmake[1]: "/path/to/freebsd/share/mk/src.sys.obj.mk" line 112: Cannot use MAKEOBJDIR=
Unset MAKEOBJDIR to get default: MAKEOBJDIR='${.CURDIR:S,^${SRCTOP},${OBJTOP},}'

By including .../share/mk in the default sys path like FreeBSD's system
bmake we ensure that we get the in-tree mk files for the top-level make,
not just sub-makes, and avoid such issues.

Note that we cannot (yet) stop using the installed mk files, since the
MAKEOBJDIRPREFIX check in Makefile runs in the object directory and uses
env -i, thereby losing the MAKESYSPATH exported by src.sys.env.mk. Other
such issues may also exist, though are likely rare if so.

Reviewed by: sjg
Differential Revision: https://reviews.freebsd.org/D41544

tools/build/make.py: Keep bootstrapped bmake binary up-to-date

We currently assume that any existing bootstrapped bmake binary will
work, but this means it never gets updated as contrib/bmake is, and
similarly we won't rebuild it as and when the configure arguments given
to boot-strap change. Whilst the former isn't necessarily a huge problem
given WANT_MAKE_VERSION rarely gets bumped in Makefile, having fewer
variables is a good thing, and so it's easiest if we just always keep it
up-to-date rather than trying to do something similar to what's already
in Makefile (which may or may not be accurate, given updating FreeBSD
gives you an updated bmake, but nothing does so for our bootstrapped
bmake on non-FreeBSD). The latter is more problematic, though, and the
next commit will be changing this configuration.

We thus now add in two checks. The first is to compare MAKE_VERSION
against _MAKE_VERSION from contrib/bmake/VERSION. The second is to
record at bootstrap time the exact configuration used, and compare that
against what we would bootstrap with.

Reviewed by: arichardson, sjg
Differential Revision: https://reviews.freebsd.org/D41556

tools/build/make.py: Grow the list of MI targets

tools/build/make.py: Make flake8 clean

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

tools/build/make.py: Drop problematic Linux hack for bmake bootstrap

Since e2eeea75eb8b ("Merge bmake-20201117") missing/sys/cdefs.h has
been present in bmake, and _GNU_SOURCE seems to have been defined by
config.h for much longer than that, possibly for the entire time OS
cross-build support has been in-tree, so these are obsolete. Moreover,
since 79e02149fcb4 ("Fix dtrace tools bootstrap on non-FreeBSD after
OpenZFS import"), HAVE_STRLCAT and HAVE_STRLCPY have been defined by our
cross-build headers in order to placate DTrace tools (which is not the
right way to solve that problem, but motivates fixing this one). Commit
4fde40d9b540 ("Merge/update to bmake-20230126") changed the strlcpy.c in
bmake from including config.h directly to including make.h, which means
it includes string.h and thus sees these bogus definitions, causing it
to not define the strlcpy compat function on Linux even though it needs
to and thus failing to link. Thus, fix this whole mess by removing the
hack we no longer need.

tools/build/make.py: fix cross build on Fedora Linux

Fedora defines shell functions for some commands used by FreeBSD build
scripts. Unortunatelly it makes them behave incorrectly for our purposes.

For instance 'which which' returns something like:

which ()
{
( alias;
eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias ...
}

instead of

/usr/bin/which

This patch unsets those functions to restore original/expected behavior

Reviewed by: emaste, imp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D36900

Allow bootstrapping llvm-tblgen on macOS and Linux

This is needed in order to build various LLVM binutils (e.g. addr2line)
as well as clang/lld/lldb.

Co-authored-by: Jessica Clarke <jrtc27@FreeBSD.org>
Test Plan: Compiles on ubuntu 18.04 and macOS 11.4
Reviewed By: dim
Differential Revision: https://reviews.freebsd.org/D31057

tools/build/make.py: Avoid calling brew --prefix on macOS unnecessarily

If all the require variables (XCC/XCXX/XCPP/XLD) are already set in the
environment, we don't have to infer a default value for the cross toolchain
path. This avoids an additional `brew --prefix` call when building with
cheribuild (since it already sets all these variables).

tools/build/make.py: Don't call brew --prefix if --cross-bindir is set

Also updated the logic to use subprocess.run() instead of the old
subprocess.getoutput() which also includes stderr and therefore
can trigger an exception inside Path().exists().

Reported by: gnn

tools/build/make.py: drop workaround for cc --version not being parsed

Previously bsd.compiler.mk was not able to detect the compiler type for
Ubuntu's /usr/bin/cc unless we were invoking the /usr/bin/gcc symlink.
This problem has been fixed by 9c6954329a9285547881ddd60e393b7c55ed30c4
so we can drop the workaround from make.py.

Reviewed By: jrtc27
Differential Revision: https://reviews.freebsd.org/D28323

tools/build/make.py: -DNO_CLEAN -> -DWITHOUT_CLEAN

tools/build/make.py: Fix macOS build after a920b9817

If we set STRIPBIN, we also have to set XSTRIPBIN since we otherwise
use the host /usr/bin/strip during buildworld. However, this does not
work on macOS since /usr/bin/strip doesn't handle ELF binaries.

make.py: Also pass STRIPBIN

This is required for cross-building to allow stripping the installed binaries.

Submitted By: Henry Vogt <hv@tue.mpg.de>

GH Actions: Use pre-installed clang packages

Also fix the run by setting up the environment in non-deprecated way.

Always run with --debug to understand better what sort of stuff is happening in
the background. Also split out the bmake bootstrap stage (takes about 31s on
ubuntu, but 1m14 on macOS?)

Drops the dependency on coreutils (realpath, nproc) and thus (?) fixes macOS to
be just as fast (4 logical cores vs 2 physical cores before, go figure.)

Reviewed by: arichardson

Revert "When building on Ubuntu bootstrap bmake with bash as the default shell"

This reverts r365950 since the latest bmake update includes fixes for the test
failures that prompted the change.

When building on Ubuntu bootstrap bmake with bash as the default shell

The Ubuntu /bin/sh (dash) removes all environment variables that contain
characters outside the [a-zA-Z0-9_] range and this breaks the bmake tests that
run as part of bootstrapping bmake.
This can be reverted when the bmake tests have been updated.

Add a tools/build/make.py script that bootstraps bmake and then runs the build

This makes it possible to compile on non-FreeBSD systems since make will
usually be GNU make there. Even if they include bmake, it will often
either be a broken version or too old to build FreeBSD.

This should be the last commit needed to compile FreeBSD on Linux+macOS.
After over two years, I've finally managed to upstream all our local CheriBSD
changes to allow building on Linux (and as a result of being reviewed by more
people they are slightly less ugly than they were before).

It should now be possible to run the following to build on Linux+macOS if you
have LLVM/Clang 10 or newer installed:
MAKEOBJDIRPREFIX=/somewhere ./tools/build/make.py TARGET=amd64 TARGET_ARCH=amd64 buildworld

I have only tested macOS 15, Ubuntu 18.04 and openSUSE Leap, but other Linux
distributions might also work (as long as they ship a recent GLibc and compiler).

Reviewed By: emaste (should be fine to commit to tools/)
Differential Revision: https://reviews.freebsd.org/D16767