showconfig: Set MACHINE for src.opts.mk

Also set MACHINE and MACHINE_ARCH when reading config options from
src.opts.mk. This ensures any machine-dependent options are reported
correctly.

Reviewed by: emaste, imp
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D44838

Makefile.inc1: Fix typo

Signed-off-by: Elyes Haouas <ehaouas@noos.fr>
Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/882

pkgbase: add a mechanism to be able to force a give ucl include

This is made in order to be able to find add the post-install scripts
for the kernel, where PKGNAME varies for each KERNCONF but we don't want
to dynamically duplicated the kernel.ucl file.

At the same time we don't want the *-dbg* packages to actually include
those post-install scripts

pkgbase: fix packaging of dtb

pkgbase: fix typo preventing packaging kernels

pkgbase: fix kernel packing after switch ucl rework

pkgbase: rework certctl package to only run rehash on the main package

Rework how ucl manifest are generated leveraging ucl features and flua

now the ucl generation is done via a lua script which uses libucl to
ingest the template and use variables as defined in its command line.

the template will include only if it exist a ucl file named after the
package name which will complement the template or overwrite what was
defined in the template if defined in this specific ucl file

this allows to overwrite license, but add script only to the packages
who actually needs them.

As a results the post install scripts are now only added to the right
package and not also added to the subpackages like -man or -dev

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D44374

libc: link libsys as a auxiliary filter library

At runtime, when rtld loads libc it will also load libsys. For each
symbol that is present in both, the libsys one will override the libc
one. It continues to be the case that program need only link against
libc (usually implicitly). The linkage to libsys is automatic.

Reviewed by: kib, emaste, imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/908

Reapply "pkgbase: Create a FreeBSD-dtb package"

This reverts commit 9bbe06b004e2cadb55a3792c77991ee640bf0f52.

Before that dtbs where included in each kernel packages which prevents
us to install multiple kernels.

Fix mtree-to-plist.awk to only generate a dtb.plist when we create
the kernel one (because dtb are installed during make installkernel).

Sponsored by: Beckhoff Automation GmbH & Co. KG

Revert "pkgbase: Create a FreeBSD-dtb package"

Somehow this doesn't work iwth make packages due to some kind of a race.
The package is first created correctly but later in the process it is
overwritten by a badly created empty package.
Revert in the mean time so we can have working pkgbase on arm/arm64

This reverts commit a5afd7920d50ff23f91bdbabb1f0fd7ba028ba9e.

pkgbase: Create a FreeBSD-dtb package

Before that dtbs where included in each kernel packages which prevents
us to install multiple kernels.

Differential Revision: https://reviews.freebsd.org/D43632
Reviewed by: bapt
Sponsored by: Beckhoff Automation GmbH & Co. KG

pkgbase: Create two new rules for creating repo

This adds two new rules named create-packages-kernel-repo and
create-packages-world-repo.
The goal of those rules is to create the {kernel,world} packages and
after that the repository.
It helps a lot for developing with pkgbase by adding the dev machine
repository created by those rules on top of the official pkgbase one.

Differential Revision: https://reviews.freebsd.org/D43623
Reviewed by: bapt, emaste
Sponsored by: Beckhoff Automation GmbH & Co. KG

build: Use ls -i | cut rather than stat -f %i to extract inode

ls -i dates back to 5th edition Unix and is more portable than the stat
command, though %% is a newer shellism, it works on any shell that
FreeBSD builds with.

Sponsored by: Netflix

Revert "kerberos: Fix numerous segfaults when using weak crypto"

This revision breaks Linux and MacOS cross builds because
TARGET_ENDIANNESS is not define during bootstrapping on these
platforms.

I think the correct approach would be to separate the new
fbsd_ossl_provider_load() and unload functions into their own
library (instead of libroken). This avoids the less desirable
option of including bsd.cpu.mk in secure/lib/Makefile.common,
which does build but could complicate future work.

Reported by: jrtc27

This reverts commit cb350ba7bf7ca7c4cb97ed2c20ab45af60382cfb.

kerberos: Fix numerous segfaults when using weak crypto

Weak crypto is provided by the openssl legacy provider which is
not load by default. Load the legacy providers as needed.

When the legacy provider is loaded into the default context the default
provider will no longer be automatically loaded. Without the default
provider the various kerberos applicaions and functions will abort().

PR: 272835
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D43009
Tested by: netchild, Joerg Pulz <Joerg.Pulz@frm2.tum.de>

dtrace: Add the 'oformat' libdtrace option

This option can be used to specify a format to use in DTrace output.
The following formats are supported:
- json
- xml
- html
- none (default DTrace output)

This is implemented using libxo and integrated into libdtrace. Client
code only works with the following API:

- dtrace_oformat_setup(dtrace_hdl_t *) -- to be called when output is starting.
- dtrace_oformat_teardown(dtrace_hdl_t *) -- to be called when output is finished
- dtrace_oformat(dtrace_hdl_t *) -- check if oformat is enabled.
- dtrace_set_outfp(FILE *) -- sets the output file for oformat.
- Ensure that oformat is correctly checked in the drop handler and record
processing callbacks.

This commit also adds tests which check if the generated output is
valid (JSON, XML) and extends the dtrace(1) describing the structured output.

Reviewed by: markj
Discussed with: phil
MFC after: 2 months
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D41745

Make kldxref a bootstrap tool and use unconditionally

Now that kldxref is a generic cross tool and can be built on non-FreeBSD
we can bootstrap it during the build and thus remove the condition for
whether it exists. We also need to make sure to add it to the METALOG
for -DNO_ROOT builds.

Reviewed by: brooks, imp
Differential Revision: https://reviews.freebsd.org/D43051

Makefile.inc1: Forward on METALOG and DISTBASE for kernel targets

Currently IMAKE_INSTALL, which includes -M METALOG, is enough for the
sub-makes to work, but using kldxref for -DNO_ROOT builds will require
manually adding linker.hints to the METALOG, and thus both METALOG
itself and DISTBASE need to be exposed directly to the sub-makes, so do
so.

Reviewed by: brooks
Differential Revision: https://reviews.freebsd.org/D43050

Makefile.inc1: Reinstate building sbin/md5 for BOOTSTRAP_ALL_TOOLS

The sha256 and sha512 binaries are used when building release images.
Bootstrapping them used to be gated on MK_BOOT, which wasn't quite right
but in practice worked, at least in our use cases downstream. Add back
bootstrapping sbin/md5 and its aliases for this purpose to fix building
release images on Linux and macOS.

Fixes: f213da893ca8 ("Makefile.inc1: Remove beri straggler")

pkgbase: pass --recurse-submodules to `git ls-files`

When generating source packages. Although submodules are not used by
FreeBSD itself they may be used by downstream projects. By default
for submodules `git ls-files` just emits the submodule directory name,
which resulted in:

pkg: pkg_checksum_hash_sha256_file(read failed): Is a directory

Passing --recurse-submodules lists all of the files in each submodule
(which is desired when submodules are in use), and has no effect when
submodules are not present.

Reviewed by: bapt, manu
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D42983

build: use bare (and portable) echo instead of echo -n

There's no need to use echo -n here. A single echo will do nicely.
This fixes the post-buildworld output on a macos build, where echo -n
is implemented like System V instead of BSD (so you get two lines
first one starting with -n).

Sponsored by: Netflix
Reviewed by: jrtc27, emaste
Differential Revision: https://reviews.freebsd.org/D42869

pkgbase: propagate SRCRELDATE to the packages correctly

MFC After: 3 days
Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D42892

pkgbase: create source package

FreeBSD-src for all the sources but the kernel
FreeBSD-src-sys just for the kernel

MFC After: 3 days
Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D42651

Makefile.inc1: Remove beri straggler

Beri's boot loader needed md5 to build. It was the only thing that
needed that, so remove it (confirmed with univers and grep).

Sponsored by: Netflix

pkgbase: create package with SRCRELDATE not OSRELDATE as OSVERSION

In 188fe88ec50eac7c10e1d8350bf1180f8c16e463 OSVERSION has been set
to OSRELDATE which is the RELDATE of the building OS while we wanted
to use SRCRELDATE which is the RELDATE of the target system

pkgbase: use <major>.<minor> format for alpha, beta and release package versions

This is consistent with version numbers used in releng/13.2.

PR: 275051
Reviewed by: bapt
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D42562

pkgbase: force OSVERSION

This is used to the package annotation helping pkg to know about
backward compatibility is set to the version of the packages not
the version of the host building the packages

pkgbase: set a default set of kernel for when PACKAGE_BUILDING=1

PACKAGE_BUILDING is already known in the ports tree as a variable
use to defined when the packages is being actually built in an
automation process, reuse that variable to define the default set
of kernel we plan to build for the default pkgbase.

Reviewed by: manu, jrtc27
Accepted by: manu
MFC After: 1 week
Differential Revision: https://reviews.freebsd.org/D42444

ASan: Add asan_static to build

This is a partial fix for building with -DWITH_ASAN.

Reviewed by: Fangrui Song, dim, emaste
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D42388

Remove MOVED_LIBS handling from list-old-libs

In 922337e8d398 I added MOVED_LIBS into list-old-files, so that
delete-old-files would remove the old /usr/lib/libc++.so.1 as soon as
possible (after the library moved to /lib).

I left it in list-old-libs in case a user updated their src tree between
delete-old-files and delete-old-libs. Now that some time has passed,
tremove the redundant MOVED_LIBS entry.

PR: 272642
Sponsored by: The FreeBSD Foundation

pkgbase: compress packages with zstandard

MFC After: 3 days
Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D42375

certctl: invoke with LOCALBASE set

When invoking certctl as part of installworld, set LOCALBASE in the
environment to match the build environment. That ensures that LOCABASE
is non-empty on systems without the user.localbase sysctl and avoids
allowing a system configuration detail to leak into the build. Users
who wish to build targeting a non-standard LOCALBASE should ensure it is
set in src.conf or similar.

Reviewed by: Mina Galić <freebsd@igalic.co>
Differential Revision: https://reviews.freebsd.org/D40530

Makefile.inc1: remove ncurses/form from _prebuild_libs

As of d287d3282f43 libbsddialog has a built-in form implementation and
does not need a formw dependency.

This reverts commit 483a226238ed8949c6d280ae0757a0683962a74b.

Reported by: asiciliano
Sponsored by: The FreeBSD Foundation

Handle MOVED_LIBS in list-old-files

MOVED_LIBS is used when a library moves from one directory to another,
e.g. /usr/lib/libc++.so.1 to /lib/libc++.so.1.

delete-old-files and delete-old-libs are two separate steps, so that
old libraries can be retained until third party software packages are
rebuilt or reinstalled.

Having two copies of the same shared library with the same so version
(as can happen when delete-old-libs hasn't been run) causes trouble.
The PR below gives one example.

Libraries listed in MOVED_LIBS are logically equivalent to updating a
library without changing the so version, and should be removed as soon
as possible. Handle them in list-old-files and thus delete-old-files.

Leave them also in *-old-libs for now, in case the user updates their
tree between running delete-old-files and delete-old-libs.

PR: 272642
Reviewed by: dim
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Co-authored-by: Dimitry Andric <dim@FreeBSD.org>
Differential Revision: https://reviews.freebsd.org/D42197

libcrypto: link engines and the legacy provider to libcrypto

OpenSSL's legacy provider module and engines need to link to
libcrypto.so, as it provides some of the actual implementations of
legacy routines.

This is a little tricky due to build order issues. Introduce a small
hack (LIBCRYPTO_WITHOUT_SUBDIRS) that builds libcrypto.so in its usual
early phase without any OpenSSL provider modules or engines. This is
intended to restore the test suite; a future change should remove the
hack and replace it with a better approach.

PR: 254853, 273528
Discussed with: Folks at EuroBSDCon in Coimbra
Sponsored by: The FreeBSD Foundation

Teach make showconfig about group options

Output OPT_ variables in addition to MK_ variables.

Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D41682

zfs: merge openzfs/zfs@804414aad

Notable upstream pull request merges:
#15024 Add missed DMU_PROJECTUSED_OBJECT prefetch
#15029 Do not request data L1 buffers on scan prefetch
#15036 FreeBSD: catch up to __FreeBSD_version 1400093
#15039 Fix raw receive with different indirect block size
#15047 FreeBSD: Fix build on stable/13 after 1302506
#15049 Fix the ZFS checksum error histograms with larger record sizes
#15052 Reduce bloat in ereport.fs.zfs.checksum events
#15056 Avoid extra snprintf() in dsl_deadlist_merge()
#15061 Ignore pool ashift property during vdev attachment
#15063 Don't panic if setting vdev properties is unsupported for this vdev type
#15067 spa_min_alloc should be GCD, not min
#15071 Add explicit prefetches to bpobj_iterate()
#15072 Adjust prefetch parameters
#15076 Refactor dmu_prefetch()
#15079 set autotrim default to 'off' everywhere
#15080 ZIL: Fix config lock deadlock
#15088 metaslab: tuneable to better control force ganging
#15096 Avoid waiting in dmu_sync_late_arrival()
#15097 BRT should return EOPNOTSUPP
#15103 Remove zl_issuer_lock from zil_suspend()
#15107 Remove fastwrite mechanism
#15113 libzfs: sendrecv: send_progress_thread: handle SIGINFO/SIGUSR1
#15122 ZIL: Second attempt to reduce scope of zl_issuer_lock
#15129 zpool_vdev_remove() should handle EALREADY error return
#15132 ZIL: Replay blocks without next block pointer
#15148 zfs_clone_range should return descriptive error codes
#15153 ZIL: Avoid dbuf_read() before dmu_sync()
#15172 copy_file_range: fix fallback when source create on same txg
#15180 Update outdated assertion from zio_write_compress

Obtained from: OpenZFS
OpenZFS commit: 804414aad224b432590afe3f9ec114ffb49e0f13

kbdcontrol: Support building as a bootstrap tool on old and non-FreeBSD

Systems that predate 971bac5ace7a ("kbd: consolidate kb interfaces
(phase one)") cannot build kbdcontrol since kbdelays and kbrates moved
to sys/kbio.h. Moreover, on non-FreeBSD, it requires all kinds of ioctls
and sysctls that are highly FreeBSD-specific to build, but we use it as
a bootstrap tool to generate the keymaps used by some kernels (LINT ones
in particular). Thus, when bootstrapping kbdcontrol, disable everything
that's not needed for that singular use, and use the in-tree kbio.h to
get the definitions of the necessary structures.

This allows KBDMUX_DFLT_KEYMAP, UKBD_DFLT_KEYMAP and ATKBD_DFLT_KEYMAP
to be enabled when building on non-FreeBSD, and thus LINT kernels.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D41541

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

Makefile.inc1: Remove redundant test for armv[67]

If MACHINE is arm, then MACHINE_ARCH is going to be either armv6* or
armv7*.

Sponsored by: Netflix

Makefile.inc1: Enable requesting the universe toolchain.

make universe builds a cross toolchain under HOST_OBJTMP/tmp via the
universe-toolchain target. However, doing a plain 'make buildworld'
after a universe/tinderbox run (e.g. to reproduce a failure and test
the fix for it), will try to build a new cross toolchain under
OBJTMP/tmp which can be tedious. This commit adds a make variable
(UNIVERSE_TOOLCHAIN) which can be used similar to CROSS_TOOLCHAIN to
request an external toolchain. If this variable is set (value doesn't
matter), the the universe toolchain is used as an external toolchain.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D40777

Makefile.inc1: Fix list-old-libs by deleting stray pipe

Reported by: Yuri <yuri@aetern.org>
Fixes: 264594efbe69 ("Makefile.inc1: Automatically generate _LIBCOMPATS and list-old-* libcompats")

Makefile.inc1: Reinstate kbdcontrol bootstrap for BUILD_WITH_STRICT_TMPPATH

During BUILD_WITH_STRICT_TMPPATH builds we need to symlink the host's
kbdcontrol into WORLDTMP/legacy/bin so that it can be used. For
completeness, also check if the host has it, since technically the host
could have been built with WITHOUT_LEGACY_CONSOLE (though given nobody's
screamed since this code was removed that's likely not something that
gets hit in practice), and handle BOOTSTRAP_ALL_TOOLS (especially since
that case will cover cross-building if and when kbdcontrol can be built
on non-FreeBSD).

This allows a BUILD_WITH_STRICT_TMPPATH tinderbox to pass.

This partially reverts commit 33550b47391330362ea58984aa3fd691f6219ee0.

Reviewed by: imp, emaste
Differential Revision: https://reviews.freebsd.org/D41202

Makefile.inc1: Add new buildetc and installetc targets

These implement the steps that etcupdate (and mergemaster) need in order
to build their pristine trees, avoiding the need to use internal targets
and variables. Additionally, buildetc includes the steps needed to build
with BUILD_WITH_STRICT_TMPPATH enabled, since we need some host tools to
be available during the various build and install steps.

Reviewed by: jhb, imp
Differential Revision: https://reviews.freebsd.org/D41205

Makefile.inc1: Split _cleanworldtmp out from _worldtmp

This mirrors _cleanobj vs _obj, and will be used by etcupdate to ensure
its build tree has host tools available when BUILD_WITH_STRICT_TMPPATH
is enabled by default (whether in the source tree, like CheriBSD, or in
a system config file).

Note that due to line length the .MAKE .PHONY dependency line is split
into two, one for the filtered WMAKE_TGTS and one for the filtered
.ALLTARGETS.

Reviewed by: brooks, jhb
Differential Revision: https://reviews.freebsd.org/D41190

tools/build: Create toolchain symlinks for non-absolute compiler/linker

If any of the toolchain variables are not absolute then we need to
create a symlink in WORLDTMP/legacy/bin in order to make them available
during a BUILD_WITH_STRICT_TMPPATH build.

Reviewed by: brooks, jhb
Differential Revision: https://reviews.freebsd.org/D41188

depend-cleanup.sh: Generalise lib32 code and avoid duplication

By passing through _ALL_libcompats we can avoid hard-coding the list of
libcompats in depend-cleanup.sh. This cleanup also makes clean_dep
shorter by using a loop instead of handling each case explicitly (at the
expense of slightly tweaked logging).

Reviewed by: brooks, jhb
Differential Revision: https://reviews.freebsd.org/D41187

Makefile.inc1: Automatically generate _LIBCOMPATS and list-old-* libcompats

Reviewed by: brooks, jhb
Differential Revision: https://reviews.freebsd.org/D41180

arm64 lib32: enable building of lib32 on arm64

Enable LIB32 option on aarch64, defaulting to YES; it had defaulted
to "broken". Add required variables for how to compile lib32 on
arm. Use /usr/include/arm for armv7 (32-bit) headers, analogous to
/usr/include/i386 on amd64. Omit libomp from lib32; it is not
supported on armv7.

Reviewed by: jrtc27
Differential Revision: https://reviews.freebsd.org/D40945

Makefile.inc1: add LIBCOMPAT_INCLUDE_DIRS to reduce arch ifdefs

In preparation for adding support for building lib32 on arm64,
add a list of architecture-specific include directories,
LIBCOMPAT_INCLUDE_DIRS, then replace the architecture-specific
ifdefs throughout the file with simple loops. Another commit
will add a definition of LIBCOMPAT_INCLUDE_DIRS for aarch64.

Reviewed by: jrtc27
Differential Revision: https://reviews.freebsd.org/D40977

Make sure MACHINE(_ARCH) are exported for depend-cleanup.sh

We run depend-cleanup.sh twice during the build. The second time is the
normal run, where we run it under WMAKEENV and thus have CROSSENV's
MACHINE(_ARCH)=${TARGET(_ARCH)} in the environment. However, the first
time is for bootstrap-tools, where it's run under BMAKEENV and we don't
have any assignments to MACHINE(_ARCH) in the environment, meaning the
script sees them as unset. In practice this doesn't matter since the
only use doesn't apply to bootstrap-tools, but it could be a future
issue. Thus, explicitly export them for depend-cleanup.sh and have the
script verify they're set.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D40968

Generalise libcompat to be a list rather than a single option

Whilst the kernel can support any number of COMPAT_FOO, world can only
build a single libfoo. Upstream this isn't such an issue, since the only
option is lib32 anyway, but downstreams, such as CheriBSD, may wish to
support multiple at the same time. Thus, adjust the top-level Makefiles
to turn _LIBCOMPAT into a _LIBCOMPATS list that gets iterated over, and
adjust bsd.compat.mk to support this use-case.

For the normal NEED_COMPAT/WANT_COMPAT case, LIBCOMPATFOO remain set and
refer to the requested compat's, preserving the current interface. For
the top-level Makefiles those variables are no longer set (since there
is no longer "the" compat) and only the per-compat ones are available.

Reviewed by: brooks, jhb, imp
Differential Revision: https://reviews.freebsd.org/D40571

Fix prebuild race between libpam and libtacplus.

Fixes: 21850106fdda
MFC after: 1 week
Sponsored by: Klara, Inc.
Reviewed by: kevans, emaste
Differential Revision: https://reviews.freebsd.org/D40702

Makefile.inc1: Fix distributeworld mtree mangling for dist root dir

The trailing slash means that ./base itself doesn't get mangled and
remains as-is in the output, leading to a stray /base in base.txz for
NO_ROOT builds and thus in the installed system. Since this action is
running on a line whose file matches one listed by find (and we're
printing all of these as part of that distribution), we don't need to
care about the possibility of a path like ./basefoo/bar where the path
prefix isn't ./base, and can thus just drop the slash rather than
needing something more complicated like "slash or whitespace or EOL" as
one might first think.

Makefile.inc1: Use INSTALL_DDIR for distributeworld's distrib-dirs

INSTALL_DDIR is the canonicalised version of DESTDIR/DISTDIR. Whilst
most of what distrib-dirs does doesn't need the canonicalised form, it
is responsible for installing the POSIX and en_US.US_ASCII NLS symlinks
to C, and therefore needs the canonicalised version for those two uses
of install for NO_ROOT builds, since our install does a naive text-based
prefix strip when creating the METALOG entry rather than a smarter path
semantics-aware one (which itself is really a bug, and has bitten us
many times). As a result, using plain DESTDIR/DISTDIR instead can result
in the METALOG having ./path/to/destdir/base/usr/share/nls/$LOCALE
rather than ./base/usr/share/nls/$LOCALE and then being filtered out
when creating base.meta (or, if you're unlucky and the absolute path
begins with base or tests, weird things will probably happen).

Given this footgun an audit of DESTDIR uses is probably in order,
especially those using DESTDIR/DISTDIR, but this is sufficient for now.

Makefile.inc1: use make builtin :ts instead of sed

Commit 3eb1b4da3cf7 replaced "xargs -n1" with a sed expression to
convert from space to newline as a list separator for *-old-* targets.
Dan Nelson followed up with a suggestion to use make's built-in :ts
instead, which should be slightly more efficient.

Reviewed by: sjg
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39972

showconfig - ignore options that do not start with A-Z

When building on various hosts it can be handy to create pseudo
options like MK_host_egacy - building on a host that needs libegacy.

Such options should be ignored by showconfig

Reviewed by: emaste

Stop stripping 'sf' suffixes from architecture names.

riscv64sf was the only architecture with an 'sf' suffix.

Reviewed by: imp, emaste
Differential Revision: https://reviews.freebsd.org/D39497

Remove the riscv64sf architecture.

Reviewed by: jrtc27, arichardson, br, kp, imp, emaste
Differential Revision: https://reviews.freebsd.org/D39496

libcasper: Move helper libraries from /lib/casper to /lib.

These libraries are linked to directly by applications rather than
opened at runtime via dlopen().

Discussed with: oshogbo
Reviewed by: markj, emaste
Differential Revision: https://reviews.freebsd.org/D39245

Add new DISK_IMAGE_TOOLS_BOOTSTRAP option

This will build etdump, makefs and mkimg as bootstrap tools to allow
easily creating disk images. Note that etdump is bootstrapped due to its
use in the release scripts for building ISO images.

Reviewed by: emaste, arichardson
Differential Revision: https://reviews.freebsd.org/D39072

make showconfig: set MACHINE and MACHINE_ARCH

Previously these were not set for the showconfig target, so took the
host's value rather than the target. As a result some machine-dependent
kernel options were not reported with correct defaults in the src.conf
manpage.

PR: 269994
Reviewed by: imp
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D38926

Makefile.inc1: Support building with macOS Ventura's AMFI Launch Constraints

As of macOS Ventura, Apple-signed binaries cannot be run if copied away from
their system location. This security feature doesn't really make sense for
boring things like sh(1), more so for applications with special entitlements,
but it's universally present, and results in the following error:

>>> Install check world
bmake[2]: "/Users/Jess/cheri/freebsd/Makefile.inc1" line 572: warning: "MAKEFLAGS= CPUTYPE=dummy /Users/Jess/cheri/build/freebsd-riscv64-build/bmake-install/bin/bmake -f /dev/null -m /Users/Jess/cheri/freebsd/share/mk MK_AUTO_OBJ=no -V CPUTYPE" exited on a signal
bmake[2]: "/Users/Jess/cheri/freebsd/Makefile.inc1" line 575: CPUTYPE global should be set with ?=.

As with host-symlinks, we don't actually need to copy the files on macOS, since
we're not updating the current machine, so copy its approach and just symlink
them instead.

MFC after: 1 week

Remove support for the base/* toolchain ports.

These ports have been removed so these knobs are no longer meaningful.

This reverts commit 608289394f655d0635c71233248b734010fb4d2f.
This reverts commit 39eb07f172921a581ee9b33ca2765905f4235202.

Reviewed by: imp, bapt, emaste
Differential Revision: https://reviews.freebsd.org/D38562

buildworld: drop config(8) dependencies from bootstrap

We may still need them for other parts, so just remove the dependency
relationship for now and simplify config's place in bootstrap-tools.

Reviewed by: imp
Sponsored by: Klara, Inc.
Sponsored by: NetApp, Inc.
Differential Revision: https://reviews.freebsd.org/D38277

depend-cleanup.sh: Simplify the logic, and clean bootstrap tools.

* There's no need to check if the file exists before grepping it; if it does not exist, grep will fail, which is what we want. Just redirect the error message to /dev/null.
* There's no need to split the .o and .pico cases; the bodies are identical, so combine the conditions. Use a glob to avoid a false negative if one exists but not the other.
* Also run depend-cleanup.sh on the bootstrap build tree. This unbreaks the build after 29c5f8bf9a01.

Sponsored by: Klara, Inc.
Sponsored by: NetApp, Inc.
Reviewed by: imp, emaste
Differential Revision: https://reviews.freebsd.org/D38447

Merge llvm-project main llvmorg-15-init-15358-g53dc0f10787

This updates llvm, clang, compiler-rt, libc++, libunwind, lld, lldb and
openmp to llvmorg-15-init-15358-g53dc0f10787.

PR: 265425
MFC after: 2 weeks

Fix _REVISION to be always major.minor

Introduced in commit 613fe53, _REVISION was redefined to be just the
major version (14) instead of major.minor (14.0) iff:

- PKG_VERSION was not overridden (default), and
- Branch was -CURRENT, -STABLE, or -PRERELEASE.

This introduced side effects in other parts of the code which expected
_REVISION to be major.minor:

- make-pkg-package.sh target triple became amd64-portbld-freebsd14
(should be amd64-portbld-freebsd14.0);
- Versioned cross dev symlink name became amd64-freebsd14-<tool>
(should be amd64-freebsd14.0-<tool>)

Also, compounded with commit ea9a92d (which was introduced afterward
and removed minor version from _REVISION incorrectly using :S with a
regex) this also caused pkgbase repo to use a wrong PKG_ABI with a minor
version (FreeBSD:14.0:amd64) when a custom PKG_VERSION was specified.
(Note, without a custom PKG_VERSION the bug was not triggered because in
that case _REVISION was already major-only.)

This commit fixes both problems by introducing and using MAJOR_REVISION
instead of redefining _REVISION. Existing uses of _REVISION now see
major.minor again, except PKG_ABI (ex: FreeBSD:14:amd64, as described
above) and PKG_VERSION (ex: 14.snap20220311121531) need only the major
version and now use MAJOR_REVISION instead.

PR: 262600, 262601
Reviewed by: imp, emaste
Pull Request: https://github.com/freebsd/freebsd-src/pull/592

Makefile.inc1: Restore adding tzsetup to _basic_bootstrap_tools

ITOOLS is only the list of programs to make a copy of during install, it
doesn't cause anything to be bootstrapped. Thus, by removing tzsetup
from _basic_bootstrap_tools, we end up without it on non-FreeBSD, and so
we error out trying to copy it to INSTALLTMP.

Note that _basic_bootstrap_tools is only used for BOOTSTRAP_ALL_TOOLS
(which is why zic was moved away from it). Should tzsetup evolve such
that the host version is insufficient on older FreeBSD it can be moved
to be more like zic, but that seems unnecessary for what is likely to
remain a simple tool.

This partially reverts commit 7a4a520064cb7fbff79560113682d6fe1a1ae9ee.

Fixes: 7a4a520064cb ("Fix cross-build from STABLE or older CURRENT.")

Makefile.inc1: Print the right PATH for missing install tool error

We override PATH for which but then go and print the non-overridden
PATH, which leads to rather confusing messages.

Fixes: 5bb9250e0af4 ("Add necessary Makefile.inc1 infrastructure for building on non-FreeBSD")

Revert "Don't omit bsd.compiler.mk for the nested delete-old in buildworld."

The original motivation (src.opts.mk needing bsd.comppiler.mk to
determine if CXX is supported) is no longer relevant now that CXX is
non-optional.

This reverts commit b9cb80883bce6dc992cf05ae2e59089a60d311ec.

Retire WITHOUT_CXX option

Several important base system components are written in C++, and the
WITHOUT_CXX option produced a system that was not fully functional.
Just accept this, and remove the option to build without C++ support.

This reverts commit adc3c128c6603054586a993d117e5dd808deac17.

Reviewed by: brooks, kevans, jhb (earlier)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33108

Fix cross-build from STABLE or older CURRENT.

Previously, zic and tzsetup were both listed as install tools and basic
bootstrap tools. Actually, tzsetup is an install tool while zic is a
non-basic bootstrap tool.

Fixes: 783c318fd118
Sponsored by: Klara, Inc.
Reviewed by: jrtc27, emaste
Differential Revision: https://reviews.freebsd.org/D38195

Shell-escape assignments to PATH in the top-level makefiles

Since 16fbf0191243e7c9dff6615b1424b5d39186b36c PATH is no longer set
to a hardcoded value on non-FreeBSD build hosts, so we can end up with
spaces in $PATH. Instead of only escaping PATH I updated all `env PATH=`
uses in the toplevel makefile. While many of these currently can't
contain any special characters (since the build would have failed
already), in theory this gets us closer to allowing build/source
directory to contain e.g. spaces.

Retire CLANG_IS_CC option

A small reduction in build infrastructure complexity; when we had both
Clang and GCC in the tree it was useful to have both built, and choose
one or the other to install as /usr/bin/cc. Now only Clang is in the
tree, and there is no point in building and installing base Clang but
not providing it as cc (and c++, cpp).

Reviewed by: imp
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D37075

Don't omit bsd.compiler.mk for the nested delete-old in buildworld.

src.opts.mk will need bsd.compiler.mk to determine if CXX is
supported. Without this, src.opts.mk always marks CXX as broken and
attempts to delete all dependencies of MK_CXX from WORLDTMP.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D36890

build: Use `rm -fv` for BATCH_DELETE_OLD_FILES

It's possible to have files with odd permissions in the tmproot (or
sysroot), causing rm to prompt for each one during e.g. buildworld.
Add -f to forcibly delete these.

Reviewed by: brooks
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D37111

Rename MACHINE_ABI and TARGET_ABI

The MACHINE_ABI and TARGET_ABI variables are used to set the middle of
the target triple (e.g., "-unknown-" or "-gnueabihf-"). They are not set
by any tool in the base system and I've only found the latter mentioned
in one review online. As such, rename them to to MACHINE_TRIPLE_ABI and
TARGET_TRIPLE_ABI to clear the way to use MACHINE_ABI as a supplement to
MACHINE_CPU, etc.

Reviewed by: imp, jhb
Obtained from: CheriBSD
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D36420

Put OPIE to rest.

Differential Revision: https://reviews.freebsd.org/D36592

Makefile.inc1 release bsd.own.mk: Introduce and use TAR_CMD

Our uses of tar rely on BSDisms, and so do not work in environments
where GNU tar is the default tar. Providing a TAR_CMD variable like
some other commands allows it to be overridden to use bsdtar in such
cases.

Reviewed by: brooks, delphij, gjb
Obtained from: CheriBSD
Differential Revision: https://reviews.freebsd.org/D35813

Makefile.inc1: Set LC_COLLATE in distributeworld for glibc compatibility

distributeworld relies on "foo" sorting directly before "foo type=...",
but with glibc both en_US and en_GB have "fooa" sort between "foo" and
"foo z", resulting in some files (in particular, id due to "ident"
sorting before "id type=" but after "id") not being included in the meta
files and thus not included in the dist tarballs. Forcing use of the C
locale ensures this does not occur.

Reviewed by: brooks
Obtained from: CheriBSD
Differential Revision: https://reviews.freebsd.org/D35812

Makefile.inc1: Honour DB_FROM_SRC for NO_ROOT distributeworld

Currently the host's database files are used, but on non-FreeBSD these
are not necessarily sufficient; in particular, Linux does not have a
wheel group. Instead, use -N to use the in-tree database files when
creating the METALOG entries, as is done for the recursive makes via
IMAKE_MTREE.

Reviewed by: brooks
Obtained from: CheriBSD
Differential Revision: https://reviews.freebsd.org/D35810

Bootstrap crunchgen after removing -dc from linker invocation

In ec81497cc726 crunchgen was updated to remove -dc from the linker
invocations in its generated makefile output, as this flag is no longer
necessary, and is going to be an error with lld 15.

Update the BOOTSTRAPPING conditions for copying the crunchgen binary
from the host, or actually bootstrapping it when necessary. Since
ec81497cc726 did not bump __FreeBSD_version, I have chosen the nearest
values.

Fixes: ec81497cc726
MFC after: 3 days

Makefile.inc1: Fix -DNO_ROOT distributeworld certctl usage

Currently for distributeworld we pass DESTDIR to certctl.sh as an
environment variable, which sets the default value in the script.
However, for -DNO_ROOT builds, CERTCTLFLAGS has METALOG_INSTALLFLAGS
which includes -D ${DESTDIR}, overriding the custom DESTDIR pointing at
the base dist directory.

Moreover, in order to ensure that the METALOG includes the base/ prefix
for all the files, we need to have certctl call install with -D set to
DESTDIR/DISTDIR without the /base suffix but also ensure the files get
installed to DESTDIR/DISTDIR/base.

Fix these by passing the custom DESTDIR to certctl via -D rather than in
the environment and to pass the /base suffix in the distributeworld case
via the newly-added -d option.

We also need to run certctl rehash before we generate the .meta files
from the METALOG, not after, otherwise they won't include the METALOG
additions, so move the certctl rehash call.

Finally, add a missing semicolon that results in no message being
printed in the missing openssl case. By not including the semicolon,
else echo "..." is treated as extra arguments to certctl, which is lax
in its argument parsing and ignores additional arguments, and the
semicolon and fi after the intended echo terminate the if statement as
normal so there's no syntax error at the shell level. This is harmless
as we weren't trying to do anything other than echo anyway, all that
happens is the echo doesn't actually get run.

Reported by: markj (missing semicolon)
Reviewed by: brooks, kevans
Obtained from: CheriBSD
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D35809

Makefile.inc1: Add a missing ${dist} to distributeworld for usr/include/i386

Not including ${dist} results in the following non-fatal error printed
once per extra distribution:

mkdir //usr/obj/usr/src/amd64.amd64/release/dist/usr/include/i386
mkdir: //usr/obj/usr/src/amd64.amd64/release/dist/usr/include: No such file or directory
*** Error code 1 (ignored)

Also fix a whitespace nit on this line whilst here.

Reviewed by: brooks
Fixes: a09ea2bbc305 ("amd64: add an i386 include directory")

installworld: improve portability of ldd use

b3b462229f97 added a case statement to ignore lines containing strings
in square brackets such as "[vdso]" and "[preloaded]". On MacOS
Monterey where /bin/sh may be zsh, this fails with:

/bin/sh: -c: line 0: syntax error near unexpected token `;;'

Invoke grep in the pipeline to remove such lines instead.

Reviewed by: emaste
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D35618

amd64: add an i386 include directory

This directory will hold i386-specific headers that are needed for
-m32 support on amd64 and where the amd64 and i386 cases have too
little in common for combining them to make sense. Files to be
installed will come in later commits.

With the currently required set of files, this could be done with
another INCGROUP in include/Makefile, but at least one file that
might want -m32 support (ieeefp.h) conflicts with a files installed
in /usr/include.

Reviewed by: jhb, imp

Merge llvm-project release/14.x llvmorg-14-init-18315-g190be5457c90

This updates llvm, clang, compiler-rt, libc++, libunwind, lld, lldb and
openmp to llvmorg-14-init-18315-g190be5457c90.

PR: 261742
MFC after: 2 weeks

build target triple variable from sys/conf/newvers.sh

Retrieve FreeBSD revision number directly from sys/conf/newvers.sh
when building the compiler target triple value, avoiding manual
intervention on other files every new release.

Reviewed by: imp
MFC after: 2 months
Sponsored by: Instituto de Pesquisas Eldorado (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D34429

bintrans: move files to a new directory

And reflect the change in various places.

Modularize uuencode and uudecode by wrapping them in bintrans.c

The program will be installed as bintrans, uuencode, uudecode,
b64encode, and b64decode and will be responsible for running the coders
according to their historical behavior.

Additionally, bintrans will be able to take a parameter designating
the coder and accept all its options in this form:
bintrans <coder> [options]
and the behavior should be the same as if
<coder> [options]
was invoked.
This has the advantage that adding coders won't require installing them
as binaries.

Move uudecode files to uuencode since the latter is the one that
provides the manual page.

Reviewed by: delphij (previous version)
Differential Revision: https://reviews.freebsd.org/D32943

Speed up *-old-* make targets by using sed instead of xargs

Targets like 'list-old-files' used "xargs -n1" to produce a list with
one file per line. Using xargs resulted in one fork+exec for each
Argument, resulting in rather long runtime. Instead, use sed to split
the list. On one machine `make list-old-files` took 30s wall clock time
with xargs and less than 1s with sed.

Reviewed by: jhb
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34741

installworld: handle ldd including preloaded objects

The installworld target makes a temporary copy of binaries to be used
during the install. Libraries that they depend on are also included,
found by using `ldd`.

After commit 0913953c9ed0 ldd started listing preloaded objects,
including [vdso], under a [preloaded] header. Skip ldd output that is
enclosed in square brackets.

Reviewed by: cy, kib [earlier version]
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34734

bootstrap: bump minimum supported version

Bump the minimum supported version to build -current from to 11.3R in
preparation of removing support for older systems. 11.4R was selected
as the most recent version to go out of support.

Sponsored by: Netflix
Reviewed by: delphij, emaste
Differential Revision: https://reviews.freebsd.org/D32444

bootstrap: No need for kbdcontrol bootstrap anymore

We only need kbdcontrol when bootstrapping from FreeBSD 10 or
pre-FreeBSD 11.0 current. Since we can no longer build from these
versions of FreeBSD, remove the support for bootstrapping them.

Sponsored by: Netflix

cleankernel: A target to delete the kernel compile file

With the meta-build, it's always a NO_CLEAN build. Provide a way to
remove so one can rebuild from scratch. 'cleankernel' will delete the
kernel and modules object directories. Document this in build(7).

Sponsored by: Netflix
Reviewed by: debdrup, markj
Differential Revision: https://reviews.freebsd.org/D32978

Makefile.inc1: synthesize PKG_ABI from newvers.sh variables

Previously we inspected ${WSTAGEDIR}/usr/bin/uname to determine PKG_ABI,
but the file will not exist in some cases - for example, if building
only kernel packages. We can instead synthesize the PKG_ABI from
information already provided by newvers.sh.

Reviewed by: kevans, manu (both earlier rev)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34249

Remove more quotes around Makefile .error/.warn/.info strings.

Reviewed by: imp, emaste
Differential Revision: https://reviews.freebsd.org/D34175

Stop trimming 'hf' suffixes from MACHINE_ARCH.

This was only used for MIPS hard-float architectures.

Reviewed by: imp
Sponsored by: The University of Cambridge, Google Inc.
Differential Revision: https://reviews.freebsd.org/D34018

Remove quotes around Makefile .error/.warn/.info strings

The text after .error et al is emitted verbatim.

Reviewed by: sjg
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33904

check/delete-old: Auto-generate lib32 entries for libraries.

- Add usr/lib32/libfoo.so.N for lib/libfoo.so.N.

- Add usr/lib32/foo for usr/lib/foo.

- Treat casper libraries special since they are installed to
/usr/lib32 instead of /usr/lib32/casper and thus map
usr/lib/casper/foo to usr/lib32/foo.

Note that OLD_DIRS and MOVED_LIBS entries are not duplicated, only
OLD_FILES and OLD_LIBS.

Reviewed by: imp, emaste
Sponsored by: The University of Cambridge, Google Inc.
Differential Revision: https://reviews.freebsd.org/D33324

Use MOVED_LIBS for libraries moved between /usr/lib and /lib.

Add a MOVED_LIBS variable similar to OLD_LIBS except that MOVED_LIBS
is used for the cases that a library's name doesn't change, but it
just moves between /usr/lib and /lib. This will be used by a future
change to auto-generate lib32 old files entries for which these cases
need to be ignored (a moved library remains in /usr/lib32).

Suggested by: emaste
Reviewed by: emaste
Sponsored by: The University of Cambridge, Google Inc.
Differential Revision: https://reviews.freebsd.org/D33848

Add list-old-{dirs,files,libs} targets.

These targets generate a raw list of the candidate old files roughly
corresponding to the values of OLD_DIRS, OLD_FILES, and OLD_LIBS.
Currently list-old-files also includes uncompressed manpages in
addition to compressed manpages.

Use these targets in the implementation of check-old-* and
delete-old-* to replace duplicated logic.

Reviewed by: imp, emaste
Sponsored by: The University of Cambridge, Google Inc.
Differential Revision: https://reviews.freebsd.org/D33327

Build libclang also if LLDB is enabled

LLDB depends on libclang as it uses Clang as the expression parser.
Previously setting WITHOUT_CLANG but leaving LLDB enabled (as default)
resulted in a build failure.

Users who set WITHOUT_CLANG in order to reduce build time or size
might want to set WITHOUT_LLDB in addition to WITHOUT_CLANG, or use
WITHOUT_TOOLCHAIN instead.

PR: 260993
Reported by: eugen
Reviewed by: dim
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

libsoft: Remove targets to generate libsoft

Remove the targets needed to make and install libsoft.

Sponsored by: Netflix

test-includes: Build a library instead of a binary

We need to build the .o's, but don't need to link. Build a static
library instead of a binary to accmoplish this. This removes the need to
have all the libc and crt stuff built, which is required for a
binary. In addition, trying to build a non-standard binary runs into
trouble with undefined symbols on arm related to EABI, even when using a
simplified startup with -nostdlib -e start.

Turn back on testing includes, now that it works.

Sponsored by: Netflix
Reviewed by: brooks, markj (prior version)
Differential Revision: https://reviews.freebsd.org/D33452

Remove test-includes build

This is making the CI red. I believe this is because we do a clean build
w/o metamode in the tinderbox, so none of the artifacts needed to build
a binary are present. However, I've not recreated the problem locally
yet to confirm. Remove this while I investigate. This partially reverts
dd55767b86bc. The rest of the commit causes no harm w/o the explicit
test here.

Sponsored by: Netflix

Test various header files to ensure they can be included by themselves.

A number of header files in sys/* have, going back to 7th Edition Unix
in 1979, reqiured other files (like sys/types.h) to compile. Likewise
the 4BSD networking code has had prerequisites. However, going back to
around the turn of the 21st century, other systems have made them be
independently include-able (wide-spread header include protection
post-dates 7th edition Unix by maybe 3 or so years judging from USENET
source postings). Start down the path of making them all independently
include-able by creating this test that fails buildworld when they are
not.

The file 'badfiles.inc' contains a list of the currently broken files
that cannot be included w/o any prerequisites. As files are fixed, 'make
badfiles.inc' should be re-run to remove them from the list. Note: All
files that start with an underscore are considered internal and not
tested.

Please note: once a file is removed from badfiles.inc, it must pass on
all architectures. Buildworld through at least the _includes target is
needed to ensure its working (though a buildkernel should also be done
on all architectures as well).

Sponsored by: Netflix
Reviewed by: brooks, markj
Differential Revision: https://reviews.freebsd.org/D32498

mips: Remove mips as a recognized build target.

Remove mips as a recognized build target in advance of its removal from
the tree.

Sponsored by: Netflix

pkgbase: Allow to specify PKG_ABI on the command line

This is useful for creating kernel package without having to buildworld
before as we use uname from the world stage dir to get the ABI.

Reviewed by: emaste, imp
Differential Revision: https://reviews.freebsd.org/D33323
Sponsored by: Beckhoff Automation GmbH & Co. KG

Makefile.inc1: Bootstrap libcrypt on non-FreeBSD for libroken

After f0a6ab51e6da, we need libroken's dependencies to be present; in
particular, libcrypt is not present on macOS, and so needs to be
bootstrapped. We include this on Linux too for consistency and to avoid
future compatibility issues, even if strictly unnecessary currently.

Fixes: f0a6ab51e6da ("src.libnames.mk: Include dependencies when bootstrapping from non-FreeBSD"
MFC after: 1 week

Bootstrap libz when cross-building from non-FreeBSD

This is needed now libdwarf depends on libz.

Fixes: dbf05458e3bd ("libdwarf: Support consumption of compressed ELF sections")
MFC after: 1 week

kern.pre.mk: allow to use and customize elfdump during kernel build

Reviewed by: emaste
Discussed with: jrtc27
Tested by: pho
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D32960

Makefile.inc1: Silence an error in make packages

Silence an error in "packages" target, like:

eval: /usr/src: Permission denied

on systems without subversion present.

Reviewed by: emaste
Approved by: emaste
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D33153

Hoist cddl prebuild lib dependency definitions out of a MK_ZFS block

The compilation of several libraries under cddl/lib is not conditional
on MK_ZFS = "yes", so their dependency on libspl is not conditional
either. Unbreak buildworld when WITHOUT_ZFS is set.

Reported by: bz
Fixes: 9e9c651caceb ("cddl: fix missing ZFS library dependencies")
MFC after: 1 week

Add ncurses/form to _prebuild_libs to fix build

Reported by: bz

Revert "Bootstrap: Prune building from pre-FreeBSD 11 support"

This reverts commit a420a672bc53bd3928626aa5d0ec5b41c0471356.

kevans pointed out some potential problems here, so reverting until
I can fix them.

Bootstrap: Prune building from pre-FreeBSD 11 support

We don't need to bootstrap lex or md4 anymore.
Cat doesn't need to be bootstrapped (but is needed for buildkernel)
cruncgen doesn't need to be bootstrapped at all.
kbdcontrol isn't needed

Sponsored by: Netflix
Pull Request: https://github.com/freebsd/freebsd-src/pull/554

Add warning that MIPS is being removed in FreeBSD 14.0

MFC After: 3 days
Sponsored by: Netflix
Reviewed by: brooks, jhb
Differential Revision: https://reviews.freebsd.org/D32853

cddl: fix missing ZFS library dependencies

In 9fae47666 zfsd got a libspl dependency to avoid undefined references.
However that workaround did not help external consumers of libzfs_core.

Fix all missing dependencies lld 13 and the rtld complain about.

Reviewed by: freqlabs, markj
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D32521

Remove arm/linux from sysent toplevel target

Sponsored by: The FreeBSD Foundation
Fixes: 65e485014b0dcf53e0

Makefile.inc1: Remove mentions of removed target "update"

This is follow-up to commits e290182bcf38 and 1f7d11e636ab.

Makefile.inc1: remove lorder from build/cross tools

As lorder is not used by the base system build there is no need to
include it in bootstrap-tools or NXBDIRS.

Fixes: 0e1e341b486c ("Stop using lorder and ranlib when...")
Sponsored by: The FreeBSD Foundation

native-xtools: avoid libllvm while populating the sysroot

Prior to 021385aba562, MK_CLANG=no was sufficient to avoid descending
into lib/clang, but the referenced change added a couple of other
enabling knobs. Turn those off, too, to continue avoiding libllvm.

With this change, we no longer end up with a libllvm using the wrong
default target triple; `poudriere jail -cx` works once again.

Reported by: bhughes, imp, probably others
Fixes: 021385aba562 ("Add WITH_LLVM_BINUTILS to install LLVM ...")

Revert "bootstrap: No need to disable shared libraries for bootstrap tools"

This reverts commit e8f26e5dc859a8fbc663e108421f657325ca3054.

Although the change worked locally, it's breaking something in the CI
build for the riscv64 build (which makes no sense it would only break
that since we're building host tools to bootstrap at that point).

Sponsored by: Netflix

bootstrap: No need to disable shared libraries for bootstrap tools

There's no need to disable shared libraries when building the bootstrap
tools. This was added on 2000 (commit ad879ce9552c) when the perl
bootstrap was added (libperl and miniperl) and saved a fair amount of
time (perl took a long time to build on 2000-era hardware).

For many years now, however, we rarely build any libraries when
bootstrapping. Even when we do, the optimization saves at most a few
seconds when upgrading since the libraries built have been small. Shared
libraries are more robust accross versions that static libraries due to
creaping dependencies (we aren't crossing versions of share libraries,
though, just using what's on the host). In addition, linux and macos
have been building like this for some time because static binaries on
those systems are difficult to impossible.

Sponsored by: Netflix
Reviewed by: arichardson, bapt
Differential Revision: https://reviews.freebsd.org/D32443

build: slap a libtinfo dependency on libncurses

This mirrors the SUBDIR_DEPEND in lib/ncurses/Makefile.

Reported by: jenkins (e.g., riscv64 build #23984)
Fixes: 396851c20aeb ("ncurses: split libtinfo from libncurses")

ncurses: chase dependency changes in the source tree

Differential Revision: https://reviews.freebsd.org/D32098

Drop cloudabi

According to https://github.com/NuxiNL/cloudlibc:
CloudABI is no longer being maintained. It was an awesome experiment,
but it never got enough traction to be sustainable.

There is no reason to keep it in FreeBSD.

Approved by: ed (private mail)
Reviewed by: emaste
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D31923

Fix spelling of BUILDENV_SHELL

Add WITH_LLVM_BINUTILS to install LLVM binutils instead of Elftoolchain

When WITH_LLVM_BINUTILS is set, we will install the LLVM binutils as
ar/ranlib/nm/objcopy/etc. instead of the elftoolchain ones.
Having the LLVM binutils instead of the elftoolchain ones allows us to use
features such as LTO that depend on binutils that understand LLVM IR.
Another benefit will be an improved user-experience when compiling with
AddressSanitizer, since ASAN does not symbolize backtraces correctly if
addr2line is elftoolchain addr2line instead of llvm-symbolizer.
See https://lists.freebsd.org/archives/freebsd-toolchain/2021-July/000062.html
for more details.

This is currently off by default but will be turned on by default at some
point in the near future.

Reviewed By: emaste

Differential Revision: https://reviews.freebsd.org/D31060

Fix bootstrapping to actually build lldb-tblgen for later use

Because MK_LLDB=no is in BSARGS, the bootstrap-tools recursive make does
not add lldb-tblgen to _clang_tblgen, causing it to not be built. This
means that the build currently always uses the host's lldb-tblgen
(which, whilst currently it appears to work, could in future break if
TableGen backends are added or altered) and, if it doesn't exist (either
because the current FreeBSD system was built with it disabled, or you're
building on macOS/Linux), fails. Linux and macOS cross-builds used to
work simply because LLDB was previously in BROKEN_OPTIONS when building
on non-FreeBSD.

Instead, move MK_LLDB=no from BSARGS to XMAKE. This ensures that the
lib/clang build in cross-tools continues to not build LLDB parts for the
bootstrap toolchain (both to save time/space on FreeBSD, and because our
vendored LLDB does not include the macOS and Linux host files so those
would fail to build).

The DIRDEPS target is updated to move MK_LLDB=no from the BSARGS block
that mirrors Makefile.inc1 to the line that disables additional
toolchain components. The DIRDEPS build likely suffers from the same
issue currently, but having never used it and not being familiar with
how it works I am leaving that as-is. If it does suffer from the same
issue it should be easily reproducible by renaming /usr/bin/lldb-tblgen
or moving it to a directory not in PATH.

Fixes: 31ba4ce8898f ("Allow bootstrapping llvm-tblgen on macOS and Linux")
MFC after: 1 week
Reviewed by: dim, emaste, imp
Differential Revision: https://reviews.freebsd.org/D31531

Makefile.inc1: Make sure sub-makes see MK_CLANG_BOOTSTRAP=no when XCC is a path

Currently we override MK_CLANG_BOOTSTRAP to no so we don't build a
bootstrap compiler, but subdirectories don't see that and so the hack in
bsd.sys.mk to prefer our includes over Clang's resource dir for external
toolchains is not enabled unless you use -DWITHOUT_CLANG_BOOTSTRAP
explicitly on top of XCC (which tools/build/make.py does not do),
causing duplicate definition errors when building rtld-elf due to the
use of -ffreestanding (Clang's stdint.h will use the system one when
hosted, but its own when freestanding, and only has glibc's preprocessor
guards, not FreeBSD's).

This broke when dropping CLANG_BOOTSTRAP from BROKEN_OPTIONS.

Fixes: 31ba4ce8898f ("Allow bootstrapping llvm-tblgen on macOS and Linux")
MFC after: 1 week
Reviewed by: imp, arichardson
Differential Revision: https://reviews.freebsd.org/D31529

localedef: unbreak WITHOUT_LOCALES

After 0fa5403d493b ("pkgbase: move locales into their own package") we
need usr.bin/localedef as a bootstrap tool independent on where
WITHOUT_LOCALE was specified as we ALWAYS process C.UTF-8.

At the same time LOCALES= in the local Makefile is empty but
C.UTF-8 with WITHOUT_LOCALES. C.UTF-8 is excluded from FILES, and thus
after the replacement FILES= is set to only .LC_CTYPE which results in
a build failure not knowing how to build that. Tweak the substitution to
replace only non-empty words so that FILES remains harmlessly empty.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D31589

bootstrap-tools: Don't copy host tools with make -n

_worldtmp: Fix some sub-makes not working with make -n

Add missing lex/yacc dependency for mkesdb/mkcsmapper bootstrap

This causes build failures on macOS where the build can end up invoking
an incompatible m4 binary.

Fxies: 2de949cf85d0 ("Remove mkcsmapper_static and mkesdb_static from build-tools")

Makefile.inc1: Avoid hanging if pkg is not installed

For `pkg --version`, Redirect stdin from /dev/null to avoid waiting on
/usr/sbin/pkg's bootstrap prompt if the pkg package is not installed.
Also redirect stderr to /dev/null to discard the warning message in
this case.

Reported by: mjg
Fixes: 4e224e4be7c3 ("pkgbase: accommodate pkg < 1.17")
Sponsored by: The FreeBSD Foundation

Prefer MK_SSP=no to SSP_CFLAGS=

It is more idiomatic. CFLAGS is only augmented with $SSP_CFLAGS when
$MK_SSP != "no".

Reviewed by: imp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D31401

mk: format some option lists

Alphabetize and give each option its own line, ahead of making another
change to these lists. This makes future diffs easier to read.

Reviewed by: imp, emaste
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D31399

Fix native-xtools build

Fixes https://github.com/freebsd/poudriere/issues/894
Fixes: d0c737e18 ("Makefile: Fix MAKEOBJDIRPREFIX command-line")
X-MFC-With: d0c737e18

Revert "Fix native-xtools build"

This reverts commit 36269b82318280ef184c953b90674f5905e0f53f.

This had an unintended change included.

Fix native-xtools build

Fixes https://github.com/freebsd/poudriere/issues/894
Fixes: d0c737e18 ("Makefile: Fix MAKEOBJDIRPREFIX command-line")
X-MFC-With: d0c737e18

Remove mkcsmapper_static and mkesdb_static from build-tools

Build them as part of the bootstrap-tools phase instead.

Reviewed by: emaste (no objections)
Differential Revision: https://reviews.freebsd.org/D28181

Add build system support for ASAN+UBSAN instrumentation

This adds two new options WITH_ASAN/WITH_UBSAN that can be set to
enable instrumentation of all binaries with AddressSanitizer and/or
UndefinedBehaviourSanitizer. This current patch is almost sufficient
to get a complete buildworld with sanitizer instrumentation but in
order to actually build and boot a system it depends on a few more
follow-up commits.

Reviewed By: brooks, kib, markj
Differential Revision: https://reviews.freebsd.org/D31043

add the time(1) command to the list of install tools

Reported by: dhw
Approved by: dhw
Differential Revision: https://reviews.freebsd.org/D31373

`make buildworld' with time logging for each stage

PR: 257141
Reviewed by: sjg,emaste
Approved by: emaste
Differential Revision: https://reviews.freebsd.org/D31154

pkgbase: improve pkg --version parsing

In some cases `pkg --version` might produce unexpected or additional
output. Use a regex /^[0-9.]+$/ to match only the line containing the
version number.

Reported by: Michael Butler on freebsd-current@
Fixes: 4e224e4be7c3 ("pkgbase: accommodate pkg < 1.17")
Sponsored by: The FreeBSD Foundation

pkgbase: accommodate pkg < 1.17

6cafdee71d2b adapted the pkgbase build for 1.17, but broke Cirrus-CI's
use of PKG_FORMAT=tar (the quarterly package set still has pkg 1.16).
Because of this I disabled the pkgbase build and test in 2bfba2a04b05.

Now, check `pkg --version` and use the old logic for < 1.17.

To be reverted once we no longer encounter pkg 1.16 in Cirrus-CI (i.e.,
via GCP cloud images) to avoid keeping this extra complexity around.

PR: 257422
Reviewed by: manu
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31324

pkgbase: Track pkg 1.17

With pkg 1.17 the extention is always .pkg, packing_format is also now deprecated.

Don't build lldb-tablegen if MK_LLDB=no

Split out from D31060.

Reviewed by: emaste

Remove svnlite.

Reviewed by: bcr, imp, emaste
Differential Revision: https://reviews.freebsd.org/D30737

Remove 'make update'.

In the CVS days this used be a wrapper around either CVS or CVSup and
used to support updating src, doc, and ports checkouts. With the move
to subversion this only supported updating src and was itself a
wrapper around 'svn update'. With Git, users are probably better off
using appropriate Git commands directly to update without needing an
explicit make target as a wrapper.

Reviewed by: bcr, imp, emaste
Differential Revision: https://reviews.freebsd.org/D30736

libpmc: always generate libpmc_events.c

The jevents build tool will create an empty table if it doesn't find any
events, so we can remove the extra $MACHINE_CPUARCH checks.

Reviewed by: gnn, ray, emaste
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D30531

Add elfctl to toolchain.

Add elfctl program to toolchain to allow modifying ELF feature flags
during system build.

Submitted by: Dawid Gorecki <dgr@semihalf.com>
Obtained from: Semihalf
Sponsored by: Stormshield
Differential Revision: https://reviews.freebsd.org/D29552

Makefile.inc1: unbreak bootstrap when kbdcontrol does not exist

Reviewed by: arichardson
MFC after: 12 hours
MFC target: stable/13, releng/13.0
Differential Review: https://reviews.freebsd.org/D29200
Sponsored by: Rubicon Communications, LLC ("Netgate")

Revert "bootstrap: add tic to the bootstrap tools"
Since we don't build and install the terminfo db anymore this is not
needed

This reverts commit b6a51d39e3a2e2f75d5b42a8c17a531063258a15.
This reverts commit e5e4845959ac60110677df302a68c220dff75abe.
This reverts commit 0af562d7e1850bbef230d30805c101b26588a3ed.

Makefile.inc1 "Avoid duplicate script for target" warning

This happens if -DBOOTSTRAP_ALL_TOOLS or when building on non-FreeBSD.

bootstrap: fix a last edit typo

Reported by: cy

bootstrap: add tic to the bootstrap tools

tic is necessary early in the build to be able to build the terminfo
database later on.

Tested by: manu, jbeich

Remove the MK_LIBCPLUSPLUS option

This option has been equivalent to any form of C++ support since libstdc++
was removed. Therefore, replace all MK_LIBCPLUSPLUS uses with MK_CXX.

Reviewed By: emaste
Differential Revision: https://reviews.freebsd.org/D27974

build: remove LIBPTHREAD/LIBTHR build options

WITHOUT_LIBTHR has been broken for a little over five years now, since the
xz 5.2.0 update introduced a hard liblzma dependency on libthr, and building
a useful system without threading support is becoming increasingly more
difficult.

Additionally, in the five plus years that it's been broken more reverse
dependencies have cropped up in libzstd, libsqlite3, and libcrypto (among
others) that make it more and more difficult to reconcile the effort needed
to fix these options.

Remove the broken options.

PR: 252760
Reviewed by: brooks, emaste, kib
Differential Revision: https://reviews.freebsd.org/D28263

Bump CURRENT to 14.0

This one goes to 14.

Approved by: re (implicit)
Sponsored by: Rubicon Communications, LLC ("Netgate")

pkgbase: allow update-packages for first-run of packaging

If ${REPODIR}/${PKG_ABI} does not exist when we begin real-update-packages,
skip the comparison with the non-existent previous repository and just
finish the repo off. This allows external scripts to just assume they can
run `update-packages` rather than figuring out if they'd previously run
`packages` for this Version/Arch combo.

PKG_VERSION_FROM_DIR was added so that we could perhaps detect the three
distinct cases:

1.) If the repo has not yet been created, PKG_VERSION_FROM_DIR will be
empty.
2.) If the repo is in some intermediate state between created and fully
initialized, PKG_VERSION_FROM_DIR may point to the ABI directory.
3.) If the repo is fully initialized, then PKG_VERSION_FROM_DIR points to
the latest build to compare to.

Option #2 is explicitly unhandled at the moment, but this is no different
than it was before.

Reviewed-by: manu
Differential-Revision: https://reviews.freebsd.org/D28229

pkgbase: differentiate package versions for ALPHA/BETA/PRERELEASE/RC phases

The current postfix conversions are:

CURRENT / STABLE / PRERELEASE, 12.x-CURRENT becomes 12.snapYYYYMMDDhhmmss
ALPHAx -> .ax, so 11.3-ALPHA1 becomes 11.3.a1.YYYYMMDDhhmmss
BETAx -> .bx, so 12.1-BETA2 becomes 12.1.b2.YYYYMMDDhhmmss
RCx -> .rcx, so 13.0-RC3 becomes 13.0.rc3.YYYYMMDDhhmmss
RELEASE -> (nothing), so 12.1-RELEASE becomes 12.1
RELEASE-pX -> pX, so 12.1-RELEASE-p1 becomes 12.1p1

Note that for development branches we will start to drop the minor version
component entirely, which more closely matches how these branches are
physically named (stable/NN).

snap is a new prefix that was added to pkg in [0], which is simply a more
verbose version of the current ".s" used.

As noted, build timestamps are also added to ALPHA/BETA/RC versions. This
is largely irrelevant for re@ snapshots because they will only produce one
set of snapshots for each alpha/beta/rc, but external folks may produce
multiple in that timeframe -- at least for alpha. For them, it is
imperative that the builds have a differentiating characteristic like this
rather than multiple builds across multiple revisions being versioned
identically.

[0] https://github.com/freebsd/pkg/pull/1929

Reviewed by: gjb, manu
Submitted by: rene (original, original version)
Differential Revision: https://reviews.freebsd.org/D28167

pkgbase: limit PKG_VERSION_FROM calculation to real-update-packages

PKG_ABI is defined in some other targets that do not need to shell out and
calculate PKG_VERSION_FROM. Moreover, it produces extra errors when
bootstrapping an initial pkgbase repo, as the /latest link doesn't exist
yet.

pkgbase: collapse -PRERELEASE into STABLE/CURRENT case for version suffix

-PRERELEASE exists as a given stable/X branch while the next minor version
release is in progress. From a functional standpoint, it should be treated
as -STABLE since it'll typically be a superset of what's included in the
concurrent releng branch.

Reviewed by: gjb
Differential Revision: https://reviews.freebsd.org/D28166

pkgbase: default to installing all specified kernels

These only get staged off to the side, not installed into the system. Let's
change the default for packages/update-packages to be less surprising.

MFC after: 1 week
Reviewed by: manu, meena <me igalic co>
Differential Revision: https://reviews.freebsd.org/D28190

Revert "pkgbase: differentiate package versions for ALPHA/BETA/PRERELEASE/RC phases"

This reverts commit aef769614f921660cb0262412426034cf5395ae5.

While the package names are a bit prettier this confuse pkg about upgrading :
$ pkg version -t 13.0.s2021011313063 13.0.a1
>
$ pkg version -t 13.0.s2021011313063 13.0_ALPHA1
<

Note that the current scheme isn't good when bumping from ALPHA to BETA or
even BETA to RC:
$ pkg version -t 13.0_ALPHA1 13.0_BETA1
=
$ pkg version -t 13.0_BETA1 13.0_RC1
=

But more thoughts have to be put into this renaming.

pkgbase: differentiate package versions for ALPHA/BETA/PRERELEASE/RC phases

The current postfix conversions are:

CURRENT / STABLE -> .sYYYYMMDDhhmmss
ALPHAx -> .ax, so 11.3-ALPHA1 becomes 11.3.a1
BETAx -> .bx, so 12.1-BETA2 becomes 12.1.b2
RCx -> .rcx, so 13.0-RC3 becomes 13.0.rc3
PRERELEASE -> .p, so 11.3-PRERELEASE becomes 11.3.p
RELEASE -> (nothing), so 12.1-RELEASE becomes 12.1

Submitted by: rene (original version)
Differential Revision: https://reviews.freebsd.org/D20734

Makefile.inc1: Avoid using release/Makefile for VERSION.

release/Makefile.inc1 has git executions that were being ran for each of
these lookups. The results were not needed so just lookup what we want
directly instead.

Reviewed by: gjb, rlibby, emaste (maybe)
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D27643

Rename NO_WERROR -> MK_WERROR=no

As suggested in D27598. This also supports MK_WERROR.clang=no and
MK_WERROR.gcc=no to support the existing NO_WERROR.<compiler> uses.

Reviewed By: brooks
Differential Revision: https://reviews.freebsd.org/D27601

Stop passing MK_WARNS=no for bootstrap tools

I got annoyed by the number of warnings that the CheriBSD build was
emitting. It turns out that we are emitting lots of warnings during
bootstrap because bootstrap tools are built with the default compiler
flags and ignore the warnings flags that are set in bsd.sys.mk.

Looking at git blame, it appears that MK_WARNS=no has been passed since
rS112869, replacing the -DNO_WERROR option that was added in rS87775.

This commit changes MK_WARNS=no back to -DNO_WERROR. We need to pass
-DNO_WERROR, since the system compiler might have new warnings that we
don't know about yet, and we shouldn't fail the build in that case.

Reviewed By: imp, brooks
Differential Revision: https://reviews.freebsd.org/D27598

Add mkimg to the list of bootstrap tools

Having this as part of the bootstrap tools is useful to build disk images
to boot in QEMU (especially when building on a Linux/macOS host where
mkimg is not available). We have been bootstrapping mkimg in CheriBSD for
a long time (using LOCAL_XTOOL_DIRS) but I believe this is also useful
upstream.

Reviewed By: emaste, brooks
Differential Revision: https://reviews.freebsd.org/D27602

vtfontcvt is part of bootstrap toolset and should always be built

Reported by: Peter Jeremy

ncurses: only keep the version with widechar support

Only keep the widechar version of ncurses as libncursesw.so.9

Keep the old name to avoid breaking the ABI compatibility (the non
widechar version libncurses.so.9 is not binary compatible with
libncursesw.so.9) since all ports and base are already only linking
against the widechar version we can simply remove libncurses.so.9

Since the .9 version only lived in the dev branch and never ended in a
release, it is simply removed and not added to any binary compat
package.

Add symlinks to keep build time compatibility for anyone linking against
-lncurses

pkgbase: rename PKGSIGNKEY to PKG_REPO_SIGNING_KEY

this way, it'll be automatically picked up by poudriere
That's quite handy when building pkgbase!

Submitted by: Mina Galić <me@igalic.co>
Reviewed By: bapt, bcr (manpages)
Differential Revision: https://reviews.freebsd.org/D27690

Remove additional GDB leftovers missed in r368667

crunchgen: fix NULL-deref bug introduced in r364647

While porting over the local changes from CheriBSD for upstreaming, I
accidentally committed a broken version of find_entry_point(): we have to
return NULL if the value is not found instead of a value with
ep->name == NULL, since the checks in main were changed to check ep instead
of ep->name for NULL.

This only matters if the crunched tool cannot be found using normal lookup
and one of the fallback paths is used, so it's unlikely to be triggered
in rescue. However, I noticed that one of our CheriBSD test scripts was
failing to run commands under `su` on minimal disk images where all
binaries are hardlinks to a `cheribsdbox` tool generated with crunchgen.

This also updates the bootstrapping check in Makefile.inc1 to bootstrap
crunchgen up to the next version bump.

Reviewed By: kevans
Differential Revision: https://reviews.freebsd.org/D27474

Add lists for customizing legacy and bootstrap-tools.

Reviewed by: arichardson
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D27200

[POWERPC64LE,POWEPCSPE] set default kernel config for powerpc64le and powerpcspe variants

Default KERNCONF for powerpc64le should be GENERIC64, and powerpcspe should
select MPC85XXSPE

Reviewed by: bdragon,emaste
Sponsored by: Eldorado Research Institute (eldorado.org.br)
Differential Revision: https://reviews.freebsd.org/D27257

Makefile.inc1: remove no-longer required variable

This variable is unsed since r364760 but I forgot to delete it in that commit.

Reported By: bdrewery

fix the docs, this was always wrong... In some cases, DISTDIR is set
automatically by tools via /etc/make.conf, so remind people (me) where
to find where it's set..

It would be nice for someone to document what DISTDIR is better than:
where the file for a distribution gets installed

pkgbase: Add incremental packages

This adds a new target update-packages which will create the new packages
compared to the last run.

This is how to use it:
At this point we cut a release
$ make buildworld ...
$ make buildkernel
$ make packages

There is now a PKG_VERSION directory with latest link pointing to it
Distribute the packages to server

$ something something that update the source tree
$ make buildworld ...
$ make buildkernel
$ make update-packages
You know have a PKG_VERSION directory in the REPODIR and latest link pointing to it.
In PKG_VERSION dir only the packages which differs from the latest run are
named PKG_VERSION, otherwise the old packages are there.

The process is :
Build the new packages in the PKG_VERSION directory
Compare the internal data with the PKG_VERSION_FROM version. The comparison is done
by checking the internal hash of the packages.
By default PKG_VERSION_FROM is set to what the latest link points to.
If the old and new version matches, we rm the new package and cp the old one.

Differential Revision: https://reviews.freebsd.org/D25984

flua: Add a libjail module

libjail is pretty small, so it makes for a good proof of concept demonstrating
how a system library can be wrapped to create a loadable Lua module for flua.

* Introduce 3lua section for man pages
* Add libjail module

Reviewed by: kevans, manpages
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D26080

Significantly speed up mkimg_test

It turns out that the majority of the test time for the mkimg tests isn't
mkimg itself but rather the use of jot and hexdump which can be quite slow
on emulated platforms such as QEMU.

On QEMU-RISC-V this reduces the time for `kyua test mkimg_test` from 655
seconds to 200. And for CheriBSD on QEMU-CHERI this saves 4-5 hours (25%
of the time for the entire testsuite!) since jot ends up triggering slow
functions inside the QEMU emulation a lot.

Reviewed By: lwhsu
Differential Revision: https://reviews.freebsd.org/D26796

Fix build with -DBOOTSTRAP_ALL_TOOLS

sbin/sysctl can no longer be bootstrapped on FreeBSD 12 after r366465,
so create a symlink to the host tool instead of trying to build it.

Makefile.inc1: sysent: allow subordinate sysent targets to run in parallel

makesyscalls.lua (and indeed makesyscalls.sh) are both safe to be run in
parallel, so let's do it.

This is a trivial difference because runtime per-target is pretty small, but
I like seeing it run in parallel when my muscle memory types `make -sj4`.

Reviewed by: brooks, emaste
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D26594

[PowerPC64LE] Set up powerpc.powerpc64le architecture

This is the initial set up for PowerPC64LE.

The current plan is for this arch to remain experimental for FreeBSD 13.

This started as a weekend learning project for me and kinda snowballed from
there.

(More to follow momentarily.)

Reviewed by: imp (earlier version), emaste
Sponsored by: Tag1 Consulting, Inc.
Differential Revision: https://reviews.freebsd.org/D26399

Prefer bootstrapped tools when running certctl.sh

Otherwise we get lots of warnings when building on Linux/macOS during
installworld:
Scanning /local/scratch/alr48/cheri/output/freebsd-x86/usr/share/certs/blacklisted for certificates...
install: invalid option -- 'U'
Try 'install --help' for more information.
install: invalid option -- 'U'
....

Reviewed By: kevans
Differential Revision: https://reviews.freebsd.org/D26481

loader: zfs should support bootonce an nextboot

bootonce feature is temporary, one time boot, activated by
"bectl activate -t BE", "bectl activate -T BE" will reset the bootonce flag.

By default, the bootonce setting is reset on attempt to boot and the next
boot will use previously active BE.

By setting zfs_bootonce_activate="YES" in rc.conf, the bootonce BE will
be set permanently active.

bootonce dataset name is recorded in boot pool labels, bootenv area.

in case of nextboot, the nextboot_enable boolean variable is recorded in
freebsd:nvstore nvlist, also stored in boot pool label bootenv area.
On boot, the loader will process /boot/nextboot.conf if nextboot_enable
is "YES", and will set nextboot_enable to "NO", preventing /boot/nextboot.conf
processing on next boot.

bootonce and nextboot features are usable in both UEFI and BIOS boot.

To use bootonce/nextboot features, the boot loader needs to be updated on disk;
if loader.efi is stored on ESP, then ESP needs to be updated and
for BIOS boot, stage2 (zfsboot or gptzfsboot) needs to be updated
(gpart or other tools).

At this time, only lua loader is updated.

Sponsored by: Netflix, Klara Inc.
Differential Revision: https://reviews.freebsd.org/D25512

build: provide a default WARNS for all in-tree builds

The current default is provided in various Makefile.inc in some top-level
directories and covers a good portion of the tree, but doesn't cover parts
of the build a little deeper (e.g. libcasper).

Provide a default in src.sys.mk and set WARNS to it in bsd.sys.mk if that
variable is defined. This lets us relatively cleanly provide a default WARNS
no matter where you're building in the src tree without breaking things
outside of the tree.

Crunchgen has been updated as a bootstrap tool to work on this change
because it needs r365605 at a minimum to succeed. The cleanup necessary to
successfully walk over this change on WITHOUT_CLEAN builds has been added.

There is a supplemental project to this to list all of the warnings that are
encountered when the environment has WARNS=6 NO_WERROR=yes:
https://warns.kevans.dev -- this project will hopefully eventually go away
in favor of CI doing a much better job than it.

Reviewed by: emaste, brooks, ngie (all earlier version)
Reviewed by: emaste, arichardson (depend-cleanup.sh change)
Differential Revision: https://reviews.freebsd.org/D26455

Promote the installworld `certctl rehash` to distributeworld

Contrary to my belief, installworld is not sufficient for getting certs
installed into VM images. Promote the rehash to both installworld and
distributeworld (notably: not stageworld) and rehash the base distdir so we
end up with /etc/ssl/certs populated in the base dist archive. A future
commit will remove the rehash from bsdinstall, which doesn't really need to
happen if they're installed into base.txz.

While here, fix a minor typo: s/CERTCLTFLAGS/CERTCTLFLAGS/

MFC after: 1 week

installworld: run `certctl rehash` after installation completes

This was originally introduced back in r360833, and subsequently reverted
because it was broken for -DNO_ROOT builds and it may not have been the
correct place for it.

While debatably this may still not be 'the correct place,' it's much cleaner
than scattering rehashes all throughout the tree. brooks has fixed the issue
with -DNO_ROOT by properly writing to the METALOG in r361397.

Do note that this is different than what was originally committed; brooks
had revisions in D24932 that made it actually use the revised unprivileged
mode and write to METALOG, along with being a little more friendly to
foreign crossbuilds and just using the certctl in-tree.

With this change, I believe we should now have a populated /etc/ssl/certs in
the VM images.

MFC after: 1 week

Makefile.inc1: remove more old stale depend hacks

Current stale dependency hacks are in tools/build/depend-cleanup.sh.
These ones were almost a year old; remove them from Makefile.inc1.

Ensure that the makewhatis symlink is added in the bootstrap-tools stage

We currently set MK_MAN=no in $BSARGS so MK_MAN_UTILS will also be false
which means that the makewhatis symlink will not be created.
This change fixes the build when using both -DBUILD_WITH_STRICT_TMPPATH and
-DBOOTSTRAP_ALL_TOOLS.

Tested by: andrew
Differential Revision: https://reviews.freebsd.org/D16761

Add WITH_/WITHOUT_CLEAN option to replace NO_CLEAN

This allows use of the standard src.conf configuration for controlling
whether the tree is cleaned before build or not. The default is still
to clean.

Setting either NOCLEAN or NO_CLEAN will mention the new src.conf option.
NOCLEAN remains a .warning, while for now NO_CLEAN is .info.

Reviewed by: bdrewery (earlier version)
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D22762

Makefile.inc1: comment .endif to ease finding matching .if

MFH

Sponsored by: Rubicon Communications, LLC (netgate.com)

libzfs: Also add the crypto dependency to Makefile.inc1

Reported by: kevans
Discussed with: kevans
Sponsored by: iXsystems, Inc.

Use bootstrapped install(1) install of tools/install.sh in world stage

This should be noticeably faster due to fewer processes being forked and
also handles other flags such as -S or writing to METALOG.

Reviewed By: brooks
Differential Revision: https://reviews.freebsd.org/D26039

Fix running the builddtb target on a noexec file system

Obtained from: CheriBSD

Add necessary Makefile.inc1 infrastructure for building on non-FreeBSD

The most awkward bit in this patch is the bootstrapping of m4:
We can't simply use the host version of m4 since that is not compatible
with the flags passed by lex (at least on macOS, possibly also on Linux).
Therefore we need to bootstrap m4, but lex needs m4 to build and m4 also
depends on lex (which needs m4 to generate any files). To work around this
cyclic dependency we can build a bootstrap version of m4 (with pre-generated
files) then use that to build the real m4.

This patch also changes the xz/unxz/dd tools to always use the host version
since the version in the source tree cannot easily be bootstrapped on macOS
or Linux.

Reviewed By: brooks, imp (earlier version)
Differential Revision: https://reviews.freebsd.org/D25992

Merge OpenZFS support in to HEAD.

The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.

I would advise against doing 'zpool upgrade'
or creating indispensable pools using new
features until this change has had a month+
to soak.

Work on merging FreeBSD support in to what was
at the time "ZFS on Linux" began in August 2018.
I first publicly proposed transitioning FreeBSD
to (new) OpenZFS on December 18th, 2018. FreeBSD
support in OpenZFS was finally completed in December
2019. A CFT for downstreaming OpenZFS support in
to FreeBSD was first issued on July 8th. All issues
that were reported have been addressed or, for
a couple of less critical matters there are
pull requests in progress with OpenZFS. iXsystems
has tested and dogfooded extensively internally.
The TrueNAS 12 release is based on OpenZFS with
some additional features that have not yet made
it upstream.

Improvements include:
project quotas, encrypted datasets,
allocation classes, vectorized raidz,
vectorized checksums, various command line
improvements, zstd compression.

Thanks to those who have helped along the way:
Ryan Moeller, Allan Jude, Zack Welch, and many
others.

Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D25872

When copying over the binaries, use '-p' to preserve date/time

Although I can't reproduce it, others are seeing different lex/yacc
programs always regenerated after my change to copy rather than
symlink the files. The reported fix is to add '-p' to the copies.
Since it doesn't hurt, go head and add it, though the reasons for
this mattering remain at best obscure and poorly articulated.

Also print number of available CPUs on Linux

Without this change the buildworld/buildkernel epilogue looks like this:
>>> World built in 249 seconds, sysctl: cannot stat /proc/sys/hw/ncpu: No such file or directory
ncpu: , make -j72.

Reviewed By: emaste, bdrewery
Differential Revision: https://reviews.freebsd.org/D26056

Pass the installworld install(1) flags to make buildenv

This ensure that running make install inside buildenv correctly includes
the METALOG flags when building with -DNO_ROOT.

Reviewed By: brooks
Differential Revision: https://reviews.freebsd.org/D26038

After the clang/llvm version 11 import LLD_VERSION is no longer used
upstream so Version.inc now only defines LLD_VERSION_STRING.

This breaks the WANT_LINKER_VERSION magic and might lead to us building
more than needed (e.g., for croos-tools).

Change the awk script to parse LLD_VERSION_STRING instead of LLD_VERSION,
which not only unbreaks the current situation but should also be backwards
compatible as dim points out.

PR: 248818
Reviewed by: emaste, dim (seems right and the way to go)
MFC after: 4 weeks
X-MFC before: 364284

MFH

Sponsored by: Rubicon Communications, LLC (netgate.com)

Fix -DBUILD_WITH_STRICT_TMPPATH dtrace builds

Some of the scripts used for libdtrace invoke nawk instead of awk
(for example cddl/contrib/opensolaris/lib/libdtrace/common/mknames.sh).
When bootstrapping all tools, we get the nawk -> awk link while building
usr.bin/awk, but when linking/copying the dependencies from the host we
were only adding awk but not nawk.

This was silently generating invalid files when building libdtrace with
BUILD_WITH_STRICT_TMPPATH=1 since those scripts invoke nawk instead of
awk. In addition to adding the missing link this commit also adds
set -e to those scripts to catch errors like this in the future.

Reviewed By: markj, emaste
Differential Revision: https://reviews.freebsd.org/D26025

Allow overriding the tool used for stripping binaries

Since the make variable STRIP is already used for other purposes, this
uses STRIPBIN (which is also used for the same purpose by install(1).
This allows using LLVM objcopy to strip binaries instead of the in-tree
elftoolchain objcopy. We make use of this in CheriBSD since passing
binaries generated by our toolchain to elftoolchain strip sometimes results
in assertion failures.

This allows working around https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248516
by specifying STRIPBIN=/path/to/llvm-strip

Obtained from: CheriBSD
Reviewed By: emaste, brooks
Differential Revision: https://reviews.freebsd.org/D25988

pkgbase: Replace / with | for PKG_WWW

PKG_WWW contain / char, replace the delimiter by a '|'.

Reported by: 0mp

pkgbase: Add PKG_NAME_PREFIX, PKG_MAINTAINER and PKG_WWW

This is useful for downstream users to customize the packages.
Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D26019

The practice of creating symbolic links is somewhat fragile. Always
make copies instead.

There's too many times that we can't run the new binaries with old
libraries. Making the links when things are known to be 'safe' is a
nice optimization, but a copy of all the binaries is only 30MB, so
saving the copies at the cost of increased support when new symbols
are added and used as part of the bootstrap seems to be unwise.

There may be additional optimizations possible here, especially for
!FreeBSD hosts. However, that's beyond the scope of the problem I'm
trying to fix with make failing mid-way through an installworld across
change r363679. This optimization there caused us to run a new binary
with an old library once a new make was installed due to the symbolic
link. One could just copy make, but then other binaries fail as well,
so rather than play whack-a-mole, I opted to take us back to the old
way. Before r340157 or so we did copies (thogh of a lot fewer
artifacts), and we didn't have issues like this.

Reviewed by: arichards@
Differential Revision: https://reviews.freebsd.org/D25967

Fix duplicate assignment of _localedef in Makefile.inc1

The same .if exists a few lines below.

pkgbase: Add the possibility to choose the output dir

The output dir is set to ${REPODIR}/${PKG_ABI}/${PKG_VERSION} now.
Add the possibility to specify the last componant and set it by default
to ${PKG_VERSION} as before.
This is useful for tests and also for building packages with the same
PKG_VERSION provided to check differences.

pkgbase: Add a new variable PKG_TIMESTAMP

libarchive uses the SOURCE_DATE_EPOCH env variable to set the date of file
in an archive, this is useful for reproducibility.
Add a variable name PKG_TIMESTAMP that take a epoch time and set SOURCE_DATE_EPOCH
to this. By default it is the current time so no changes here.

Ensure libregex is built in time for googletest

In lib/Makefile, we document the dependency with SUBDIR_DEPEND

For buildworld orchestration, just prebuild libregex if GOOGLETEST is
enabled. googletest will get built in a later pass.

Move GIT_CMD a few lines down, so it is not in between SVN_*
lines.

Sponsored by: Rubicon Communications, LLC (netgate.com)

Add GIT_CMD, similar to SVN_CMD.

Sponsored by: Rubicon Communications, LLC (netgate.com)

Add WITH_CLANG_FORMAT option

clang-format is enabled conditional on either WITH_CLANG_EXTRAS or
WITH_CLANG_FORMAT. Some sources in libclang are build conditional on
either rule, and obviously the clang-format binary itself depends on the
rule.

clang-format could still use a manual page.

Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D25427

Squash liblzma build race

As of r362452, liblzma depends on libmd but the buildworld build order
hadn't been amended to document the new dependency.

Reported by: jenkins via freqlabs
X-MFC-With: r362452

flua: add ucl library

libucl comes with a Lua library binding. Build it into flua.

This lets us parse/generate config files in the various formats supported by
libucl with flua. For example, the following script will detect the format of
an object written to stdin as one of UCL config, JSON, or YAML and write it to
stdout as pretty-printed JSON:

local ucl = require('ucl')
local parser = ucl.parser()
parser:parse_string(io.read('*a'))
local obj = parser:get_object()
print(ucl.to_format(obj, 'json'))

Reviewed by: kevans, pstef
Approved by: mmacy (mentor)
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D25009

Retire BINUTILS and BINUTILS_BOOTSTRAP options

As of r361857 all BINUTILS options are disabled by default - ports
have been changed to depend on binutils if they require GNU as, and
all base system assembly files have been switched to use Clang's
integrated assembler.

Relnotes: Yes
Sponsored by: The FreeBSD Foundation

Makefile.inc1: remove stale dependency cleanup from 2018

Makefile.inc1: remove BINUTILS_BOOTSTRAP linker support

As of r359347 BINUTILS_BOOTSTRAP does not include the bfd linker.

pkgbase: use -dev,-dbg instead of -development,-debug

-development is long and awkward, and is also inconsistent with prior art
from the Linux world, which uses -dev (Debian) or -devel (Red Hat). Follow
the Debian convention, and similarly for debug info packages.

Also remove redundant pkgbase development tag from includes. We already tag
include files with package=runtime,dev; there is no need to separately tag
them as dev.

Discussed with: bapt
Reviewed by: manu
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D24139

Revert r360833, r360882: certctl rehash in installworld

This was solving the correct-ish problem in the wrong place. Noted by
brooks; while he didn't request a prompt revert, doing so now will
facilitate proper testing for the revised version of this.

buildworld: add back in missing semicolon

if foo; then blah else blah; fi has a drastically different meaning than
if foo; then blah; else blah; fi. Fix it.

Reported by: 0mp
X-MFC-With: r360833

installworld: attempt a certctl rehash at the tail end

This can be run as root or normal user with no problem; if they hadn't
twisted the WITHOUT_CAROOT knob, we'll attempt to use the host certctl to
rehash the DESTDIR. This would allow one to build systems WITHOUT_OPENSSL +
WITH_CAROOT with a populated /etc/ssl that they can then use with an
appropriate *ssl from somewhere else.

Cross-builds are fine because this will always use the host certctl, or just
nag if it's missing and it wasn't a WITHOUT_CAROOT build.

MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D24641

Fix race between prebuilding libsbuf and libgeom

The latter needs the former, but with a multi-job build on a fast
machine, the race is sometimes lost. This leads to "ld: error: unable to
find library -lsbuf", when linking libgeom.so.

Submitted by: kevans
MFC after: 3 days

Move shm_open dependency cleanup into a new home

r359461 introduced this nifty script to centralize these things, so add
shm_open.c there to remove a total of one (1) bad example from
Makefile.inc1.

Looked over by: emaste

Normalize deployment tools usage and definitions by putting into one place
instead of sprinkling them out over many disjoint files. This is a follow-up
to achieve the same goal in an incomplete rev.348521.

Approved by: imp
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D20520

Fix compilation with upstream clang builtin headers.

By using -nobuiltininc and adding the clang builtin headers resource dir
to the end of the compiler header search path, we can still find headers
such as immintrin.h but find the FreeBSD version of stddef.h/stdarg.h/..
first.

This is a workaround until we are able to settle on and complete a plan
to harmonize guard macros with LLVM. We've mostly worked out this on
FreeBSD systems by removing select headers from the installed set of
devel/llvm*, but that isn't a good solution for cross build.

Submitted by: arichardson
Obtained from: CheriBSD
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D17002

Pass the real value of MK_TESTS_SUPPORT to distribution.

This allows kyua's config file to actually be installed.

Reported by: arichardson
Reviewed by: arichardson
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D24268

add shell script for stale dependency hack

It's rather awkward to debug issues with the dependency cleanup hacks
when implemented via make. Add a cleanup shell script and move the
libomp hack there as an initial example.

Reviewed by: brooks
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D24228

Relax existence check of ${LOCALBASE}/bin/kyua

After the base kyua import the testsuite can utilize the in-base kyua
just fine.

Submitted by: Dries Michiels
Differential Revision: https://reviews.freebsd.org/D24230

Use the real value of MK_TESTS_SUPPORT in _libraries.

We need to build kyua libraries for kyua. Because we set MK_TESTS=no,
we can't not set MK_TESTS_SUPPORT=${MK_TESTS_SUPPORT} because the latter
defaults MK_TESTS_SUPPORT to no.

This fixes WITHOUT_TESTS + WITH_TESTS_SUPPORT builds.

Reported by: Ruslan Garipov
Reviewed by: emaste
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D24206

Stop building libl and liby

Those libs aren't needed anymore so stop building them for the bootstrap tools
or the compat libs.

Reviewed by: bapt, emaste
X-Differential Revision: D24171

Makefile.inc1: override MACHINE for native-xtools

For the final step of the native-xtools target, "everything" is built
with TARGET and TARGET_ARCH set to the architecture we wish to
cross-build for. However, CROSSENV overwrites the values of MACHINE and
MACHINE_ARCH, setting them to be identical to TARGET and TARGET_ARCH.
For native-xtools this is undesirable since we are building binaries to
run on the host architecture, that can cross compile for the target
architecture.

When building native-xtools for RISC-V, this issue manifests as an invalid
argument for "-march". The compiler is invoked with the target triple of the
host architecture, but the CFLAGS inherited from bsd.cpu.mk are that of the
target architecture.

Reviewed by: imp, bdrewery
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D23838

Makefile.inc1: remove old (2018) stale dependency hacks

We handle a number of stale dependency issues on an ad-hoc basis, in
order to support ongoing NO_CLEAN builds. These hacks do not need to
be maintained indefinitely; now remove those for issues that are more
than a year old.

Sponsored by: The FreeBSD Foundation

Makefile.inc1: set DB_FROM_SRC also for stageworld

stageworld, invoked from e.g. `make packages`, shouldn't rely on the
build host's users/groups.

Sponsored by: The FreeBSD Foundation

Revert r359083, fixed properly by r359088

r359083 introduced a workaround for stale libomp dependencies during a
regular (no -DNO_CLEAN) buildworld. r359088 addressed the reason the
clean step missed libomp, so revert the workaround.

Sponsored by: The FreeBSD Foundation

invoke _cleanobj_fast_depend_hack unconditionally

Apparently make ${CLEANDIR} is leaving stale entries in .depend files;
for now invoke the hacky cleanup in both the -DNO_CLEAN and normal
(no -DNO_CLEAN) cases.

In collaboration with: dim
Sponsored by: The FreeBSD Foundation

Add an internal liblua and use it in flua.

The new liblua will be used in a forthcoming import of kyua.

Reviewed by: kevans
Obtained from: CheriBSD
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D24090

Makefile.inc1: add a note when deleting stale dependencies

We have ad-hoc stale dependency handling in Makefile.inc1 to handle the
cases where file extensions change, but it appears that some cases are
not functional. Add a note when about to clean stale deps to help
when investigating failure reports.

Sponsored by: The FreeBSD Foundation

Makefile.inc1: move dependency hack comment to the block it applies to

Extend r358907 to explicitly remove stale lib32 dependency

After r325072 stale lib32 dependencies were not remooved. A more
holistic approach is needed to address this but for the immediate issue
(-DNO_CLEAN builds across r358851) just readd the explicit lib32 path.

Reported by: dim
Sponsored by: The FreeBSD Foundation

Allow -DNO_CLEAN build across r358851.

The openmp 10.0.0 import renamed one .c file to .cpp, and this is
something our dependency system does not handle correctly. Add another
ad-hoc cleanup to get rid of the stale dependency.

PR: 244251
MFC after: 6 weeks
X-MFC-With: 358851

retire amd(8)

autofs was introduced with FreeBSD 10.1 and is the supported method for
automounting filesystems. As of r296194 the amd man page claimed that it
is deprecated. Remove it from base now; the sysutils/am-utils port is
still available if necessary.

Discussed with: cy
Relnotes: Yes
Sponsored by: The FreeBSD Foundation

retire in-tree GPL dtc devicetree compiler

Now that we no longer have GCC 4.2.1 in the tree and can assume FreeBSD
is being built with a C++11 compiler available, we can use BSDL dtc
unconditionally and retire the GPL dtc.

GPL dtc now has FreeBSD CI support via Cirrus-CI to help ensure it
continues to build/work on FreeBSD and is available in the ports tree
if needed.

The copy of (copyfree licensed) libfdt that we actually use is in
sys/contrib/libfdt so the extra copy under contrib/dtc/libfdt can be
removed along with the rest of the GPL dtc.

Reviewed by: kevans, ian, imp, manu, theraven
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23192

retire the LLVM_LIBUNWIND option

LLVM's libunwind is used on all FreeBSD-supported CPU architectures and
is a required component.

Reviewed by: brooks (earlier)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23123

remove GCC 4.2.1 build infrastructure

As described in Warner's email message[1] to the FreeBSD-arch mailing
list we have reached GCC 4.2.1's retirement date. At this time all
supported architectures either use in-tree Clang, or rely on external
toolchain (i.e., a contemporary GCC version from ports).

GCC 4.2.1 was released July 18, 2007 and was imported into FreeBSD later
that year, in r171825. GCC has served us well, but version 4.2.1 is
obsolete and not used by default on any architecture in FreeBSD. It
does not support modern C and does not support arm64 or RISC-V.

Thanks to everyone responsible for maintaining, updating, and testing
GCC in the FreeBSD base system over the years.

So long, and thanks for all the fish.

[1] https://lists.freebsd.org/pipermail/freebsd-arch/2020-January/019823.html

PR: 228919
Reviewed by: brooks, imp
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23124

Take LINKER_FREEBSD_VERSION from numerical field after dash

Summary:
With COMPILER_FREEBSD_VERSION, we use a numeric value that we bump each
time we make a change that requires re-bootstrapping, but with the
linker variant, we instead take the entire part after "FreeBSD", as in
this example version output:

LLD 9.0.1 (FreeBSD c1a0a213378a458fbea1a5c77b315c7dce08fd05-1300006) (compatible with GNU linkers)

E.g., LINKER_FREEBSD_VERSION is currently being set to
"c1a0a213378a458fbea1a5c77b315c7dce08fd05-1300006". This means that
*any* new upstream lld version will cause re-bootstrapping.

We should only look at the numerical field we append after a dash
instead. This review attempts to make it so.

The only thing I am not happy about is the post-processing of awk output
in Makefile.inc1. I notice that our awk does not have gensub(), so it
can't substitute a numbered sub-regex with \1, \2, etc. Suggestions
welcome. :)

MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D23691

Restore missing comment

I was overly agressive about removing the entire comment. It was still
valid, except the part about being only for some architectures.

Reviewed by: emaste
Differenial Revision: https://reviews.freebsd.org/D23523

Per the firm plan, start to remove sparc64

The sparc64 architecture is being removed from FreeBSD 13, starting
now. This removes it from the top level only. It is the only
architecture that didn't see substantial work after the call to get
things working with the external toolchain.

retire BSD_CRTBEGIN option

BSD crt is currently used on all architectures (other than sparc64).
Remove the option and use BSD crt everywhere as part of the GCC 4.2.1
retirement plan.

https://lists.freebsd.org/pipermail/freebsd-arch/2020-January/019823.html

PR: 239851
Reviewed by: andrew, brooks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23122

Remove support for auto-selecting an external binutils.

All of the in-tree architectures not supported by in-tree binutils are
supported by lld, so the condition is now always false. It also
didn't fully work since the external binutils are installed into a
directory that uses the host's OS version, not the target OS version.

Reviewed by: emaste, imp
Differential Revision: https://reviews.freebsd.org/D23294

pkgbase: allow the pkg format to be overridden

Compressing .txz packages can be rather slow, and speed is likely more
important than disk space during development. Allow package format to
be set via PKG_FORMAT make variable.

Reviewed by: bapt
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23257

pkgbase: cache pkg ABI for all world/kernel packages

Rather than invoking `pkg config ABI` repeatedly.

Reviewed by: manu
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D23255

Makefile.inc1: push /usr/libexec into the BPATH/TMPPATH

${WORLDTMP}/legacy/usr/libexec will only have libexec/ bits that we've
pushed as bootstrap tools, so this is generally safe to include prior to
PATH. The following are the ramifications of this change:

- BPATH addition gets us at least bootstrap flua in WMAKEENV path for
buildenv, for those earlier systems where it's bootstrapped still

- Reworked the sysent target to just set PATH and let it get worked out in
src.lua.mk or individual sysent makefiles -- this gives us back the
ability to overwrite LUA_CMD and use a different/external lua for these
targets. sysent can also now work cleanly in buildenv.

- tools/build/Makefile will now symlink the host flua into build's host
tools so that the above can work without needing to add the host's
/usr/libexec explicitly into TMPPATH.

Reviewed by: arichardson, brooks, imp (all slightly earlier version)
Differential Revision: https://reviews.freebsd.org/D22464

Remove prereq lib support for GNU unwinder

LLVM's libunwind is used for all archs, as of r356514. As GCC 4.2.1
will soon be removed, and with it the GNU unwinder, start simplifying
the prereq/startup lib logic in Makefile.inc1. From here if there are
any unwinder bugs (on any arch) the path foward is to fix LLVM's
libunwind.

Provide libssp based on libc

For libssp.so, rebuild stack_protector.c with FORTIFY_SOURCE stubs that just
abort built into it.

For libssp_nonshared.a, steal stack_protector_compat.c from
^/lib/libc/secure and massage it to maintain that __stack_chk_fail_local
is a hidden symbol.

libssp is now built unconditionally regardless of {WITH,WITHOUT}_SSP in the
build environment, and the gcclibs version has been disconnected from the
build in favor of this one.

PR: 242950 (exp-run)
Reviewed by: kib, emaste, pfg, Oliver Pinter (earlier version)
Also discussed with: kan
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D22943

Makefile.inc1: remove libcompiler_rt from _startup_libs

libcompiler_rt is already present in _prereq_libs, which will get built just
before _startup_libs and in advance of shared libc. Dedupe.

Look for cross toolchain makefiles in /usr/share/toolchains.

The freebsd-binutils and freebsd-gcc* packages install toolchain
makefiles to /usr/share/toolchains rather than LOCALBASE.

Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D22985

Remove arm/arm as a valid target.

TARGET=arm now defaults to TARGET_ARCH=armv7
TARGET_ARCH=arm is no longer valid.

Bump __FreeBSD_version to 1300073

Tested with make universe. Any stale LINT-V5 config files remaining in the tree
will fail the universe build. However, LINT-V5 was removed in r355119.

This retirement has been planned since last summer. The armv5 port is fragile:
it works OK for some peeople, and fails badly for others. There's a number of
subtle bugs in busdma, pmap and other MD parts of thee system that present
themselves under load or in unusual circumstances (like fsck after a
crash). stable/8, branched 10 years ago, was the last reliable release. Since
the support burden is larger then the benefit, the consensus view is armv5
should be removed from the tree.

Discussed with: arm@ mailing list and arm developer community.

revert r355609

libpmc: build json event support also on arm64

connect arm linuxulator to top-level make sysent

Reported by: kevans

Make NOCLEAN an error instead of a warning

The warning was added in r289728 (over four years ago) and at that time
NO_CLEAN was already the correct spelling for over a decade.

Make NOCLEAN an error as the next step to removing these backward
compatibility shims.

Update Makefile.inc1 dtc comment

We use the BSDL dtc by default now (as long as we're using a C++11
compiler).

WITH_SYSTEM_LINKER: Fix rebuilding lld every time.

This is due to LLD_REVISION_STRING being renamed to LLD_REVISION in
r351442 and the value being moved to another location in r351965.

`make test-system-linker` can be used to see the values being used here.

Reported by: ler

Convert in-tree sysent targets to use new makesyscalls.lua

flua is bootstrapped as part of the build for those on older
versions/revisions that don't yet have flua installed. Once upgraded past
r354833, "make sysent" will again naturally work as expected.

Reviewed by: brooks
Differential Revision: https://reviews.freebsd.org/D21894

Refine r354661 to unbreak the GCC_BOOTSTRAP case.

MK_CLANG_IS_CC controls installing links for GCC, not just clang. Set
MK_CLANG_IS_CC to the value of MK_CLANG_BOOTSTRAP. This will leave it
as "no" if no bootstrap compiler is being built or GCC 4.2.1 is being
used as the bootstrap compiler, and "yes" if clang is being used as
the bootstrap compiler.

Submitted by: bdrewery (kind of, he suggested this on IRC while I was
testing the original patch)
Reviewed by: kevans, imp
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D22350

Force MK_CLANG_IS_CC on in XMAKE.

This ensures that a bootstrap clang compiler is always installed as cc
in WORLDTMP. If it is only installed as 'clang' then /usr/bin/cc is
used during the build instead of the bootstrap compiler.

Reviewed by: imp
MFC after: 1 month
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D22332

suggest xtoolchain package if binutils and GCC bootstraps are both broken

Previously we checked for only BINUTILS_BOOTSTRAP as a broken option
and suggested installing the binutils package. This was originally done
for arm64 where we used the in-tree Clang with external binutils package.

Add a case to the warning to suggest instead the full xtoolchain package
if we have no in-tree compiler either.

Differential Revision: https://reviews.freebsd.org/D21851

Revert r354518 and commit the intented fix rather than the diagnostic
check.

This fixes the definition of MK_LIB32 in Makefile.inc1.

Fix the ARCH check for LIB32 from Makefile.inc1.

libcompat: build 32-bit rtld and ldd as part of "everything"

Alter bsd.compat.mk to set MACHINE and MACHINE_ARCH when included
directly so MD paths in Makefiles work. In the process centralize
setting them in LIBCOMPATWMAKEENV.

Alter .PATH and CFLAGS settings in work when the Makefile is included.

While here only support LIB32 on supported platforms rather than always
enabling it and requiring users of MK_LIB32 to filter based
TARGET/MACHINE_ARCH.

The net effect of this change is to make Makefile.libcompat only build
compatability libraries.

Changes relative to r354449:

Correct detection of the compiler type when bsd.compat.mk is used
outside Makefile.libcompat. Previously it always matched the clang
case.

Set LDFLAGS including the linker emulation for mips where -m32 seems to
be insufficent.

Reviewed by: imp, kib (origional version in r354449)
Obtained from: CheriBSD (conceptually)
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22251

Revert r354449: libcompat: build 32-bit rtld and ldd as part of "everything"

Additional testing is required..

libcompat: build 32-bit rtld and ldd as part of "everything"

Alter bsd.compat.mk to set MACHINE and MACHINE_ARCH when included
directly so MD paths in Makefiles work. In the process centralize
setting them in LIBCOMPATWMAKEENV.

Alter .PATH and CFLAGS settings in work when the Makefile is included.

While here only support LIB32 on supported platforms rather than always
enabling it and requiring users of MK_LIB32 to filter based
TARGET/MACHINE_ARCH.

The net effect of this change is to make Makefile.libcompat only build
compatability libraries.

Reviewed by: imp, kib
Obtained from: CheriBSD (conceptually)
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22251

Building head on stable/11 requires libzstd

Add lib/libzstd to _elftoolchain_libs

tools/build/Makefile needs to create the install dir for libzstd
Since this would make the line too long, rework to use a list
in one per line format (easier to add in future)
and dispense with the .for loop

Reviewed by: emaste bapt
Differential Revision: https://reviews.freebsd.org/D220134

Strip "sf" suffix when generating a target triple.

This fixes the target triple used when compiling riscv64sf with clang.

Discussed with: mhorne
MFC after: 2 weeks
Sponsored by: DARPA

Additional fix for -DNO_CLEAN build across r353340 and r353381

opensolaris_atomic.S is now only used on i386 with opensolaris_atomic.c
used on other platforms. After r353381 it doesn't exist on those
platforms so the stale dependency would result in a build error.

r353408 addressed this issue for cddl/lib/libzpool, but it persisted
with the opensolaris and zfs modules.

buildkernel: always add standard kernel configuration include path

This should change nothing for kernel configurations at the standard
locations in the source tree. However, if KERNCONFDIR is used to
specify a custom location for a kernel configuration file (e.g., out of
tree), then both the custom location and the standard location, in this
order, will be used as include paths for config(8). This will allow the
kernel configuration to include files from both locations.

Reviewed by: bdrewery
MFC after: 16 days
Differential Revision: https://reviews.freebsd.org/D22057

Build compat libraries before "everything".

This is required for us to link programs against compat versions of
libraries.

Reviewed by: bdrewery, jhb
Sponsored by: DARPA, AFRL

Rename top-level LIBCOMPAT to _LIBCOMPAT.

This avoids a conflict with LIBCOMPAT defined in bsd.libnames.mk.

Reviewed by: bdrewery
Sponsored by: DARPA, AFRL

Fix -DNO_CLEAN build across r353340 and r353381

opensolaris_atomic.S is now only used on i386 with opensolaris_atomic.c
used on other platforms. After r353381 it doesn't exist on those
platforms so the stale dependency would result in a build error.

Merge ^/head r353316 through r353350.

Allow -DNO_CLEAN build across r352689.

Split the LIBCOMPAT case because the usual egrep only matches in
LIBCOMPAT on amd64.

Merge ^/head r352764 through r353315.

Add a top-level makeman target

Abstracting away the details of how src.conf(5) is generated is arguably a
good thing; do so with a top-level makeman target.

Update cloudabi(32|64) sysents with "make sysent".

MFC after: 1 week

Disable build of LOCAL_MODULES for cross-builds by default.

WITHOUT_LOCAL_MODULES can be set to disable LOCAL_MODULES for native
builds. WITH_LOCAL_MODULES can be set to leave it enabled for cross
builds.

This does not use a knob in kern.opts.mk because the options framework
does not currently support options whose default varies on the build
type. I discussed a few options there with Warner (e.g. maybe having
a tri-state where the default value is "auto" and having Makefile.inc1
apply logic when MK_LOCAL_MODULES is set to "auto"), but Warner ok'd
this approach for now until a better solution is implemented.

Requested by: many
Reviewed by: imp (in person at EuroBSDCon)
Differential Revision: https://reviews.freebsd.org/D21608

Merge ^/head r352587 through r352763.

Adjust Makefile.inc1 syscall sub commit

Add linux-compatible memfd_create

memfd_create is effectively a SHM_ANON shm_open(2) mapping with optional
CLOEXEC and file sealing support. This is used by some mesa parts, some
linux libs, and qemu can also take advantage of it and uses the sealing to
prevent resizing the region.

This reimplements shm_open in terms of shm_open2(2) at the same time.

shm_open(2) will be moved to COMPAT12 shortly.

Reviewed by: markj, kib
Differential Revision: https://reviews.freebsd.org/D21393

Merge ^/head r352537 through r352586.

Ensure libthr is always built before libprivatezstd when building the
startup libs

Reported by: "Galazka, Krzysztof" <krzysztof.galazka@intel.com>

Merge ^/head r352436 through r352536.

Add native support for zstd to libarchive

Note that old pkg will failed to build after this. A recent ports tree (one
providing pkg 1.12+) is required to build. Older already built pkg, should
continue working as expected

PR: 238797
Exp run by: antoine
Reviewed by: cem
Approved by: cem
Differential Revision: https://reviews.freebsd.org/D20752

Always build lldb-tblgen, for now, and add a note.

Update build glue for a build of clang plus lldb (enabled via MK_LLDB).

Add head(1) to native-xtools so that it can be used in qemu-user jails

Differentiate package versions for ALPHA/BETA/PRERELEASE/RC phases.

Currently APLHA packages are treated as CURRENT or STABLE versions,
resulting in e.g. 13.0.s20190615125609. This version number is indeed
different from the next version number but ALPHA2 would be nicer IMO.

For the BETA, PRERELEASE and RC phases the packages are versioned the
same as for releases, so 11.3-BETA1 is 11.3 and so is 11.3-RC1, meaning
that pkg cannot easiliy upgrade from the former the next. This happened
on my Raspberry Pi which runs pkgbase.

Submitted by: rene
Approved by: manu
Event: Berlin hackathon 2019
Differential Revision: https://reviews.freebsd.org/D20651

Makefile.inc1: clean up stale dependency hacks

Carrying on from r339607, remove ad-hoc dependency cleanup for changes
prior to June 2018.

Sponsored by: The FreeBSD Foundation

Rollback the rest of the botched r348521. Re-work would be posted to
reviews. Sorry.

Reported by: Enji Cooper

Leave mtree hardcoded for now. Reverting partially 348521 and also
the followup stopgap change, because I don't think it's a correct. I still
need to figure out where to stick it in. In cannot be in Makefile.inc1
and it cannot be in etc/Makefile from the looks of it to avoid
chicken-and-egg problem.

Stopgap fix to breakage from r348521.

MTREE is defined, while MTREE_CMD isn't, use it instead.
This may not be 'right' but it fixes the CI and other issues.

Submitted by: Andreas Nilsson

Fix several places where tool name has been hardcoded:

install -> ${INSTALL}
mtree -> ${MTREE_CMD}
services_mkdb -> ${SERVICES_MKDB_CMD}
cap_mkdb -> ${CAP_MKDB_CMD}
pwd_mkdb -> ${PWD_MKDB_CMD}
kldxref -> ${KLDXREF_CMD}

If you do custom FreeBSD builds you may want to override those
in some cases.

Sponsored by: Sippy Software, Inc.

Update __FreeBSD_version and Makefile check for r348347

libdwarf needs forcibly rebuilt after r348347.

Change package the MTREEFLAGS and INSTALLFLAGS missed in r348185

Reported by: lwhsu

Move master.passwd and group to lib/libc/gen/

libc was picked as the destination location for these because of the syscalls
that use these files as the lowest level place they are referenced.

Approved by: will (mentor), rgrimes, manu
Differential Revision: https://reviews.freebsd.org/D16728

Add wc(1) to native-xtools so that it can be used in qemu-user jails