build: Order kernel-toolchain and buildkernel targets

MFC after: 1 week

Import bmake-20240309

Intersting/relevant changes since bmake-20240108

ChangeLog since bmake-20240108

2024-03-10 Simon J Gerraty <sjg@beast.crufty.net>

* boot-strap: tests can take a long time; use a cookie to
skip them if bmake has not been updated since tests last
ran successfully.

* Makefile: Cygwin handles MANTARGET man

* unit-tests/Makefile: set BROKEN_TESTS for Cygwin

2024-03-09 Simon J Gerraty <sjg@beast.crufty.net>

* VERSION (_MAKE_VERSION): 20240309
Merge with NetBSD make, pick up
o set .ERROR_EXIT to the exit status of .ERROR_TARGET
this allows a .ERROR target to ignore the case of
.ERROR_EXIT==6 which just means that the build actually
failed somewhere else.

2024-03-04 Simon J Gerraty <sjg@beast.crufty.net>

* VERSION (_MAKE_VERSION): 20240303

* var.c: on IRIX we need both inttypes.h and stdint.h

2024-03-01 Simon J Gerraty <sjg@beast.crufty.net>

* VERSION (_MAKE_VERSION): 20240301
Merge with NetBSD make, pick up
o export variables with value from target scope
when appropriate.

2024-02-12 Simon J Gerraty <sjg@beast.crufty.net>

* VERSION (_MAKE_VERSION): 20240212
Merge with NetBSD make, pick up
o remove unneeded conditional-compilation toggles
INCLUDES, LIBRARIES, POSIX, SYSVINCLUDE, SYSVVARSUB,
GMAKEEXPORT NO_REGEX and SUNSHCMD

* configure.in: add check for regex.h

* var.c: replace use of NO_REGEX with HAVE_REGEX_H

2024-02-04 Simon J Gerraty <sjg@beast.crufty.net>

* VERSION (_MAKE_VERSION): 20240204
Merge with NetBSD make, pick up
o var.c: fix some lint (-dL) mode parsing issues

2024-02-02 Simon J Gerraty <sjg@beast.crufty.net>

* VERSION: (_MAKE_VERSION): 20240202
Merge with NetBSD make, pick up
o make.1: note that arg to :D and :U can be empty
o var.c: $$ is not a parse error when .MAKE.SAVE_DOLLARS=no

mk/ChangeLog since bmake-20240108

2024-03-09 Simon J Gerraty <sjg@beast.crufty.net>

* install-mk (MK_VERSION): 20240309

* meta.sys.mk: _metaError: if .ERROR_EXIT == 6, we do not
want to save the .ERROR_META_FILE

2024-02-20 Simon J Gerraty <sjg@beast.crufty.net>

* install-mk (MK_VERSION): 20240220

* sys.dirdeps.mk, dirdeps-targets.mk, init.mk:
do not set .MAIN: dirdeps in sys.dirdeps.mk
dirdeps-targets.mk will do that for top-level builds
and init.mk will do it for others.
This allows a Makefile which has no need of 'dirdeps' to
set .MAIN for itself and "just work".

2024-02-18 Simon J Gerraty <sjg@beast.crufty.net>

* bsd.*.mk: for makefiles that get a bsd. symlink,
use _this in multiple inclusion tags since .PARSEFILE will not
DTRT when such a makefile is included directly by Makefile and
automatically (without bsd. prefix).
Since we cannot guarantee that our sys.mk will be used, we provide
a default _this in each makefile that gets a bsd. prefix such that
the value is the same regardless of bsd. prefix.

* subdir.mk: drop the !target guard on $SUBDIR_TARGETS

2024-02-12 Simon J Gerraty <sjg@beast.crufty.net>

* install-mk (MK_VERSION): 20240212

* SPDX-License-Identifier: BSD-2-Clause
Add SPDX-License-Identifier to inidicate that I consider
my copyright on any of these makefiles equivalent to BSD-2-Clause

* autoconf.mk: allow for configure.ac as currently recommended

* subdir.mk: support @auto
which is replaced with each subdir that
has a [Mm]akefile.

* subdir.mk: include local.subdir.mk if it exists.

* subdir.mk: rework to handle .WAIT

2024-02-11 Simon J Gerraty <sjg@beast.crufty.net>

* subdir.mk: _SUBDIRUSE report the target we are entering subdirs for.

2024-02-10 Simon J Gerraty <sjg@beast.crufty.net>

* prog.mk: treat empty SRCS the same as undefined

2024-02-02 Simon J Gerraty <sjg@beast.crufty.net>

* Avoid undefined errors in lint (-dL) mode

* man.mk (CMT2DOC_FLAGS): note that -mm does mdoc(7)

2024-01-28 Simon J Gerraty <sjg@beast.crufty.net>

* install-mk (MK_VERSION): 20240128

* FILES: add ccm.dep.mk for C++ modules
add suffixes.mk for common location for generic SUFFIX rules.

* auto.dep.mk autodep.mk meta.autodep.mk: include ccm.dep.mk
replace OBJ_EXTENSIONS with OBJ_SUFFIXES

* autodep.mk: leverage CXX_SUFFIXES for __depsrcs
and update style (spaces around = etc)

* init.mk: add OBJS_SRCS_FILTER to filter SRCS when
setting OBJS

* meta2deps.py: handle multiple ./ embedded in path better.

Import tzcode 2024a

Import tzdata 2024a

pkgbase: Create two new rules for creating repo

This adds two new rules named create-packages-kernel-repo and
create-packages-world-repo.
The goal of those rules is to create the {kernel,world} packages and
after that the repository.
It helps a lot for developing with pkgbase by adding the dev machine
repository created by those rules on top of the official pkgbase one.

Differential Revision: https://reviews.freebsd.org/D43623
Reviewed by: bapt, emaste
Sponsored by: Beckhoff Automation GmbH & Co. KG

Makefile: Deorbit fmake support

fmake has been out of the tree for 10 years / 5 major releases now. The
need to bootstrap from it has been gone for at least 6 if not 8
years. While we may still need to bootstrap bmake, we don't need to do
it from fmake, so only retail the infrastructure to update from bmake to
bmake. Retain, for now, the WANT_MAKE_VERSION stuff, though we're always
up to date when building from supported and quasi-supported platforms.

Also remove all the checks to see if .PARSEDIR is defined. It is always
defined and was an early, fail-safe way to tell fmake from bmake during
the transition.

Adjust comments that refer to old fmake and remove those no longer
relevant.

Sponsored by: Netflix
Reviewed by: brooks
Pull Request: https://github.com/freebsd/freebsd-src/pull/980

Import tzdata 2023d

pkgbase: create source package

FreeBSD-src for all the sources but the kernel
FreeBSD-src-sys just for the kernel

MFC After: 3 days
Reviewed by: manu
Differential Revision: https://reviews.freebsd.org/D42651

bsddialog: Import version 1.0

Makefile: Support universe-toolchain on non-FreeBSD

We currently pass MACHINE and MACHINE_ARCH as TARGET and TARGET_ARCH
respectively for universe-toolchain, but on non-FreeBSD these may not
have values that we understand (e.g. on Linux it will be x86_64 rather
than amd64) for TARGET/TARGET_ARCH (note that we do support them for
MACHINE/MACHINE_ARCH). Since the choice is a bit arbitrary and merely
determines what LLVM's default triple will be, use amd64 on non-FreeBSD
as a known-good default.

Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D41545

Import unifdef 2.11

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

universe: Demote armv6 to an extra architecture.

Per discussions over how to lighten the load of armv6, move it to an
extra arch. You can still build TARGET_ARCH=armv6 if you desire, but
it won't be built as part of 'make universe' without -DEXTRA_TARGETS.

Sponsored by: Netflix

Makefile.inc1: Add new buildetc and installetc targets

These implement the steps that etcupdate (and mergemaster) need in order
to build their pristine trees, avoiding the need to use internal targets
and variables. Additionally, buildetc includes the steps needed to build
with BUILD_WITH_STRICT_TMPPATH enabled, since we need some host tools to
be available during the various build and install steps.

Reviewed by: jhb, imp
Differential Revision: https://reviews.freebsd.org/D41205

Makefile.inc1: Split _cleanworldtmp out from _worldtmp

This mirrors _cleanobj vs _obj, and will be used by etcupdate to ensure
its build tree has host tools available when BUILD_WITH_STRICT_TMPPATH
is enabled by default (whether in the source tree, like CheriBSD, or in
a system config file).

Note that due to line length the .MAKE .PHONY dependency line is split
into two, one for the filtered WMAKE_TGTS and one for the filtered
.ALLTARGETS.

Reviewed by: brooks, jhb
Differential Revision: https://reviews.freebsd.org/D41190

Makefile: Automatically generate list of libcompat targets

Reviewed by: brooks, jhb
Differential Revision: https://reviews.freebsd.org/D41179

lua: Update to 5.4.6

Sponsored by: Netflix

Import bmake-20230622

Relevant/interesting changes:

o optimize string matching for ':M' and ':N'
o warn about malformed patterns in ':M', ':N' and '.if make(...)'
o allow guards to be targets as well as variables
The guard targets may include variable references like
__${.PARSEDIR:tA}/${.PARSEFILE}__
o optimization for makefiles protected from multiple-inclusion
skip even opening the file after first include.
o var.c: do not allow delete of readOnly variable
o parse.c: .break takes no args

Move MACHINE/MACHINE_ARCH handling to sys.machine.mk

Rather than define the TARGETS and TARGET_ARCHES in src/Makefile
put them in sys.machine.mk so they can also be leveraged by
non-top-level builds.

We have TARGET_MACHINE_LIST as the list of MACHINES we build for.

For each MACHINE we have a MACHINE_ARCH_LIST_${MACHINE}
and the first entry of each list is used as default for
MACHINE_ARCH_${MACHINE}

We can now remove a bunch of MACHINE_ARCH.* handling from
local.sys.dirdeps*mk

Reviewed by: imp
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D40083

Update the Arm Optimized Routines

Import the v23.01 release of the Arm Optimized Routines [1]. This updates
the license to MIT OR Apache-2.0 WITH LLVM-exception.

[1] https://github.com/ARM-software/optimized-routines/tree/v23.01

Approved by: core (License change)
Sponsored by: Arm Ltd

Import tzcode 2023a

Import bmake-20230414

Relevant/interesting changes:

* make.1: document seemingly unexplained Error code 6.
o compat.c: CompatDeleteTarget skip .PHONY targets to be
consistent with JobDeleteTarget.
o job.c: fix memory leak in handling sysv :from=to modifiers
o unit tests for .MAKE.META.IGNORE_{FILTER,PATHS,PATTERNS}
o var.c: fix parsing of unevaluated subexpressions with
unbalanced '{}'
o cond.c: reduce complexity of evaluating expressions

Add jobs.mk to allow for target-jobs

jobs.mk automates -j$JOB_MAX and capturing build log based on target.

Compute a default for JOB_MAX in local.sys.mk

Reviewed by: stevek, imp
Differential Revision: https://reviews.freebsd.org/D39683

Remove the riscv64sf architecture.

Reviewed by: jrtc27, arichardson, br, kp, imp, emaste
Differential Revision: https://reviews.freebsd.org/D39496

Import tzdata 2023a

Import bmake-20230208

ChangeLog since bmake-20230126

* VERSION (_MAKE_VERSION): 20230208
Merge with NetBSD make, pick up
o var.c: always use SCOPE_GLOBAL for :_ to avoid problems
when it has been used within conditional expressions

* install-sh: if making directories ensure umask is set
to match mode.

* Makefile: use DIRMODE for directories and
NONBINMODE for man pages and mk files

Import Lua 5.4.4

Import lua 5.4.3

Import bmake-20230126

Relevant/interesting changes (see ChangeLog for more):

o variables like .newline and .MAKE.{GID,PID,PPID,UID}
should be read-only.
o .[NO]READONLY: for control of read-only variables
o .SYSPATH: for controlling the path searched for makefiles
o allow for white-space between command specifiers @+-
o add more details to warning 'Extra targets ignored'
o make.1: sync list of built-in variables with reality
sort list of built-in variables
o cond.c: add more details to error message for numeric comparison
o job.c: fix handling of null bytes in output
o Allow .break to terminate a .for loop early
o var.c: fix out-of-bounds errors when parsing
o fix exit status for '-q' (since 1994)

universe: Switch GCC toolchains to GCC 12.

Reviewed by: imp, emaste
Differential Revision: https://reviews.freebsd.org/D38230

Import zstd 1.5.2

Shell-escape assignments to PATH in the top-level makefiles

Since 16fbf0191243e7c9dff6615b1424b5d39186b36c PATH is no longer set
to a hardcoded value on non-FreeBSD build hosts, so we can end up with
spaces in $PATH. Instead of only escaping PATH I updated all `env PATH=`
uses in the toplevel makefile. While many of these currently can't
contain any special characters (since the build would have failed
already), in theory this gets us closer to allowing build/source
directory to contain e.g. spaces.

Makefile: Avoid sanitizing PATH on non-FreeBSD systems

Allow the build process to find host binaries during the host-symlinks target when
cross-building on non-FreeBSD systems. Whilst most non-FreeBSD systems have all
the needed tools in /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin (the final
path added by host-symlinks itself), Homebrew for macOS on Arm defaults to
/opt/homebrew/bin, other more niche systems may also deviate and users may
expect tools in a customised PATH to be picked up, unlike on FreeBSD where we
want to ensure everything comes from base. In particular, (un)xz are needed
from Homebrew on macOS, and thus cannot be found on Arm without this.

Note that non-FreeBSD builds enforce BUILD_WITH_STRICT_TMPPATH, and so the
actual main build steps will still use a sanitised PATH.

Reviewed by: jrtc27, arichardson
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D37991

Import sendmail 8.17.1

Import tzcode 2022g

Import tzcode 2022f

Import tzcode 2022c

Unsplit vendor import of tzcode 2010n

Historically, to work around CVS limitations, imports on the tzcode
vendor branch were split into subdirectories, which were then merged to
their respective counterparts in contrib. In the Git era, this split is
unnecessary and adds more work.

Undo the split for the most recent vendor import of tzcode 2010n to
prepare for the vendor import of tzcode 2022c.

Import tzdata 2022g

Import tzdata 2022f

Makefile: replace mergemaster references with etcupdate

Similar to UPDATING and handbook changes, treat Makefile the same.

PR: 260822
Approved by: jhb, philip
Differential Revision: https://reviews.freebsd.org/D36877

contrib/bsddialog: Import version 0.4

Improvements and changes to integrate bsddialog(1) with scripts in BASE.
Overview:

* New options. --and-widget, --keep-tite, --calendar.
* Change output format. Menus and --print-maxsize.
* Redefine sizing. Fixed rows, cols and menurows became at the most.
* Add DIAGNOSTICS. Error messages for bad arguments and options.
* Add keys. Space for --menu, fast keys for --msgbox and --yesno.
* Text. Change default text modification, add --cr-wrap.

See /usr/src/contrib/bsddialog/CHANGELOG '2022-09-24 Version 0.4'
for more detailed information.

contrib/bsddialog: Import version 0.3

New features overview:
* Unicode. User interface handles multi-column characters. API can
handle char* like a multibyte character string. Internally wide
characters are used for keyboard input, to adapt word wrapping and
dynamic text auto-sizing for multi-column characters.
* Forms refactoring. Complete rewrite deleting libformw dependency.
* Theme. New utility options to save and load custom theme at run-time.
* TUI navigation. Added keys to navigate input components. Changed
default focus behavior of input dialogs to be LGPL-dialog-like; a new
option can set the previous whiptail-like behavior.

See /usr/src/contrib/bsddialog/CHANGELOG '2022-08-29 Version 0.3'
for more detailed information.

Import tzdata 2022c

Import tzdata 2022b

Import bmake-20220726

Relevant/interesting changes:

* Auto-create objdir for bmake/unit-tests if appropriate

Switch to GCC 9 for the GCC tinderbox.

This also adds arm and riscv to the GCC tinderbox.

Reviewed by: emaste (older version)
Differential Revision: https://reviews.freebsd.org/D26202

Import tzdata 2022a

bsddialog: Import version0.2

This version provides improvements and fixes mainly to use bsddialog
utility in bsdinstall/scripts. The lib API is not broken so the
previous converted utilities (tzsetup, distextract, etc.) are OK.

Approved by: bapt (mentor)
Differential Revision: https://reviews.freebsd.org/D34458

cleankernel: A target to delete the kernel compile file

With the meta-build, it's always a NO_CLEAN build. Provide a way to
remove so one can rebuild from scratch. 'cleankernel' will delete the
kernel and modules object directories. Document this in build(7).

Sponsored by: Netflix
Reviewed by: debdrup, markj
Differential Revision: https://reviews.freebsd.org/D32978

import terminus-font-4.49.1

bsddialog: import version 0.1

Remove quotes around Makefile .error/.warn/.info strings

The text after .error et al is emitted verbatim.

Reviewed by: sjg
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33904

Add list-old-{dirs,files,libs} targets.

These targets generate a raw list of the candidate old files roughly
corresponding to the values of OLD_DIRS, OLD_FILES, and OLD_LIBS.
Currently list-old-files also includes uncompressed manpages in
addition to compressed manpages.

Use these targets in the implementation of check-old-* and
delete-old-* to replace duplicated logic.

Reviewed by: imp, emaste
Sponsored by: The University of Cambridge, Google Inc.
Differential Revision: https://reviews.freebsd.org/D33327

libsoft: Remove targets to generate libsoft

Remove the targets needed to make and install libsoft.

Sponsored by: Netflix

Let powerpc64le live as not experimental architecture.

Summary: It's currently just as stable as powerpc64, with more ports working.

Reviewers: alfredo, bdragon, luporl, jhibbits, #manpages

Subscribers: imp

Differential Revision: https://reviews.freebsd.org/D33610

Import bmake-20211212

Relevant changes see ChangeLog for details:

Add support for ${.SUFFIXES} as read-only variable.

Fix memory leak in cond.c

Punt on write errors - ENOSPC etc.

Lots of code refactoring, rework to reduce memory allocations.

More unit-tests.

Test various header files to ensure they can be included by themselves.

A number of header files in sys/* have, going back to 7th Edition Unix
in 1979, reqiured other files (like sys/types.h) to compile. Likewise
the 4BSD networking code has had prerequisites. However, going back to
around the turn of the 21st century, other systems have made them be
independently include-able (wide-spread header include protection
post-dates 7th edition Unix by maybe 3 or so years judging from USENET
source postings). Start down the path of making them all independently
include-able by creating this test that fails buildworld when they are
not.

The file 'badfiles.inc' contains a list of the currently broken files
that cannot be included w/o any prerequisites. As files are fixed, 'make
badfiles.inc' should be re-run to remove them from the list. Note: All
files that start with an underscore are considered internal and not
tested.

Please note: once a file is removed from badfiles.inc, it must pass on
all architectures. Buildworld through at least the _includes target is
needed to ensure its working (though a buildkernel should also be done
on all architectures as well).

Sponsored by: Netflix
Reviewed by: brooks, markj
Differential Revision: https://reviews.freebsd.org/D32498

bsddialog: import snapshot as of 2021-11-24

Remove mips from universe

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D32851

mandoc: import v1.14.6

Import tzdata 2021c

Import tzdata 2021b

dma: import git snapshot 2021-07-10

Revert "Fix native-xtools build"

This reverts commit 36269b82318280ef184c953b90674f5905e0f53f.

This had an unintended change included.

Fix native-xtools build

Fixes https://github.com/freebsd/poudriere/issues/894
Fixes: d0c737e18 ("Makefile: Fix MAKEOBJDIRPREFIX command-line")
X-MFC-With: d0c737e18

Makefile: Fix MAKEOBJDIRPREFIX command-line variable check for bmake

Unlike the old fmake, running make FOO=bar when using bmake doesn't put
FOO=bar in .MAKEFLAGS at the top level, it instead just puts FOO in
.MAKEOVERRIDES and the full MAKEFLAGS will be formed for sub-makes.
Moreover, this only applies for sub-makes in rules, so this doesn't
apply to those in shell assignments. This means that the current check
does not catch make MAKEOBJDIRPREFIX=..., only those defined in config
files. Thus we must also check .MAKEOVERRIDES explicitly.

Reviewed by: sjg
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D31015

Import Arm Optimized Routines v21.02

We already have copies of some of these files in the repository without a
vendor import and we should really be using contrib/ instead.
We should also be able to use some of the math functions to allow the
tests to pass on AArch64 (and other architectures) instead of just x86.
We should also be able to reuse some of the tests for the kyua testsuite.

Imported using
```
curl -L https://github.com/ARM-software/optimized-routines/tarball/e823e3abf5f89ecba58a10fc0fd82c13d9984b6b | tar --strip-components=1 -xvzf -
git add .
```

Remove 'make update'.

In the CVS days this used be a wrapper around either CVS or CVSup and
used to support updating src, doc, and ports checkouts. With the move
to subversion this only supported updating src and was itself a
wrapper around 'svn update'. With Git, users are probably better off
using appropriate Git commands directly to update without needing an
explicit make target as a wrapper.

Reviewed by: bcr, imp, emaste
Differential Revision: https://reviews.freebsd.org/D30736

libedit: vendor import snapshot 20200710

Fix 'make bmake' top-level bootstrapping.

Fixes: ee10666327b62

Rename NO_WERROR -> MK_WERROR=no

As suggested in D27598. This also supports MK_WERROR.clang=no and
MK_WERROR.gcc=no to support the existing NO_WERROR.<compiler> uses.

Reviewed By: brooks
Differential Revision: https://reviews.freebsd.org/D27601

import terminus-font-4.48

Import tzdata 2020f

Import lua 5.4.2

Import zstd 1.4.8

Import tzdata 2020e

Import bmake-20201117

o allow env var MAKE_OBJDIR_CHECK_WRITABLE=no to skip writable
checks in InitObjdir. Explicit .OBJDIR target always allows
read-only directory.

o Fix building and unit-tests on non-BSD.

o More code cleanup and refactoring.

o More unit tests

Makefile: re-wordsmith the blurb about xtoolchain ports

The new version only includes a specific version once, and uses the one
that's currently advised by tinderbox: -gcc6.

It also advises just installing the pkg, but mentions in a side-note at the
end where to find the source in the ports tree.

Reviewed by: jrtc27
Suggested by: jhb (use default from tinderbox)
Differential Revision: https://reviews.freebsd.org/D26820

Import bmake-20201101

Lots of new unit-tests increase code coverage.

Lots of refactoring, cleanup and simlpification to reduce
code size.

Fixes for Bug 223564 and 245807

Updates to dirdeps.mk and meta2deps.py

pkgbase: Add incremental packages

This adds a new target update-packages which will create the new packages
compared to the last run.

This is how to use it:
At this point we cut a release
$ make buildworld ...
$ make buildkernel
$ make packages

There is now a PKG_VERSION directory with latest link pointing to it
Distribute the packages to server

$ something something that update the source tree
$ make buildworld ...
$ make buildkernel
$ make update-packages
You know have a PKG_VERSION directory in the REPODIR and latest link pointing to it.
In PKG_VERSION dir only the packages which differs from the latest run are
named PKG_VERSION, otherwise the old packages are there.

The process is :
Build the new packages in the PKG_VERSION directory
Compare the internal data with the PKG_VERSION_FROM version. The comparison is done
by checking the internal hash of the packages.
By default PKG_VERSION_FROM is set to what the latest link points to.
If the old and new version matches, we rm the new package and cp the old one.

Differential Revision: https://reviews.freebsd.org/D25984

Import tzdata 2020d

Import tzdata 2020c

Makefile: add a small blurb about building with gcc xtoolchain

The key details are to install the appropriate flavor of devel/freebsd-gcc9
and pass CROSS_TOOLCHAIN while building.

Suggested by: kib

lua: update to 5.3.6

This release contains some minor bugfixes; notably:
- 2x minor Makefile fixes (not used in base)
- Long brackets with a huge number of '=' overflow some internal buffer
arithmetic.
- Joining an upvalue with itself can cause a use-after-free crash.

See here for examples: http://www.lua.org/bugs.html#5.3.5

Eliminate building LINT makefiles

LINT config files are about to be checked in directly. Eliminate
building them by hand here from NOTES files.

Reviewed by: kevans
Differential Revision: https://reviews.freebsd.org/D26540

Import tzdata 2020b

[PowerPC64LE] Set up powerpc.powerpc64le architecture

This is the initial set up for PowerPC64LE.

The current plan is for this arch to remain experimental for FreeBSD 13.

This started as a weekend learning project for me and kinda snowballed from
there.

(More to follow momentarily.)

Reviewed by: imp (earlier version), emaste
Sponsored by: Tag1 Consulting, Inc.
Differential Revision: https://reviews.freebsd.org/D26399

Import bmake-20200902

Lots of code refactoring, simplification and cleanup.
Lots of new unit-tests providing much higher code coverage.
All courtesy of rillig at netbsd.

Other significant changes:

o new read-only variable .SHELL which provides the path of the shell
used to run scripts (as defined by the .SHELL target).

o new debug option -dl: LINT mode, does the equivalent of := for all
variable assignments so that file and line number are reported for
variable parse errors.

Fix typo in r364325 that broke tinderbox with -DBUILD_WITH_STRICT_TMPPATH

${TARGET_ARCH} is empty here which results in empy MAKE_PARAMS being
passed to the buildkernel phase. This breaks the build when using the
strict TMPPATH since cc will not be included in $PATH.

Reviewed By: jhb

Add a USE_GCC_TOOLCHAINS knob to make universe.

This uses GCC toolchains instead of LLVM on architectures supported by
GCC. Currently this uses GCC 6 on aarch64, amd64, i386, and mips.

Although this does also try to use GCC 6 on powerpc, it is always
skipped for now since a powerpc-gcc6 package is not available and the
structure of make universe makes it hard to skip a subset of arches
for a target. This should be short-lived as freebsd-gcc9 does include
a powerpc-gcc9 package so powerpc should work once this switches to
GCC 9.

Discussed with: emaste
Differential Revision: https://reviews.freebsd.org/D25732

Import bmake-20200606

Relevant items from ChangeLog:

o dir.c: cached_stats - don't confuse stat and lstat results.
o var.c: add :Or for reverse sort.

Import Zstd 1.4.5

Import bmake-20200517

Changes since 20181221 are mostly portability related
hence the large gap in versions imported.

There are however some bug fixes, and a rework of filemon handling.
In NetBSD make/filemon/filemon_ktrace.c allows use of fktrace
and elimination of filemon(4) which has not had the TLC it needs.

FreeBSD filemon(4) is in much better shape, so bmake/filemon/filemon_dev.c
allows use of that, with a bit less overhead than the ktrace model.

Summary of changes from ChangeLog

o str.c: empty string does not match % pattern
plus unit-test changes
o var.c: import handling of old sysV style modifier using '%'
o str.c: refactor brk_string
o meta.c: meta_oodate, CHECK_VALID_META is too aggressive for CMD
a blank command is perfectly valid.
o meta.c: meta_oodate, check for corrupted meta file
earlier and more often.
* meta.c: meta_compat_parent check for USE_FILEMON
patch from Soeren Tempel
o meta.c: fix compat mode, need to call meta_job_output()
o job.c: extra fds for meta mode not needed if using filemon_dev
o meta.c: avoid passing NULL to filemon_*() when meta_needed()
returns FALSE.
o filemon/filemon_{dev,ktrace}.c: allow selection of
filemon implementation. filemon_dev.c uses the kernel module
while filemon_ktrace.c leverages the fktrace api available in
NetBSD. filemon_ktrace.c can hopefully form the basis for
adding support for other tracing mechanisms such as strace on
Linux.
o meta.c: when target is out-of-date per normal make rules
record value of .OODATE in meta file.
o parse.c: don't pass NULL to realpath(3)
some versions cannot handle it.
o parse.c: ParseDoDependency: free paths rather than assert

plus more unit-tests

Import lib9p 7ddb1164407da19b9b1afb83df83ae65a71a9a66.

Approved by: trasz (mentor)
MFC after: 1 month
Sponsored by: Conclusive Engineering

Use universe-toolchain config(8)

This is a temporary hack to aid with config(8) changing in r360443.
It will not work for all cases.

env PATH is used because universe-toolchain is only built when worlds
are built, and then only if clang is needed, so it may not exist.

universe-toolchain needs to be expanded to always be built, inspected to
remove non-cross-build-safe tools, used for buildworld/buildkernel,
and potentially incremental build support.

Sponsored by: Dell EMC

Import tzdata 2020a

Add powerpcspe to the EXTRA_TARGETS

Currently, powerpcspe is broken with clang. Add it to the EXTRA_TARGETS until
that's fixed.

Reviewed by: brooks, bdrewery, emaste (LGTM)
MFC After: 3 days
Differential Revision: https://reviews.freebsd.org/D24212

Fix make kernels to match original commit message

make kernels was originally documented (in commit r295099) as the same as make
universe -DJUST_BUILD_KERNELS. However, it used the UNIVERSE_TARGET=buildkernel
which is subtly different in that it also builds the toolchains and doesn't
build the LINT modules unless they happened to already exist in the tree. This
unbreaks POLA and just builds the kernels, including LINT, now rather than all
that other stuff as well. When the commit originally happened, the two just
differed by the LINT bug. r335711 introduced the universe dependency on the
toolchain that wasn't present before, which diverged the two further. This
restores the original intent of r295099.

Reviewed by: brooks, bdrewery, emaste (LGTM)
MFC After: 3 days
Differential Revision: https://reviews.freebsd.org/D24212

Make universe configuration more consistent with rest of system

Add 'WITHOUT_WORLDS' and 'WITHOUT_KERNELS' as aliases for the inconsistently
named MAKE_JUST_KERNELS and MAKE_JUST_WORLDS respectively. I always forget the
MAKE_ part (or is it BUILD_), and it's inconsistent with everything
else. Document the new things, but leave speculation of any eventual MAKE_JUST_*
deprecation out of the manuals and comments.

Reviewed by: brooks, bdrewery, emaste (LGTM)
MFC After: 3 days
Differential Revision: https://reviews.freebsd.org/D24212

Disable rarely used architecture variants in make universe by default.

If EXTRA_TARGETS is defined, build all supported architecture
variants. By default, build architecture variants needed to provide
code coverage or that are commonly used.

Use this to disable building of all the hard-float and little-endian
MIPS architecture variants along with n32 by default.

Reviewed by: rpokala
Discussed with: imp, emaste
Differential Revision: https://reviews.freebsd.org/D24178

Use Clang and LLD as the default toolchain for MIPS

Now that we have updated the in-tree version of LLVM to 10.0, we have all the
necessary LLVM changes to use Clang+LLD as the default toolchain for MIPS.

Relnotes: yes
Reviewed By: emaste, jhb, brooks, kevans
Differential Revision: https://reviews.freebsd.org/D23204

Per the firm plan, start to remove sparc64

The sparc64 architecture is being removed from FreeBSD 13, starting
now. This removes it from the top level only. It is the only
architecture that didn't see substantial work after the call to get
things working with the external toolchain.

Use clang and lld as the default toolchain for RISCV.

- Enable clang and lld as system toolchains.
- Don't use external GCC for universe by default.
- Re-enable riscv64sf since it builds fine with clang + lld.

Reviewed by: emaste, mhorne
Relnotes: yes
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D23089

Don't fail universe kernel stage for TARGET_ARCHes without a kernel config.

This fixes a regression in r356418 where the entire universe would
fail early due to an undefined make target when a given TARGET_ARCH
had no associated kernel configs. This is true for all of the
hard-float mips TARGET_ARCHes currently.

Pointy hat to: me
Reviewed by: emaste
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D23071

Remove support for using GCC 4.2.1 from make tinderbox.

Reviewed by: emaste
Differential Revision: https://reviews.freebsd.org/D23054

Enable powerpc in make tinderbox using clang instead of GCC.

powerpcspe is disabled for now until clang/llvm issues with spe
have been fixed.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D23031

Only build kernels for enabled TARGET_ARCHes in make universe/tinderbox.

Previously, all of the kernels for a given TARGET were built if that
target was enabled. This was implemented by having each kernel built
via a universe_kernconf_<KERNEL> target that was depended on by a
universe_kernconfs target. However, this meant that if one did a
build with a limited set of TARGET_ARCH values for a given TARGET,
kernels could be built for which we hadn't built a world or toolchain.
For example, 'make TARGETS=mips TARGET_ARCHES_mips=mips64' would build
mips32 kernels.

Fix this by adding an extra layer of indirection in the kernel make
targets. universe_kernconf_<KERNEL> is now a dependency of a new
universe_kernconfs_<TARGET_ARCH>. universe_kernconfs in turn depends
on a list of universe_kernconfs_<target_arch> values, but only the
values enabled in TARGET_ARCHES_<TARGET>.

Reviewed by: imp
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D23031

Use the freebsd-gcc6 toolchains instead of powerpc64-xtoolchain-gcc.

Reviewed by: brooks
Differential Revision: https://reviews.freebsd.org/D23029

Add the missing trailing '/' when setting TARGET_ARCH from TARGET

This should fix the build when setting TARGET but not TARGET_ARCH.

Sponsored by: DARPA, AFRL

Remove arm/arm as a valid target.

TARGET=arm now defaults to TARGET_ARCH=armv7
TARGET_ARCH=arm is no longer valid.

Bump __FreeBSD_version to 1300073

Tested with make universe. Any stale LINT-V5 config files remaining in the tree
will fail the universe build. However, LINT-V5 was removed in r355119.

This retirement has been planned since last summer. The armv5 port is fragile:
it works OK for some peeople, and fails badly for others. There's a number of
subtle bugs in busdma, pmap and other MD parts of thee system that present
themselves under load or in unusual circumstances (like fsck after a
crash). stable/8, branched 10 years ago, was the last reliable release. Since
the support burden is larger then the benefit, the consensus view is armv5
should be removed from the tree.

Discussed with: arm@ mailing list and arm developer community.

Import openresolv 3.9.0

More information at: https://roy.marples.name/projects/openresolv

Import Zstd 1.4.4

Take arm.arm (armv5) out of universe

It's on the chopping block in two months, the CI tinderbox doesn't bother with
it anymore either, and buildworld fails today due to an issue linking clang.
It's not worth investigating and it just eats up CPU cycles running universe
builds.

Disconnect powerpc from the default tinderbox for now.

The wrong toolchain was set in MAKE_PARAMS_powerpc, however, there are
some other issues preventing powerpc from building in tinderbox:

1) There is no powerpc-gcc and powerpc-xtoolchain-gcc port that
provides an external 32-bit powerpc GCC toolchain.

2) On other targets, the same toolchain can build all of the
architectures for a given target. MIPS achieves this by always
setting -mabi and -EB/-EL explicitly instead of relying on the
compiler's default architecture. PowerPC might be able to do the
same thing, but as of today, powerpc-gcc would be required for
powerpc and powerpcspe and powerpc64-gcc would be required for
powerpc64. Our existing logic for make universe does not permit
per-MACHINE_ARCH toolchains.

I tried hacking TARGETS_powerpc to only include powerpc64 when
powerpc64-gcc was present, and while that skipped the 32-bit worlds,
it tried to build all the kernels.

Reported by: jeff
Discussed with: imp

Wordsmith and simplify

Simplify expressions as suggested by jhb. The extra indirection made
sense in earlier versions of this patch, but not the final one.

While here, apply suggestion from emaste for wording of universe.
Also wordsmith awkwardly worded comment about when we effectively
neuter the universe build for an architecture.

Once llvm 9.0 has been vetted for mips and powerpc, I'll take them out
of these lists.

Don't compile old gcc 4.2.1 archs by default in universe/tinderbox.

Only compile clang supporting architectures of amd64, arm, arm64,
i386, and riscv as part of universe. Compile the other architectures
if MAKE_OBSOLETE_GCC is defined. In all cases, explicit lists of
architectures in TARGETS= on the command line override.

For mips, powerpc and sparc64, do the same thing we do for risvc when
MAKE_OBSOLETE_GCC isn't defined and short-circuit their universe build
with an echo saying to install the xtoolchain port or pkg.

PR: 241134
Discussed on: arch@ (https://lists.freebsd.org/pipermail/freebsd-arch/2019-August/019674.html)
Differential Revision: https://reviews.freebsd.org/D21942

Add a top-level makeman target

Abstracting away the details of how src.conf(5) is generated is arguably a
good thing; do so with a top-level makeman target.

Import DTS files from Linux 5.3

Vendor import bzip2 -- upgrade to version 1.0.8.

pkgbase: Add the sub stage-packages targets to TGTS

This helps when you don't want to run the dependencies targets.

Reviewed by: bapt
Differential Revision: https://reviews.freebsd.org/D20955

Import tzdata 2019b

Vendor import bzip2 -- upgrade to version 1.0.7.

Modernize the MAKE_JUST_KERNELS hint in the top-level makefile.

It doesn't make sense to limit to -j12 anymore, build scalability
is better than it used to be. Fold the hint into the description
of the universe target.

Reviewed by: imp
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D20342

import zstd 1.4.0

Fix a typo when sanity checking in the bootstrap-tools target

The path is incorrect for the linker in the error message. It should have been
/usr/bin/ld, not /usr/bin/cc .

Reviewed by: emaste
Approved by: emaste (mentor)
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D19852

Import mandoc 1.4.5

Import tzdata 2019a

Import bearssl-6433cc2

Import tzdata 2018h

import zstd 1.3.8

Update comment about 'universe' disk usage

It's 167 GB now (or 81 GB with ZFS lz4).

MFC after: 2 weeks
Sponsored by: Dell EMC Isilon

Add a top-level make target to rebuild all sysent files.

The sysent target is useful when changing makesyscalls.sh, when
making paired changes to syscalls.master files, or in a future where
freebsd32 sysent entries are built from the default syscalls.master.

Reviewed by: bdrewery
Obtained from: CheriBSD
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D17899

Allow building world without inheriting $PATH

Inheriting $PATH during the build phase can cause the build to fail when
compiling on a different system due to missing build tools or incompatible
versions somewhere in $PATH. This has cause build failures for us before
due to the jenkins slaves still running FreeBSD 10.
Listing the tools we depend on explicitly instead of just using whatever
happens to be in $PATH allows us to check that we don't accidentally add a
new build dependency.

All tools that do no need to be bootstrapped will now be symlinked to
${WORLDTMP}/legacy/bin and during the build phase $PATH will only contain
${WORLDTMP}. There is also a new variable "BOOTSTRAP_ALL_TOOLS" which can
be set to force compiling almost all bootstrap tools instead of symlinking
them. This will not bootstrap tools such as cp,mv, etc. since they may be
used during the build and for those we should really only be using POSIX
compatible options.

Furthermore, this change is required in order to be able to build on
non-FreeBSD hosts. While the same binaries may exist on Linux/MacOS they
often accept different flags or produce incompatible output.

Approved By: brooks (mentor)
Differential Revision: https://reviews.freebsd.org/D16815

import zstd 1.3.7

import zstd 1.3.4

import zstd 1.3.3

Import tzdata 2018f

Don't include the broken riscv64sf TARGET_ARCH in universe.

riscv64sf has been broken due to duplicate symbols for months and
degrades the quality of universe builds. Remove it until this is
resolved leaving a comment to it is not re-added.

PR: 232085
Reviewed by: emaste
Approved by: re (gjb, kib)
Sponsored by: DARPA, AFRL

import mandoc 1.14.1

Import OpenSSL 1.1.1.

Import OpenSSL 1.0.2p.

As discussed several times on freebsd-arch, start to decommission armeb.

Remove armeb/arm from KNOWN_ARCHS.
Remove armeb from arm universe targets.

Differential Revision: https://reviews.freebsd.org/D16257

tinderbox: If the clang lookup fails fallback to the old default behavior.

This fixes errors from the MK_CLANG_BOOTSTRAP/MK_LLD_BOOTSTRAP lookups
to not force using XCC/XLD but to rather just build them as normal by
allowing their own bootstrap logic to work.

MFC after: 3 weeks
X-MFC-with: r335711 r335769
Sponsored by: Dell EMC

Include existing MAKE_PARAMS_<target> when determining the universe toolchain.

This fixes a warning for each RISCV target during universe by passing in
the required CROSS_TOOLCHAIN setting which will in turn set
CROSS_BINUTILS_PREFIX correctly. It also ensures that a tinderbox build
uses the correct compiler for riscv. Previously it was using the shared
clang compiler instead of riscv64-gcc.

Reviewed by: bdrewery
Sponsored by: DARPA / AFRL
Differential Revision: https://reviews.freebsd.org/D16049

Fix typo in top-level Makefile

Submitted by: Ben Widawsky <ben.widawsky@intel.com>
MFC after: 3 days
X-MFC-to: stable/11 stable/10
Sponsored by: Smule, Inc.
Differential Revision: https://reviews.freebsd.org/P186

tinderbox: Only build clang/lld once if needed.

Need to handle LLD_BOOTSTRAP separately (for archs like i386).
This would be much better off with an off-by-default option like
SHARED_TOOLCHAIN that universe force-enabled. Then a normal buildworld
would store the toolchain there if enabled and otherwise in WORLDTMP
with only the 1 arch selected.

MFC after: 3 weeks
Sponsored by: Dell EMC

tinderbox: Give details about kernel builds.

This is a bit noisy now but it was silent before leading to
wondering if it was doing anything.

MFC after: 1 week
Suggested by: rpokala
Sponsored by: Dell EMC

Only look for NOTES as needed.

Sponsored by: Dell EMC

Add WITH_SYSTEM_LINKER, on by default, that avoids building lld when possible.

This works similar to WITH_SYSTEM_COMPILER added in r300354. It only
supports lld via WITH_LLD_BOOTSTRAP.

When both SYSTEM_COMPILER and SYSTEM_LINKER logic passes then libclang
will not build in cross-tools. If either check fails though then
libclang is built.

The .info is reworked to notify when libclang will be built since if
either clang or lld needs to be rebuilt, but not the other, the
notification can lead to confusion on why "clang is building".

-fuse-ld= is not used with this method so some combinations of compiler
and linker are expected to fail.

A new 'make test-system-linker' target is added to see the logic results.

Makefile.inc1:
CROSS_BINUTILS_PREFIX support had to be moved higher up so that XLD
could be set and MK_LLD_BOOTSTRAP disabled before checking SYSTEM_LINKER
logic as done with SYSTEM_COMPILER. This also required moving where
bsd.linker.mk was read since XLD needs to be set before parsing it. This
creates a situation where src.opts.mk can not test LINKER_FEATURES or
add LLD_BOOTSTAP to BROKEN_OPTIONS.

Reviewed by: emaste (earlier version)
Sponsored by: Dell EMC
Differential Revision: https://reviews.freebsd.org/D15894

Restore arm, riscv, sparc64, and mips to UNIVERSE after r334128

libpmcstat: compile in events based on json description

Import tzdata 2018e

Import ELF Tool Chain snapshot at r3614

From http://svn.code.sf.net/p/elftoolchain/code/

Import OpenSSL 1.0.2o.

Import tzdata 2018d

Don't read SRC_ENV_CONF for MAKEOBJDIRPREFIX guard.

This is mostly to allow using MAKEOBJDIRPREFIX in src-env.conf on stable where
src.sys.obj.mk is not going to be MFC'd. It is still valid on head but
effectively a NOP due to MAKEOBJDIRPREFIX being handled differently in
src.sys.obj.mk.

Reported by: eadler
MFC after: 1 week
Sponsored by: Dell EMC

Add a note about why we have the conditional before including
bsd.compiler.mk. It's so fmake from older 9.x systems still
works (still a supported build config, and having the note here
will let us know when we can cull it more easily).

Also pull in a related change from include to sinclude from
arichardson@'s cross building work, as well as it's companion in
Makefile.inc1 with a note about why we do the odd thing there.

Submitted by: archardson
Differential Revision: https://reviews.freebsd.org/D14241

crossbuild: Make the CHECK_TIME variable work on Linux

Linux /usr/bin/find doesn't understand the -mtime -0s flag.
Instead create a temporary file and compare that file's mtime to
sys/sys/param.h to check whether the clock is correct.

Reviewed By: jhb, imp
Approved By: jhb (mentor)
Differential Revision: https://reviews.freebsd.org/D14157

Import tzdata 2018c

Create vendor-sys/libfdt from libfdt in vendor/dtc

There are no plans at the moment to continue updating GPL dtc in the tree as
BSDL dtc is slowly replacing it.

We use libfdt in the kernel and loaders, so fork off the libfdt bits from
vendor/dtc into vendor-sys/libfdt rather than continuing to update
vendor/dtc when we have no intention of updating anything in userland as a
result of these imports.

This also leaves us with a vendor branch whose content has a consistent
license- the entirety of libfdt is dual license, GPL/BSD, although that was
not a primary concern.

Import tzdata 2018a

Import dtc 1.4.6

Import OpenSSL 1.0.2n.

tinderbox/universe: Don't execute KERNCONFS lookup if not needed.

Sponsored by: Dell EMC

Revert r325529: No longer needed after r325699.

Sponsored by: Dell EMC

Import Lua 3.5.4

Downloaded from https://www.lua.org/ftp/lua-5.3.4.tar.gz

Use TARGET_ARCH=riscv64 when TARGET=riscv

The supported targets are riscv64 and riscv64sf. Use the former when
building with a bare TARGET=riscv and it is the more common one.

Sponsored by: Netflix

Move top-level AUTO_OBJ logic to Makefile.sys.inc.

Sponsored by: Dell

Remove build system support for lint.

Differential Revision: https://reviews.freebsd.org/D13124

Rework r325568 so all 'make LINT' targets work.

Reported by: ngie
Sponsored by: Dell EMC Isilon

META_MODE: Avoid some logic if disabled and allow enabling with make argument.

Sponsored by: Dell EMC Isilon

AUTO_OBJ: Disable in non-build targets.

There's no reason to create object directories for targets like 'installworld'
or 'distributeworld', and the others in this list. Specifying MK_AUTO_OBJ as a
make argument allows circumventing this if needed for some reason.

This fixes mergemaster creating a full object tree due to doing a 'make
installconfig' tree walk.

Reported by: Mark Millard
Sponsored by: Dell EMC Isilon

Remove rerelease target that was added in r38978.

I'm not sure what this target is but it doesn't exist now and doesn't seem
to have existed before.

Sponsored by: Dell EMC Isilon

native-xtools: Fix for GCC archs.

- This also adds in a _cleanobj step as needed.
- This redirects TARGET/TARGET_ARCH to NXB_TARGET/NXB_TARGET_ARCH in
Makefile.inc1 as the main build needs to be for MACHINE rather
than TARGET.

First build the toolchain and then use that as an external toolchain
to build the needed directories and NXB_TARGET-toolchain, all as
MACHINE files though via TARGET_TRIPLE=MACHINE_TRIPLE.

The NXBDIRS is evaluated in the 'everything' submake as it needs to be
based on TARGET's src.opts.mk values, such as MK_GCC=yes when building
on a MK_CLANG=yes MACHINE. This can likely be changed to a specific
_native-xtools-everything target later and the funky late evaluation
of SUBDIR_OVERRIDE removed.

X-MFC-With: r325001
Pointyhat to: bdrewery
MFC after: 2 months
Sponsored by: Dell EMC Isilon

Deal with src.conf for top-level MAKEOBJDIRPREFIX guard.

- Don't discard SRCCONF value since it may incorrectly have MAKEOBJDIRPREFIX
in it.
- Add note about src.conf not being a suitable place for MAKEOBJDIRPREFIX.

Sponsored by: Dell EMC Isilon

import zstd 1.3.2

universe: Fix creating LINT files with AUTO_OBJ.

These are expected to be created in .CURDIR.

Reported by: kib
Sponsored by: Dell EMC Isilon

AUTO_OBJ: Don't create TARGET. directories during 'make universe'.

Reported by: rpokala
Sponsored by: Dell EMC Isilon

Simplify location of bootstrap make.

Sponsored by: Dell EMC Isilon

AUTO_OBJ is unsafe at the moment as it is for some reason removing /etc with _cleanobj.

Something is very wrong

Import OpenSSL 1.0.2m.

Import bmake-20171028

From ChangeLog

2017-10-28 Simon J. Gerraty <sjg@bad.crufty.net>

* VERSION: 20171028
Merge with NetBSD make, pick up
o main.c: ignore empty MAKEOBJDIR

* Makefile.config.in:
make @prefix@ @machine*@ and @default_sys_path@ defaults.

2017-10-05 Simon J. Gerraty <sjg@bad.crufty.net>

* VERSION: 20171005

* unit-tests/dotwait.mk: redirect stderr through pipe for more
consistent result on some platforms.

2017-08-13 Simon J. Gerraty <sjg@bad.crufty.net>

* machine.sh: entry for AIX

2017-08-12 Simon J. Gerraty <sjg@bad.crufty.net>

* VERSION (_MAKE_VERSION): Move the setting of _MAKE_VERSION
to a file that can be included by configure as well as make.
This allows configure to set set _MAKE_VERSION in make-bootstrap.sh

2017-08-10 Simon J. Gerraty <sjg@bad.crufty.net>

* Makefile (_MAKE_VERSION): 20170810
Merge with NetBSD make, pick up
o meta.c: if target is in subdir we only need subdir name in
meta_name.

Add a 'make cleanuniverse'.

This will remove all build files for the source directory
when MK_UNIFIED_OBJDIR is enabled.

Sponsored by: Dell EMC Isilon

Suggest TARGET/TARGET_ARCH, XDEV/XDEV_ARCH are depcrecated in r269031.

MFC after: 2 weeks
Sponsored by: Dell EMC Isilon

Change native-xtools to not install by default; add a native-xtools-install.

Without this the user has to mess with 'make -f Makefile.inc1 ...' to figure
out where the files are installed in the OBJDIR and then they need to copy them
to where they really wanted them. Using DESTDIR may be problematic after
r325001 as well.

The files will be installed to DESTDIR/NXTP where NXTP defaults to /nxb-bin.

MFC after: 2 weeks
Sponsored by: Dell EMC Isilon

build(7): Document native-xtools.

MFC after: 2 weeks
Sponsored by: Dell EMC Isilon

Fix xdev TARGET/TARGET_ARCH assertion and expand to native-xtools.

The top of Makefile.inc1 requires TARGET/TARGET_ARCH be defined. Just
building 'make xdev' would already set them, so this error was never
triggered. Moving it to Makefile fixes the problem.

MFC after: 2 weeks
Sponsored by: Dell EMC Isilon

Disable AUTO_OBJ as a make argument to ensure it really is disabled.

If the user sets this to yes as a make argument too we still want
to ensure these do not get enabled.

Sponsored by: Dell EMC Isilon

Import tzdata 2017c

Add appropriate safety belts for riscv so it compiles out of the box when MK_COVERAGE ==yes

Import mandoc 1.14.3

Support armv7 builds for userland

Make armv7 as a new MACHINE_ARCH.

Copy all the places we do armv6 and add armv7 as basically an
alias. clang appears to generate code for armv7 by default. armv7 hard
float isn't supported by the the in-tree gcc, so it hasn't been
updated to have a new default.

Support armv7 as a new valid MACHINE_ARCH (and by extension
TARGET_ARCH).

Add armv7 to the universe build.

Differential Revision: https://reviews.freebsd.org/D12010

Import libedit 2017-09-05

Obtained from: NetBSD

Import ctfdump from OpenBSD (ISC license)

For now import it from my own github repository which is a fork
of mpi@OpenBSD's version for portability issues. This will be switched
to mpi's version once portability issues will have been addressed

Discussed with: mpi@
Obtained from: OpenBSD

Import zstd 1.3.1

Quote ${MAKE} when passing in env in case it contains spaces.

Downstream we are wrapping MAKE with a limits(1) call which
interferes with these non-quoted cases.

Sponsored by: Dell EMC Isilon

Make _TO_CPUARCH macro for ARCH to CPUARCH conversions

Consolidate all the regular expressions to convert from MACHINE_ARCH
to MACHINE_CPUARCH into a variable and use that variable in preference
to the almost identical copies in the tree (which should have been
identical).

Differential Revision: https://reviews.freebsd.org/D11986

Import mandoc 1.14.2

genericize target exclusion for missing external toolchain

Previously we excluded riscv from make universe / tinderbox if the
required xtoolchain package was not installed. Make that logic generic
so that we can loop over multiple architectures, in preparation to test
patches to have other architectures rely on external toolchain.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D11652

Add some .ORDER for distrib-dirs, distribute, distribution and distributeworld.

Reported by: Mark Millard
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon

Import bmake-20170720

compat.c: pass SIGINT etc onto child and wait for it to exit
before we self-terminate.

Import bmake-20170711

Several misc cleanups, refactor main() to reduce size.
Adds -v option to always fully expand variables (only interesting to NetBSD)
And in particular (from ChangeLog):

o meta.c: ensure command output in meta file has ending newline
even when filemon not being used.
When matching ${.MAKE.META.IGNORE_PATTERNS} do not use
pathname via ':L' since any ':' in pathname breaks that.
Instead set a '${.p.}' to pathname in the target context and
use that.

Import zstd 1.3.0

Update the DTS file from Linux 4.12

Expose only the create-packages-* targets since they set needed DEST/DIRDIR.

The other targets just fail confusingly otherwise.

MFC after: 2 weeks
Sponsored by: Dell EMC Isilon

Allow ALWAYS_BOOTSTRAP_MAKE to force bmake bootstrapping.

MFC after: 2 weeks
Sponsored by: Dell EMC Isilon

By popular demand: change MAKE_GENERIC_KERNELS to MAKE_LINT_KERNELS.

It appears that the same arches that lack GENERIC kernel configs also lack
LINT. But enough different arches get built to ensure a kernel change
should build everywhere (32 and 64 bit, clang and old gcc, little and big
endian).

Add support for "make universe_kernels -DMAKE_GENERIC_KERNELS" to build
just the GENERIC kernels for each arch (including variations such as
GENERIC-NODEBUG, GENERIC64, etc).

This helps with quickly doing a test build for all[*] arches without
building dozens of variant kernels for the arches that have lots of
hardware/board/system variations.

[*] Not all arches have a generic kernel (but they probably should for
test-building purposes, even if it can't boot on any real hardware).

Import mandoc 2017-06-08

META_MODE: Add framework to force rebuilding for major ABI changes.

Normally META_MODE ignores host files for "meta mode" decisions on whether a
file should be rebuilt or not. This is because a simple installworld can
update timestamps and cause the next build to rebuild all host tools, when the
previous ones may not have any changes in the source tree. These tools are
normally still ABI compatible. They are only rebuilt if NO_META_IGNORE_HOST is
set from the workaround/hack in r301467.

One of the major problems with this is when a host tool has objects spread
across many revisions that have mixed-ABI. For example, if struct stat were to
change on the host, some objects for a tool may have different ideas of that
struct's definition. If just 1 source file were modified and rebuilt and
linked into the tool, then that toll will have mixed-ABI objects and crash.
This exact thing happened with the ino64 commit in r301467 followed by a
trivial update to libbfd in r318750. The resulting binary would crash in
buildworld.

Sponsored by: Dell EMC Isilon

Import OpenSSL 1.0.2l.

Import bmake-20170510

Import Zstandard 1.2.0
Among changes: threading support is now default and supports freebsd

Import bmake-20170420

Only use -C arg "as is" if it contains no relative component.
Fix Main_SetObjdir() for relative paths (eg obj).
Fix a number of coverity complaints.

Import zstd 1.1.4

Import bmake-20170413

Cache compiler metadata and reuse it at installworld time.

Right after cross-tools, a compiler-metadata.mk file is created that
stores all of the bsd.compiler.mk metadata. It is then read in
with a fail-safe during installworld time.

The file is explicitly removed when invoking cross-tools to ensure that
a stale file is not left around from odd manual 'make _cross-tools' ->
'make installworld' invocations.

This fixes several issues:
- With WITH_SYSTEM_COMPILER (default yes on head and no on releng/11.0):
If you build on a system where the bootstrap compiler does not
build due to the host compiler matching the in-tree one, but then
installworld on another system where that logic fails (a
bootstrap compiler is needed), the installworld immediately fails
with:
sh: cc: not found
Note that fixing this logic may then hit a case where a rebuild is
attempted in installworld. Normally cc would be ran with
'CFLAGS+=ERROR-tried-to-rebuild-during-make-install' to cause an
error such as:
cc: error: no such file or directory: 'ERROR-tried-to-rebuild-during-make-install'
However, now it will just fail with the 'cc: not found' error.
Inspection of the compile line will show
'ERROR-tried-to-rebuild-during-make-install'; It's not useful to
set CC to anything other than 'cc' during install as it is more
helpful to see the attempted compile rather than some other bogus
error.
- This now avoids running bsd.compiler.mk (cc executions) even more
during installworld. There are compiler-dependent SUBDIR in the
tree which required having a compiler during install.

There is at least 1 case where CC is still executed in the install,
such as from a LOOKUP!= in secure/lib/libcrypto/Makefile.inc checking
for 'vzeroall' support. This is not significant for installworld
as the lookup has a fallback (and hides its error) and only modifies CFLAGS,
thus it's not worth fixing.

PR: 212877
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon

Include RISC-V target to universe build.

Check if RISC-V external toolchain package is installed,
otherwise skip build.

Reviewed by: emaste
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D10344

Unconditionally include arm64 in make universe / tinderbox

As of r316629 FreeBSD/arm64 uses the in-tree LLD linker by default, and
does not require an external an aarch64-binutils port or package.

Sponsored by: The FreeBSD Foundation

Import tzdata 2017b

Import dtc 1.4.3

Obtained from: https://www.kernel.org/pub/software/utils/dtc/dtc-1.4.3.tar.gz

Import bmake-20170301

Import mandoc 1.4.1rc2

Import libedit 2016-03-21

Obtained from: NetBSD

Import openresolv 3.9.0

Obtained from: http://roy.marples.name/projects/openresolv

Remove pc98 support completely.
I thank all developers and contributors for pc98.

Relnotes: yes

Import OpenSSL 1.0.2k.

Import mandoc cvs snapshot 20170121 (pre 1.14)

Add missing files in zlib 1.2.8.

Vendor import of compiler-rt trunk r290819:
https://llvm.org/svn/llvm-project/compiler-rt/trunk@290819

Import bmake-20161212

From ChangeLog
2016-12-12 Simon J. Gerraty <sjg@bad.crufty.net>

* Makefile (_MAKE_VERSION): 20161212
Merge with NetBSD make, pick up
o main.c: look for obj.${MACHINE}-${MACHINE_ARCH} too.

2016-12-09 Simon J. Gerraty <sjg@bad.crufty.net>

* Makefile (_MAKE_VERSION): 20161209
Merge with NetBSD make, pick up
o main.c: cleanup setting of .OBJDIR
o parse.c: avoid coredump from (var)=val

2016-11-26 Simon J. Gerraty <sjg@bad.crufty.net>

* Makefile (_MAKE_VERSION): 20161126
Merge with NetBSD make, pick up
o make.c: Make_OODate: report src node name if path not set

2016-09-26 Simon J. Gerraty <sjg@bad.crufty.net>

* Makefile (_MAKE_VERSION): 20160926
Merge with NetBSD make, pick up
o support for .DELETE_ON_ERROR: (remove targets that fail)

2016-09-26 Simon J. Gerraty <sjg@bad.crufty.net>

* Makefile MAN: tweak .Dt to match ${PROG}

mk/ChangeLog

2016-12-12 Simon J. Gerraty <sjg@bad.crufty.net>

* install-mk (MK_VERSION): 20161212

* meta2deps.py: set pid_cwd[pid] when we process 'C'hdir,
rather than when we detect pid change.

2016-12-07 Simon J. Gerraty <sjg@bad.crufty.net>

* install-mk (MK_VERSION): 20161207

* meta.stage.mk: add stage_as_and_symlink for staging packages.
We build foo.tgz stage_as foo-${VERSION}.tgz but want to be able
to use foo.tgz to reference the latest staged version - so we
make foo.tgz a symlink to it.
Using a target to do both operations ensures we stay in sync.

2016-11-26 Simon J. Gerraty <sjg@bad.crufty.net>

* install-mk (MK_VERSION): 20161126

* dirdeps.mk: set DIRDEPS_CACHE before we include local.dirdeps.mk
so it can add dependencies.

2016-10-10 Simon J. Gerraty <sjg@bad.crufty.net>

* dirdeps.mk: set DEP_* before we expand .MAKE.DEPENDFILE_PREFERENCE
do that they can influence the result correctly.

* dirdeps.mk (${DIRDEPS_CACHE}): make sure we pass on TARGET_SPEC

* dirdeps.mk: Add ONLY_TARGET_SPEC_LIST and NOT_TARGET_SPEC_LIST
similar to ONLY_MACHINE_LIST and NOT_MACHINE_LIST

2016-10-05 Simon J. Gerraty <sjg@bad.crufty.net>

* dirdeps.mk: remove dependence on jot (normal situations anyway).
Before we read another Makefile.depend* set DEP_* vars from
_DEP_TARGET_SPEC in case it uses any of them with :=
When bootstrapping, trim any ,* from extention of chosen _src
Makefile.depend* to get the machine value we subst for.

2016-09-30 Simon J. Gerraty <sjg@bad.crufty.net>

* dirdeps.mk: use TARGET_SPEC_VARS to qualify components added to
DEP_SKIP_DIR and DEP_DIRDEPS_FILTER

* sys.mk: extract some bits to sys.{debug,vars}.mk
for easier re-use by others.

2016-09-23 Simon Gerraty <sjg@sjg-mba13>

* lib.mk: Use ${PICO} for extension for PIC objects.
default to .pico (like NetBSD) safe on case insensitive filesystem.

2016-08-19 Simon J. Gerraty <sjg@bad.crufty.net>

* meta.sys.mk (META_COOKIE_TOUCH): use ${.OBJDIR}/${.TARGET:T} as default

Import tzdata 2016j

Add full softfloat and hardfloat support for RISC-V.

Hardfloat is now default (use riscv64sf as TARGET_ARCH
for softfloat).

Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D8529

Import tzdata2016i.

Sponsored by: The FreeBSD Foundation

Add full softfloat and hardfloat support for MIPS.

This adds new target architectures for hardfloat:
mipselhf mipshf mips64elhf mips64hf.

Tested in QEMU only.

Sponsored by: DARPA, AFRL
Sponsored by: HEIF5
Differential Revision: https://reviews.freebsd.org/D8376

Create a new MACHINE_ARCH for Freescale PowerPC e500v2

Summary:
The Freescale e500v2 PowerPC core does not use a standard FPU.
Instead, it uses a Signal Processing Engine (SPE)--a DSP-style vector processor
unit, which doubles as a FPU. The PowerPC SPE ABI is incompatible with the
stock powerpc ABI, so a new MACHINE_ARCH was created to deal with this.
Additionaly, the SPE opcodes overlap with Altivec, so these are mutually
exclusive. Taking advantage of this fact, a new file, powerpc/booke/spe.c, was
created with the same function set as in powerpc/powerpc/altivec.c, so it
becomes effectively a drop-in replacement. setjmp/longjmp were modified to save
the upper 32-bits of the now-64-bit GPRs (upper 32-bits are only accessible by
the SPE).

Note: This does _not_ support the SPE in the e500v1, as the e500v1 SPE does not
support double-precision floating point.

Also, without a new MACHINE_ARCH it would be impossible to provide binary
packages which utilize the SPE.

Additionally, no work has been done to support ports, work is needed for this.
This also means no newer gcc can yet be used. However, gcc's powerpc support
has been refactored which would make adding a powerpcspe-freebsd target very
easy.

Test Plan:
This was lightly tested on a RouterBoard RB800 and an AmigaOne A1222
(P1022-based) board, compiled against the new ABI. Base system utilities
(/bin/sh, /bin/ls, etc) still function appropriately, the system is able to boot
multiuser.

Reviewed By: bdrewery, imp
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D5683

Import OpenSSL 1.0.2j.

Import OpenSSL 1.0.2i.

Remove generated files that shouldn't have been added

Reported by: cognet
Sponsored by: The FreeBSD Foundation

Import the Linaro Cortex Strings library from
https://git.linaro.org/toolchain/cortex-strings.git

Sponsored by: The FreeBSD Foundation

Import bmake-20160818

Interesting changes:

o meta.c: move handling of .MAKE.META.IGNORE_* to meta_ignore()
so we can call it before adding entries to missingFiles.
Thus we do not track files we have been told to ignore.
o meta_oodate: apply .MAKE.META.IGNORE_FILTER (if defined) to
pathnames, and skip if the expansion is empty.
Useful for dirdeps.mk when checking DIRDEPS_CACHE.
o meta.c: remove all missingFiles entries that match a deleted
dir.
o main.c: set .ERROR_CMD if possible.

Add a 'make print-dir' that simply traverses all directories and prints them.

This is useful for finding connected directories.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Avoid showing the bootstrap make command for check-old, etc.

Reported by: koobs
MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Update ^/vendor/NetBSD/tests/dist to a more recent snapshot

Pulled on "Thu Aug 11 18:01:19 PDT 2016"

make world: Allow installworld to be ran in parallel.

This has been safe for a while.

Sponsored by: EMC / Isilon Storage Division

SYSTEM_COMPILER: Rework the logic to allow a 'make test-system-compiler'.

1. Always calculate what the expected values are.
2. Add 'make test-system-compiler' to show all of the computed values
vs the wanted values.
3. Extend the .info line to buildkernel/kernel-toolchain/toolchain/_cross-tools.
4. Consolidate all of the logic to one condition.

MFC after: 3 days
Sponsored by: EMC / Isilon Storage Division

Vendor import of lldb release_39 branch r276489:
https://llvm.org/svn/llvm-project/lldb/branches/release_39@276489

Vendor import of libc++ release_39 branch r276489:
https://llvm.org/svn/llvm-project/libcxx/branches/release_39@276489

Vendor import of clang release_39 branch r276489:
https://llvm.org/svn/llvm-project/cfe/branches/release_39@276489

Vendor import of llvm release_39 branch r276489:
https://llvm.org/svn/llvm-project/llvm/branches/release_39@276489

Update to mdocml 1.13.4

Revert r302670 and r302671 for now.

MACHINE_CPUARCH smells like MACHINE except for arm64/aarch64 which
has it backwards.

Create one list of replacements for MACHINE_CPUARCH as MACHINE_CPUARCH_SUB.

This also adds missing s/aarch64/arm64 to the sys.mk version and also
adds back armv6hf for universe since it was added to the sys.mk version
in r300438.

MFC after: 3 days
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D7159

WITH_META_MODE: Whitelist 'make kernel-toolchain'

Approved by: re (blanket, META_MODE)
Sponsored by: EMC / Isilon Storage Division

Fix top-level builds with fmake / 9.x.

Reported by: allanjude
Approved by: re (marius)
Sponsored by: EMC / Isilon Storage Division

WITH_META_MODE: Whitelist 'make kernel' and 'make world'.

installkernel is technically META_MODE safe but doesn't need an explicit
approval to use it since it's all disabled via .PHONY.

world uses 'make -B installworld' which already will disable META_MODE
via the -B (.MAKE.MODE=compat) usage.

Approved by: re (implicit)
Sponsored by: EMC / Isilon Storage Division

tinderbox/universe: Only show arm64 warning if it was in the TARGETS list.

This was a flaw in my change in r287903 but also in the original change
in r282156 since it used empty(${TARGETS}) rather than empty(TARGETS).

Reported by: lidl
Approved by: re (gjb)
Sponsored by: EMC / Isilon Storage Division

Fix makeman showing dependency of DIRDEPS_BUILD->META_MODE.

This broke in r301887 with the meta mode whitelist. 'make showconfig'
still needs WITH_META_MODE support.

Approved by: re (implicit)
Sponsored by: EMC / Isilon Storage Division

Fix build from stable/10 with fmake.

This was broken in r301888.

fmake does not look in share/mk by default and thus does not yet
have MK_META_MODE set with default.

Pointyhat to: bdrewery
Approved by: re (implicit)
Sponsored by: EMC / Isilon Storage Division

WITH_META_MODE: Lessen the filemon(4) requirement scope.

- Move the sys.mk filemon requirement to bsd.init.mk as a warning.
This is intended only to show when building directly in a subdirectory
without filemon loaded.
- Move the error into Makefile and only apply it when building
from the META_TGT_WHITELIST target list.

-DNO_FILEMON can be used to suppress both the warning and the error but
makes WITH_META_MODE less useful. It will only compare build commands
in this mode rather than track all dependencies.

This fixes installing from a jail which doesn't need filemon in this
phase [1].

Reported by: Nikolai Lifanov <lifanov@mail.lifanov.com> [1]
Approved by: re (implicit)
Sponsored by: EMC / Isilon Storage Division

WITH_META_MODE: Whitelist targets that are meta-mode-safe.

META_TGT_WHITELIST is added to define which build targets are safe for
meta mode. See comments for more details.

This fixes 'make delete-old-libs' to properly show the interactive
prompt.

Approved by: re (implicit)
Sponsored by: EMC / Isilon Storage Division

Add more missing .PHONY

Approved by: re (implicit)
Sponsored by: EMC / Isilon Storage Division

upstream hg 0289cde234a0

Import bmake-20160606

From ChangeLog:

o dir.c: extend mtimes cache to others via cached_stat()

Only include bsd.compiler.mk for bmakes that use -m .../share/mk by default.

Otherwise they error when trying to include the file, which also has
its own dependencies.

Sponsored by: EMC / Isilon Storage Division

WITH_META_MODE: Enable bmake's missing meta rebuild feature

Fix bmake version upgrade logic to use the new bmake.

Otherwise it was only used on the next build.

This was a flaw in r295980.

MFC after: 3 days
Sponsored by: EMC / Isilon Storage Division

Import bmake-20160604

Interesting items from ChangeLog:

o cached_realpath(): avoid hitting filesystem more than necessary.
o meta.c: refactor need_meta decision, add knobs for
missing meta file and filemon data wrt out-of-datedness.
o meta.c: missing filemon data is only relevant if we read a
meta file.
Also do not return oodate for a missing metafile if gn->path
points to .CURDIR

Reviewed by: rpaulo
Approved by: rpaulo
Obtained from: NetBSD external/bsd/blacklist @ 20160409
Relnotes: YES
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D5912

Initial bhyve native graphics support.

This adds emulations for a raw framebuffer device, PS2 keyboard/mouse,
XHCI USB controller and a USB tablet.

A simple VNC server is provided for keyboard/mouse input, and graphics
output.

A VGA emulation is included, but is currently disconnected until an
additional bhyve change to block out VGA memory is committed.

Credits:
- raw framebuffer, VNC server, XHCI controller, USB bus/device emulation
and UEFI f/w support by Leon Dang
- VGA, console/g, initial VNC server by tychon@
- PS2 keyboard/mouse jointly done by tychon@ and Leon Dang
- hypervisor framebuffer mem support by neel@

Tested by: Michael Dexter, in a number of revisions of this code.

With the appropriate UEFI image, FreeBSD, Windows and Linux guests can
installed and run in graphics mode using the UEFI/GOP framebuffer.

Create branch for bhyve graphics import.

Import bmake-20160512

Interesting changes (from ChangeLog):
o meta.c: ignore paths that match .MAKE.META.IGNORE_PATTERNS
this is useful for gcov builds.
o propagate errors from filemon(4).
o meta.c: apply realpath() before matching against metaIgnorePaths
o allow makefile to set .MAKE.JOBS

Make armv6 hard float abi by default. Kill armv6hf.
Allow CPUTYPE=soft to build the current soft-float abi libraries.
Add UPDATING entry to announce this.

Approved by: re@ (gjb)

Import OpenSSL 1.0.2h.

Follow-up r298220: Don't pass down META_MODE which will still enable it.

Reported by: Nikolai Lifanov <lifanov@mail.lifanov.com>
Sponsored by: EMC / Isilon Storage Division

META_MODE: Disable during installworld and similar.

META_MODE may create cookies during staging of files to WORLDTMP that would
also prevent installation of the files to the final DESTDIR, since the cookie
already exists. This is not yet the case but will be soon. Prevent other
similar issues by disabling META_MODE for any top-level install targets.

Sponsored by: EMC / Isilon Storage Division

Import openresolv 3.8.0

Obtained from: http://roy.marples.name/projects/openresolv

Mark some more .PHONY targets.

Sponsored by: EMC / Isilon Storage Division

Define the *soft targets properly.

Sponsored by: EMC / Isilon Storage Division

Remove the old depend (mkdep) code and make FAST_DEPEND the one true way.

Reviewed by: emaste, hselasky (partial), brooks (brief)
Discussed on: arch@
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D5742

Import bmake-20160315

Use the newly minted Makefile.libcompat to implement libsoft libraries
for the armv6 ABI switch. This also make WITH_LIBSOFT functional on
the arm platform. As a transition thing, this seems to work even
without switching the ABI (we basically build the same libraries
twice when MK_LIBSOFT=yes until the ABI cut over next
month). MK_LIBSOFT remains default no.

Avoid MK_TESTS error on stable/10 by just preventing SUBDIR recursion.

Avoid bmake upgrade NO_MAN warning by just setting MAN to empty.

Suggested by: imp

Revert r296645 as it breaks stable/10->head builds.

FAST_DEPEND: Use .dinclude to enable full .depend logic in bmake.

The inclusion of .MAKE.DEPENDFILE (.depend) has special logic in make
to ignore stale/missing dependencies. bmake 20160220 added a '.dinclude'
directive that uses the special logic for .depend when including the file.

This fixes a build error when a file is moved or deleted that exists in a
.depend.OBJ file. This happened in r292782 when sha512c.c "moved" and an
incremental build of lib/libmd would fail with:
make: don't know how to make /usr/src/lib/libcrypt/../libmd/sha512c.c. Stop

Now this will just be seen as a stale dependency and cause a rebuild:
make: /usr/obj/usr/src/lib/libmd/.depend.sha512c.o, 13: ignoring stale .depend for /usr/src/lib/libcrypt/../libmd/sha512c.c
--- sha512c.o ---
...
This rebuild will only be done once since the .depend.sha512c.o will
be updated on the build with the -MF flags.

This also removes -MP being passed for the .depend.OBJ generation (which
would create fake targets for system headers) since the logic is no
longer needed to protect from missing files.

Sponsored by: EMC / Isilon Storage Division

Fix bmake upgrade NO_MAN warnings.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Fix make -n upgrade_checks.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Import bmake-20160307

Don't ever create object directories here with MK_AUTO_OBJ.

Sponsored by: EMC / Isilon Storage Division

Import OpenSSL 1.0.2g.

Import openresolv 3.7.3

Obtained from: http://roy.marples.name/projects/openresolv

Import libedit 2016-02-27

Obtained from: NetBSD

Add order for installworld/installkernel.

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Support a WANT_MAKE_VERSION.

This will be used soon for .dinclude support in FAST_DEPEND.

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Include a 'package-pkg' target, intended for use for
architectures we do not provide upstream pkg(8) packages.

This is not tied to anything as-is, and likely will break
your system if used (based on experience with testing with
powerpc).

There is an overwhelming amount of evil happening here,
so until the issues are fixed, it will not be tied into the
'packages' target.

Sponsored by: The FreeBSD Foundation

Vendor import of lldb release_38 branch r260756:
https://llvm.org/svn/llvm-project/lldb/branches/release_38@260756

Import ELF Tool Chain snapshot revision 3395

This is close to the upcoming 0.7.1 release.

From http://svn.code.sf.net/p/elftoolchain/code

Fix make universe when running with non POSIX/C locales using a locale sensitive
pattern

Reported by: many
Submitted by: jilles

Add logic to rotate the package repository, keyed on PKG_VERSION,
and create a 'latest' symlink to the PKG_VERSION repository path.

Suggested by: des
Sponsored by: The FreeBSD Foundation

Rename the signpackages target to sign-packages, and include it in the
packages meta-target so 'make packages' now does everything.

Split the packages target into stage-packages and create-packages to make
it possible to roll new packages from an existing build without having to
restage them.

Simplify running the FreeBSD test suite

Replace `make regress` (legacy test make target) and `make test` (incomplete
test make target added with the FreeBSD test suite) with make check as it's
consistent with other open source projects.

`make check` defaults to running tests from `.OBJDIR`, but can be overridden
with the `CHECKDIR` variable.

Add `make checkworld` target to simplify running the FreeBSD test suite from
`TESTSBASE` (i.e. the top-level tests directory), similar to buildworld.

Document `make check` and `make checkworld` in build(7).

Other minor changes:

- Rename intermediate file (`Kyuafile.auto`) to `Kyuafile` to simplify
`make check`.
- Remove terse warnings attached to `beforetest`/`aftertest`.
- Add kyua binary check to check target in suite.test.mk; error out if it's
not found

The MFC is [partly] contingent on other build related changes being MFCed.

Differential Revision: https://reviews.freebsd.org/D4406
MFC after: 2 months
X-MFC to: stable/10
Relnotes: yes
Reviewed by: bdrewery, Evan Cramer <eccramer@gmail.com>
Sponsored by: EMC / Isilon Storage Division

Import libedit 2016-01-16

Obtained from: NetBSD

Add order for installworld/distribution.

Sponsored by: EMC / Isilon Storage Division

Import openresolv 3.7.2

Obtained from: http://roy.marples.name/projects/openresolv

Add make universe targets "kernels" and "worlds".

"make kernels" is now shorthand for "make universe -DMAKE_JUST_KERNELS"
"make worlds" is now shorthand for "make universe -DMAKE_JUST_WORLDS"

The kernels target includes modules (unless you add -DNO_MODULES).

And of course you can still add all the other universe options, such as
"make kernels TARGETS=arm" to build kernels for all arm arches, or
TARGET_ARCH=armv6 to build all armv6 kernels, etc.

Reviewed by: imp

Import OpenSSL 1.0.2f.

Add 'stagekernel' target, which invokes 'distributekernel'
with -DNO_ROOT to create the METALOG mtree(8) file.

Separate the default STAGEDIR for world (WSTAGEDIR) and kernel
(KSTAGEDIR).

Fix the 'create-kernel-packages' target to work properly.

Evaluate if 'kernel' is set when invoking mtree-to-plist.awk,
which splits the kernel and kernel.debug into separate plist
files.

Fix METALOG creation when building/packaging multiple kernels.

Sponsored by: The FreeBSD Foundation

Add a 'signpackages' target, which creates the pkg repository
metadata files and uses the path of PKGSIGNKEY for signing,
if set.

Sponsored by: The FreeBSD Foundation

Import mdocml CVS snapshot 20160116

Always try to upgrade to bmake if not already using it.

This is mostly targetting stable/10 which requires bmake to build and
has issues upgrading from <10.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division
PR: 198062

Revert r293286. It was not intended to come in yet.

Move the MAKEOBJDIRPREFIX value guard to sys.mk and expand to MAKEOBJDIR.

This will ensure that the variable was not set as a make override, in
make.conf, src.conf or src-env.conf. It allows setting the value in
src-env.conf when using WITH_AUTO_OBJ since that case properly handles
changing .OBJDIR (except if MAKEOBJDIRPREFIX does not yet exist which is
being discussed to be changed).

This change allows setting a default MAKEOBJDIRPREFIX via local.sys.env.mk.

Sponsored by: EMC / Isilon Storage Division

Vendor import of lldb trunk r256945:
https://llvm.org/svn/llvm-project/lldb/trunk@256945

Vendor import of lld trunk r256633:
https://llvm.org/svn/llvm-project/lld/trunk@256633

Vendor import of clang trunk r256633:
https://llvm.org/svn/llvm-project/cfe/trunk@256633

Import bmake-20151220

Update advice on obj disk space requirements

The Makefile is not the best place for this information, but at least it
is now less out of date.

Start support for the RISC-V 64-bit architecture developed by UC Berkeley.

RISC-V is a new ISA designed to support computer research and education, and
is now become a standard open architecture for industry implementations.

This is a minimal set of changes required to run 'make kernel-toolchain'
using external (GNU) toolchain.

The FreeBSD/RISC-V project home: https://wiki.freebsd.org/riscv.

Reviewed by: andrew, bdrewery, emaste, imp
Sponsored by: DARPA, AFRL
Sponsored by: HEIF5
Differential Revision: https://reviews.freebsd.org/D4445

Import bmake-20151201

Import OpenSSL 1.0.2e.

Fix errors being ignored in many phases of the build since the bmake integration.

Say it with me, "I will not chain commands with && in Makefiles"

This was originally fixed and explained quite well by bde@ in r36074. The
initial bmake integration caused 'set -e' to stop being used which lead to
r252419. Later 'set -e' expectations were fixed with bmake in r254980.

Because of the && here, errors would be ignored when building in parallel and
a dependency failed. Such as bootstrap-tools since it builds everything in
parallel. If any tool failed in obj/depend/all, it would just ignore the error
and continue to build. This later would result in cascaded errors that only
confused the real issue. This could also cause commands after the failed
command to still execute, leading to more confusion.

This should be fine if the command is in a sub-shell such as: (cmd1 && cmd2)

This reverts r252419.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Rename META_MODE option to DIRDEPS_BUILD

This allows META_FILES option to be renamed META_MODE.
Also add META_COOKIE_TOUCH for use in targets that can benefit
from a cookie when in meta mode.

Differential Revision: https://reviews.freebsd.org/D4153
Reviewed by: bdrewery

Move 'make cleandir' from Makefile to Makefile.inc1.

This leads the way for fixing cross-build cleanup, and eventually replacing
'cleandir' with it during the build.

Sponsored by: EMC / Isilon Storage Division
MFC after: 2 weeks

Add built-in ccache build support via WITH_CCACHE_BUILD option.

ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
used to achieve a safe pseudo-incremental build. This is explained in
more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the
cache, but with a full cache saves 30-50% in build times. When combined with
the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
resulting dependency file, which it does not do when using mkdep(1)/'CC
-E'. Stats are provided at the end of this message.

This removes the need to modify /etc/make.conf with the CC:= and CXX:=
lines which conflicted with external compiler support [3] (causing the
bootstrap compiler to not be built which lead to obscure failures [4]),
incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
Clang errors with parenthesis, and did not work with META_MODE.

The option name was picked to match the existing option in ports. This
feature is available for both in-src and out-of-src builds that use
/usr/share/mk.

Linking, assembly compiles, and pre-processing avoid using ccache since it is
only overhead. ccache does nothing special in these modes, although there is
no harm in calling it for them.

CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
compiler to hash the content of the compiler binary to determine if it
should be a cache miss. For external compilers the 'mtime' option is used
as it is more efficient and likely to be correct. Future work may optimize the
'content' check using the same checks as whether a bootstrap compiler is needed
to be built.

The CCACHE_CPP2 pessimization is currently default in our devel/ccache
port due to Clang requiring it. Clang's -Wparentheses-equality,
-Wtautological-compare, and -Wself-assign warnings do not mix well with
compiling already-pre-processed code that may have expanded macros that
trigger the warnings. GCC has so far not had this issue so it is allowed to
disable the CCACHE_CPP2 default in our port.

Sharing a cache between multiple checkouts, or systems, is explained in
the ccache manual. Sharing a cache over NFS would likely not be worth
it, but syncing cache directories between systems may be useful for an
organization. There is also a memcached backend available [5]. Due to using
an object directory outside of the source directory though you will need to
ensure that both are in the same prefix and all users use the same layout. A
possible working layout is as follows:
Source: /some/prefix/src1
Source: /some/prefix/src2
Source: /some/prefix/src3
Objdir: /some/prefix/obj
Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
all absolute paths to be relative. Using something like this is required due
to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
adds into its hash for the object without CCACHE_BASEDIR.

distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
I have not personally tested this and assume it will not mix well with
using the bootstrap compiler.

The cache from buildworld can be reused in a subdir by first running
'make buildenv' (from r290424).

Note that the cache is currently different depending on whether -j is
used or not due to ccache enabling -fdiagnostics-color automatically if
stderr is a TTY, which bmake only does if not using -j.

The system I used for testing was:
WITNESS
Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
The arc was fully populated with src tree files and ccache objects.
RAM: 76GiB
CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16

The WITH_FAST_DEPEND feature was used for comparison here as well to show
the dramatic time savings with a full cache.

buildworld:
x buildworld-before
+ buildworld-ccache-empty
* buildworld-ccache-full
% buildworld-ccache-full-fastdep
# buildworld-fastdep
+-------------------------------------------------------------------------------+
|% * # +|
|% * # +|
|% * # xxx +|
| |A |
| A|
| A |
|A |
| A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 3744.13 3794.31 3752.25 3763.5633 26.935139
+ 3 4519 4525.04 4520.73 4521.59 3.1104823
Difference at 95.0% confidence
758.027 +/- 43.4565
20.1412% +/- 1.15466%
(Student's t, pooled s = 19.1726)
* 3 1823.08 1827.2 1825.62 1825.3 2.0785572
Difference at 95.0% confidence
-1938.26 +/- 43.298
-51.5007% +/- 1.15045%
(Student's t, pooled s = 19.1026)
% 3 1266.96 1279.37 1270.47 1272.2667 6.3971113
Difference at 95.0% confidence
-2491.3 +/- 44.3704
-66.1952% +/- 1.17895%
(Student's t, pooled s = 19.5758)
# 3 3153.34 3155.16 3154.2 3154.2333 0.91045776
Difference at 95.0% confidence
-609.33 +/- 43.1943
-16.1902% +/- 1.1477%
(Student's t, pooled s = 19.0569)

buildkernel:
x buildkernel-before
+ buildkernel-ccache-empty
* buildkernel-ccache-empty-fastdep
% buildkernel-ccache-full
# buildkernel-ccache-full-fastdep
@ buildkernel-fastdep
+-------------------------------------------------------------------------------+
|# @ % * |
|# @ % * x + |
|# @ % * xx ++|
| MA |
| MA|
| A |
| A |
|A |
| A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 571.57 573.94 571.79 572.43333 1.3094401
+ 3 727.97 731.91 728.06 729.31333 2.2492295
Difference at 95.0% confidence
156.88 +/- 4.17129
27.4058% +/- 0.728695%
(Student's t, pooled s = 1.84034)
* 3 527.1 528.29 528.08 527.82333 0.63516402
Difference at 95.0% confidence
-44.61 +/- 2.33254
-7.79305% +/- 0.407478%
(Student's t, pooled s = 1.02909)
% 3 400.4 401.05 400.62 400.69 0.3306055
Difference at 95.0% confidence
-171.743 +/- 2.16453
-30.0023% +/- 0.378128%
(Student's t, pooled s = 0.954969)
# 3 201.94 203.34 202.28 202.52 0.73020545
Difference at 95.0% confidence
-369.913 +/- 2.40293
-64.6212% +/- 0.419774%
(Student's t, pooled s = 1.06015)
@ 3 369.12 370.57 369.3 369.66333 0.79033748
Difference at 95.0% confidence
-202.77 +/- 2.45131
-35.4225% +/- 0.428227%
(Student's t, pooled s = 1.0815)

[1] https://ccache.samba.org/performance.html
[2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
[3] https://reviews.freebsd.org/D3484
[5] https://github.com/jrosdahl/ccache/pull/30

PR: 182944 [4]
MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division
Relnotes: yes

Import OpenSSL 1.0.2d.

Import bmake-20151020

Rework the 'make -n -n' feature such that '-n' recurses and '-N' does not.

Bmake has a documented feature of '-N' to skip executing commands which is
specifically intended for debugging top-level builds and not recursing into
sub-directories. This matches the older 'make -n' behavior we added which made
'-n -n' the recursing target and '-n' a non-recursing target.

Removing the '-n -n' feature allows the build to work as documented in
the bmake manpage with '-n' and '-N'. The older '-n -n' feature was also
not documented anywhere that I could see.

Note that the ${_+_} var is still needed as currently bmake incorrectly
executes '+' commands when '-N' is specified.

The '-n' and '-n -n' features were broken for several reasons prior to this.
r251748 made '_+_' never expand with '-n -n' which resulted in many
sub-directories not being visited until fixed 2 years later in r288391, and
many targets were given .MAKE over the past few years which resulted in
non-sub-make commands, such as rm and ln and mtree, to be executed.

This should also allow removing some indirection hacks in bsd.subdir.mk and
other cases of .USE that have a .MAKE by using '+'.

Sponsored by: EMC / Isilon Storage Division
Discussed on: arch@ (mostly silence)

Tweak the guard more to suggest 'all' if SUBDIR_OVERRIDE is specified. In that
case 'all' does make sense.

MFC after: 2 weeks
X-MFC-With: r289411
Sponsored by: EMC / Isilon Storage Division

Tweak the default target to not suggest 'all' since it really doesn't do
anything useful for most users.

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Consider top-level targets to be .PHONY as bmake won't build them
otherwise if a file with the same name is found in the directory.

MFC after: 1 week
Sponsored by: EMC / Isilon Storage Division

Add installconfig as a top level target

Fix the .MAKE added in r251750 to properly support the historical -n -n.

The condition used matches the condition in sys.mk for setting _+_ to blank
or +.

With this -n will continue to not descend into Makefile.inc1, while -n -n will
and cause Makefile.inc1's target to run with -n.

MFC after: 2 weeks
Sponsored by: EMC / Isilon Storage Division

Rework r287900 to keep arm64/aarch64 stable in the TARGETS list.

This is relevant for makeman using the 'make targets' output in src.conf(5).

This makes a _UNIVERSE_TARGETS that removes arm64 if the build
requirements are not met.

Get arm64/aarch64 into 'make targets' output to fix makeman.

Rename kernel-pkgs into create-kernel-packages to be consistent with world

Import libc++ 3.7.0 release (r246257).

Import OpenSSL 1.0.1p.

Fix generation of src.conf.5

Since makeman turns all options on, we need to guard somethings from
make(showconfig)

targets now needs .PHONY

Move the universe done dependency on the worlds completing out of the
loop, we only need to add it once.

Obtained from: ABT Systems Ltd

Remove unneeded dependencies, these are now handled through the _done
target.

Obtained from: ABT Systems Ltd

Add a new target universe_${target}_done to print the completion message.
Without this we could print this message in the wrong place when building
with MAKE_JUST_WORLDS is set.

Obtained from: ABT Systems Ltd

Add a universe_${target}_worlds target to simplify the logic to find when
to start building kernels.

Obtained from: ABT Systems Ltd

A more compatible fix to MK_META_MODE not being defined. Also, encase
bmake specific constructs not needed for make bootstrap so fmake
doesn't see them. This works with fmake just well enough for us to
build bmake to build the rest of the tree without fatal errors. Tested
only with fmake package.

Add a comment to the end of the world and kernel cases in the universe
target to help follow the make magic.

Obtained from: ABT Systems Ltd

If MK_META_MODE is unset, assume a value of no.

This is needed to build HEAD on FreeBSD 10.1,
which has bmake, but does not have
/usr/share/mk/src.opts.mk

Reviewed by: sjg
Reported by: jenkins

Import OpenSSL 1.0.1o.

Import OpenSSL 1.0.1n.

bmake-20150606

Since sys.mk now handles META_MODE and META_FILES options
we need not worry about them here.
Checking for .PARSEDIR no longer needed.

Import bmake-20150505 detect typo's in .if variable references.

Import openresolv 3.7.0.

PR: 199854
Obtained from: http://roy.marples.name/projects/openresolv
Sponsored by: The FreeBSD Foundation

Don't add arm64 to universe builds if the user provided a TARGETS list

Differential Revision: https://reviews.freebsd.org/D2375
Reported by: andrew
Reviewed by: andrew, imp
Sponsored by: The FreeBSD Foundation

Fix a typo (linnker -> linker)

Import bmake-20150418

Add arm64 to universe if binutils is available.

arm64 relies on an external binutils port or package right now, because
the in-tree linker from binutils 2.17.50 does not support arm64. Add
arm64 to universe if the linker is available. If not output a message
that arm64 is skipped.

buildworld and buildkernel use the external binutils automatically, so
it's sufficient to run 'pkg install aarch64-binutils' to build
FreeBSD/arm64.

Differential Revision: https://reviews.freebsd.org/D2302
Reviewed by: andrew, imp
Sponsored by: The FreeBSD Foundation

Vendor import of lld trunk r233088:
https://llvm.org/svn/llvm-project/lld/trunk@233088

Import lua 5.3.0.

Make TARGET_ARCH=powerpc64 work without TARGET=powerpc.

Import OpenSSL 1.0.1m.

Start to import support for the AArch64 architecture from ARM. This change
only adds support for kernel-toolchain, however it is expected further
changes to add kernel and userland support will be committed as they are
reviewed.

As our copy of binutils is too old the devel/aarch64-binutils port needs
to be installed to pull in a linker.

To build either TARGET needs to be set to arm64, or TARGET_ARCH set to
aarch64. The latter is set so uname -p will return aarch64 as existing
third party software expects this.

Differential Revision: https://reviews.freebsd.org/D2005
Relnotes: Yes
Sponsored by: The FreeBSD Foundation

Create a "packages" target takes care of all the magic

Import mdocml 1.13.3

Add new create-world-packages target to generated the packages out of world

Dynamically figure out the list of targets based on tags passed on the mtrees
First sanity check that all packages have existing manifests
Generate the packages

Please note that for now the mtree needs more work as it has duplicate entries,
everything is not yet tagged

The packages now have generic entries and needs to be customize

Import CVS snapshot of mandoc as of 20150302

Import from device-tree git://xenbits.xen.org/people/ianc/device-tree-rebasing.git @c8c1b3a77934768c7f7a4a9c10140c8bec529059

Add a new target make stageworld which basically does the same job as:
make installworld distribution but preparing the mtree the same way
distributeworld does and respecting -DNO_ROOT

Plug kernel-pkgs target to the list of valid targets

Import OpenSSL 1.0.1l.

Import libedit 2014-01-09

Obtained from: NetBSD

Import OpenSSL 1.0.1k.

Import compiler-rt trunk r224034.

https://llvm.org/svn/llvm-project/compiler-rt/trunk@224034

Import mandoc 1.13.2

Import elftoolchain rev 3130

From svn.code.sf.net/p/elftoolchain/code/trunk

Import CVS version of mandoc as of 20141201

If we are doing META_MODE use targets/Makefile as top-level
since this one isn't suitable.