arm64: Allow userspace to be built with PAC and BTI

Add the WITH/WITHOUT_BRANCH_PROTECTION build flags. This can be used
to enable the use of pointer authentication (FEAT_PAuth) and branch
target identification (FEAT_BTI) in userspace.

The kernel already handles both of these is userspace, we just need
to enable it.

Leave disabled for a short period for this to settle before enabling.

Reviewed by: emaste
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42596

Make WITHOUT_UNDEFINED_VERSION the default

Link with --no-undefined-version by default. Will detect and prevent
the accidental removal of symbols from versioned libraries.

(cherry picked from commit 4510f2ca9170927309a423274e03f1eb8e27da27)
This reverts commit b25ceb97eddcd58cfb2b433fe301ab28c678c249.

Reviewed by: arichardson, kib, dim, emaste
Differential Revision: https://reviews.freebsd.org/D44216

Revert "Make WITHOUT_UNDEFINED_VERSION the default"

This is causing failures on gcc13 CI builds so those need to be fixed
or worked around.

This reverts commit 4510f2ca9170927309a423274e03f1eb8e27da27.

Make WITHOUT_UNDEFINED_VERSION the default

Link with --no-undefined-version by default. Will detect and prevent
the accidental removal of symbols from versioned libraries.

Reviewed by: arichardson, kib, dim, emaste
Differential Revision: https://reviews.freebsd.org/D44216

bsd.opts.mk: drop transtion aid for NO_* options

Setting NO_CTF, NO_DEBUG_FILES, NO_INSTALLLIB, NO_MAN, NO_PROFILE,
and NO_WARNS was deprecated in 2014 and made an error prior to the
13.0.0 release in commit d3a5bf95f2013af081607abd91b5175f6eafa563.
Likewise, NO_WERROR was made an error prior to 13.0.0 in commit
7fa2f2a62f04f095e1e27ad55aa22a8f59b1df8f.

Remove this transition aid making these variables no-ops as setting them
will result in an error on all supported releases.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D44345

Add an UNDEFINED_VERSION option

When enabled (current default) link with --undefined-version to allow
symbol maps to contain symbols not defined by libraries. When disabled,
link with --no-undefined-version to disallow these bugs.

WITHOUT_UNDEFINED_VERSION is currently broken. Once it is fixed it
should be made the default and this option should likely be removed.

Reviewed by: dim, emaste
Differential Revision: https://reviews.freebsd.org/D44169

pkgbase: Split out manpages by default

This helps with building small container images using pkgbase.

Reviewed by: manu bapt
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D41861

Add INIT_ALL build option

This option replaces WITH_INIT_ALL_PATTERN and WITH_INIT_ALL_ZERO with
INIT_ALL=pattern and INIT_ALL=zero respectively. As these are
relatively rarely used options no backwards compatibility is
implemented.

Reviewed by: emaste
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D41675

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

Add some more local.*.mk includes

These facilitate customizing the build with minimal churn.

Reviewed by: stevek
Sponsored by: Juniper Networks, Inc.
Differential Revision: https://reviews.freebsd.org/D37592

Add RELRO build knob, default to enabled

Note that lld enables relro by default, so that we already had either
partial or full RELRO, depending on the state of the BIND_NOW knob.

Add a RELRO knob so that the option can be disabled if desired, and so
that builds using the GNU toolchain are equivalent to those using the
standard Clang/LLVM toolchain.

Reviewed by: markj
MFC after: 3 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35545

Remove more quotes around Makefile .error/.warn/.info strings.

Reviewed by: imp, emaste
Differential Revision: https://reviews.freebsd.org/D34175

bsd.opts.mk: Remove mips support

We don't need to list all the 32-bit mips variants here anymore.

Sponsored by: Netflix

Add clarifying comments

Expand on the terse comments for where each of these files is used.

Reviewed by: emaste
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D33716

Disable PROFILE option by default

Hardware based profiling (e.g. hwpmc) is much better and produces more
useful results. Today the profiling lib archives (_p.a) serve no real
purpose other than increasing the library build time.

Both upstream and base system (in commit b762974cf4b9) Clang have been
modified to remove the special case for linking against these libraries.

Clang's -pg support and mcount() remain, so building with -pg can still
be used on code that the user builds; we just no longer provide prebuilt
libraries compiled with -pg.

Discussed on freebsd-hackers[1] / freebsd-current [2] in 2020 and
freebsd-arch [3] in 2021. A deprecation notice was added in
commit 175841285e28.

[1] https://lists.freebsd.org/pipermail/freebsd-hackers/2020-January/055551.html
[2] https://lists.freebsd.org/pipermail/freebsd-current/2020-January/075105.html
[3] https://lists.freebsd.org/archives/freebsd-arch/2021-June/000016.html

PR: 256873 [exp-run]
Reviewed by: imp, jhb, kib
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D30833

Add build system support for ASAN+UBSAN instrumentation

This adds two new options WITH_ASAN/WITH_UBSAN that can be set to
enable instrumentation of all binaries with AddressSanitizer and/or
UndefinedBehaviourSanitizer. This current patch is almost sufficient
to get a complete buildworld with sanitizer instrumentation but in
order to actually build and boot a system it depends on a few more
follow-up commits.

Reviewed By: brooks, kib, markj
Differential Revision: https://reviews.freebsd.org/D31043

pkgbase: Fix building out-of-tree manual pages

c7e6cb9e08d6 introduced MK_MANSPLITPKG but it was not available for
building out-of-tree manual pages. For example, x11/nvidia-driver fails
with the following error:

===> doc (all)
make[3]: "/usr/share/mk/bsd.man.mk" line 53: Malformed conditional (${MK_MANSPLITPKG} == "no")
make[3]: Fatal errors encountered -- cannot continue

Move the definition from src.opts.mk to bsd.opts.mk to make it visible.

Enable PIE by default on 64-bit architectures

This patch adds Position Independent Executables (PIE)
flags for building OS. It allows to enable the ASLR
feature based only on the sysctl knobs, without
need to rebuild the image. Tests showed that
no problems with stability / performance degradation
were seen when using PIEs with ASLR disabled.

The change is limited only for 64-bit architectures.

Use bsd.opts.mk instead of the src.opts.mk in order
to satisfy all build dependencies related to MK_PIE.

Reviewed by: emaste, imp
Obtained from: Semihalf
Sponsored by: Stormshield
Differential Revision: https://reviews.freebsd.org/D28328

Rename NO_WERROR -> MK_WERROR=no

As suggested in D27598. This also supports MK_WERROR.clang=no and
MK_WERROR.gcc=no to support the existing NO_WERROR.<compiler> uses.

Reviewed By: brooks
Differential Revision: https://reviews.freebsd.org/D27601

Fix `make makeman` after r367577

WITH_INIT_ALL_ZERO and WITH_INIT_ALL_PATTERN are mutually exclusive.
The .error when they were both set broke makeman so demote it to a
warning (and presumably the compiler will fail on an error later on).

We could improve this to make one take precedence but this is sufficient
for now.

MFC with: r367577
Sponsored by: The FreeBSD Foundation

Support initializing stack variables on function entry

There are two options:
- WITH_INIT_ALL_ZERO: Zero all variables on the stack.
- WITH_INIT_ALL_PATTERN: Initialize variables with well-defined patterns.

The exact pattern are a compiler implementation detail and vary by type.
They are somewhat documented in the LLVM commit message:
https://reviews.llvm.org/rL349442
I've used WITH_INIT_ALL_* to match Microsoft's InitAll feature rather
than naming them after the LLVM specific compiler flags.

In a range of consumer products, options like these are used in
both debug and production builds with debugs builds using patterns
(intended to provoke crashes on use of uninitialized values) and
production using zeros (deemed more likely to lead to harmless
misbehavior or NULL-pointer dereferences).

Reviewed by: emaste
Obtained from: CheriBSD
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D27131

Remove the SYMVER build option.

This option was added as a transition aide when symbol versioning was
first added. It was enabled by default in 2007 and is supported even
by the old GPLv2 binutils. Trying to disable it currently fails to
build in libc and at this point it isn't worth fixing the build.

Reported by: Michael Dexter
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D24637

Due to popular demand, revert r360102.

Reported by: many

Conditionally install Kerberos rc files based on MK_KERBEROS_SCRIPTS
instead of MK_KERBEROS. The reason for this change is some users
prefer to build FreeBSD WITHOUT_KERBEROS, wanting to retain the
Kerberos rc scripts to start/stop MIT Kerberos or Heimdal from ports.

PR: 197337
Reported by: Adam McDougall <ebay at looksharp.net>
Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D24252

Make the warning for deprecated NO_ variables an error.

Support for NO_CTF, NO_DEBUG_FILES, NO_INSTALLLIB, NO_MAN, NO_PROFILE,
and NO_WARNS as deprecated in 2014 with a warning added for each one
found. Turn these into error in preperation for removal of compatability
support before FreeBSD 13.

This was previously committed in r354909 and reverted in r355011 due to
unforseen impacts on ports. I've since corrected all amd64 and i386
ports reported in prior runs as well as instance of these variables I
found via grep.

Revert r354909: Make the warning for deprecated NO_ variables an error.

An unexpectidly large number of ports define NO_MAN (and sometimes the
long-dead NOMAN). I'll fix ports and then re-commit.

Make the warning for deprecated NO_ variables an error.

Support for NO_CTF, NO_DEBUG_FILES, NO_INSTALLLIB, NO_MAN, NO_PROFILE,
and NO_WARNS as deprecated in 2014 with a warning added for each one
found. Turn these into error in preperation for removal of compatability
support before FreeBSD 13.

Reviewed by: imp
Relnotes: yes
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D22448

Add WITH_PIE knob to build Position Independent Executables

Building binaries as PIE allows the executable itself to be loaded at a
random address when ASLR is enabled (not just its shared libraries).

With this change PIE objects have a .pieo extension and INTERNALLIB
libraries libXXX_pie.a.

MK_PIE is disabled for some kerberos5 tools, Clang, and Subversion, as
they explicitly reference .a libraries in their Makefiles. These can
be addressed on an individual basis later. MK_PIE is also disabled for
rtld-elf because it is already position-independent using bespoke
Makefile rules.

Currently only dynamically linked binaries will be built as PIE.

Discussed with: dim
Reviewed by: kib
MFC after: 1 month
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D18423

Add a WITH_BIND_NOW build knob

The linker's -z now flag sets the DF_BIND_NOW flag, which signals to the
runtime loader that all relocation processing should be performed at
process startup rather than on demand. In combination with lld's
default of enabling relro this causes the GOT to be made read-only when
the process starts, preventing straightforward GOT overwrite attacks.

Shawn Webb discovered a failure on HardenedBSD with BIND_NOW and ifunc
use, which resulted in my rtld fix in r340137. Add a BIND_NOW knob as
it is trivial to do so and is a useful ELF hardening feature. This
change is equivalent to HardenedBSD's but not identical as there are
other diffs/conflicts nearby.

Note that our ELF Tool Chain readelf does not currently decode the
DF_BIND_NOW flag - see PR232983.

Reviewed by: brooks
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17846

Introduce src.conf knob to build userland with retpoline

WITH_RETPOLINE enables -mretpoline vulnerability mitigation in userland
for CVE-2017-5715.

Reported by: Peter Malcom
Reviewed by: markj
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D17421

Add MK_MAKE_CHECK_WITH_COVERAGE, reliant on MK_COVERAGE and MK_MAKE_CHECK_USE_SANDBOX

Let's see if the dependent relationship between MK_COVERAGE and MK_DEBUG_FILES
I added is the source of my knob grief

Add limited sandbox capability to "make check"

== Rationale ==

r295380 introduced "make check" and consolidated means for running
test code in an attempt to simplify running tests. One could either
install files/libraries/programs and run "make check", or run "make check"
with an explicit CHECKDIR, e.g., `make check CHECKDIR=$(make -V.OBJDIR)``.

One criticism that was received is that "make check" should be run with
the intent of making dev->test->commit easier, which means that the target
audience's workflow should be developers. One developer pattern available
in other opensource projects is to run test code from a developer sandbox,
instead of installing to a system.

== Method ==

This approach is slightly different from the standard approach, in the sense
that it builds and installs into a deterministic directory under .OBJDIR (as I call it,
the "sandbox"), then runs "make check" against that. In the event the test
run is successful, the deterministic directory is removed to save space.

== Approach ==

bsd.lib.mk, bsd.prog.mk:

To support this functionality, a new variable `HAS_TESTS` is being added.

HAS_TESTS enables appropriate behavior with bsd.lib.mk and bsd.prog.mk, as
follows:
- Add "make check" as an available target from the directory.
- Pass down appropriate variables via ${TESTS_ENV}, i.e.,
${TESTS_LD_LIBRARY_PATH} and ${TESTS_PATH}.

One should add "HAS_TESTS" to directories containing tests in them, e.g. from
bin/sh/Makefile,

HAS_TESTS=
SUBDIR.${MK_TESTS}+= tests

HAS_TESTS doesn't automatically add the tests subdirectory for flexibility
reasons.

bsd.opts.mk, src.opts.mk:
- The knob ${MK_MAKE_CHECK_USE_SANDBOX} has been added, both to explicitly
direct (internally) when to set a deterministic ${DESTDIR} and to also allow
users to disable this behavior globally, i.e., via src.conf.
- MK_TESTS has been promoted from src.opts.mk to bsd.opts.mk to leverage
syntactic sugar for having MK_TESTS be a dependency for
MK_MAKE_CHECK_USE_SANDBOX, but to also ensure that src.opts.mk isn't required
to use suite.test.mk (which is a dependency of bsd.test.mk).

suite.test.mk:
- beforecheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified
from a no-op to:
-- Build.
-- Run "make hierarchy" on the sandbox dir.
-- Install the tests/files to the sandbox dir.
- aftercheck behavior (when MK_MAKE_CHECK_USE_SANDBOX is enabled) is modified
from a no-op to:
-- Remove the sandbox dir.

Again, because the dependency order set in bsd.test.mk is
beforecheck -> check -> aftercheck, "make check" will not be run unless
"beforecheck" completes successfully, and "aftercheck" will not be run unless
"beforecheck" and "check" complete successfully.

== Caveats ==

- This target must either be run with MK_INSTALL_AS_USER or as root. Otherwise
it will fail when running "make install" as the default user/group for many
makefiles when calling INSTALL is root/wheel.
- This target must be run from a suitable top-level directory. For example,
running tests from `tests/sys/fs/tmpfs` won't work, but `tests/sys/fs` will,
because `tests/sys/fs/tmpfs` relies on files installed by `tests/sys/fs`.
- Running MK_INSTALL_AS_USER may introduce determinism issues. However, using
it could identify deficiences in tests in terms of needing to be run as
root, which are not properly articulated in the test requirements.
- The doesn't negate the need for running "make installworld" and
"make checkworld", etc. Again, this just is intended to simplify the
dev->test->commit workflow.

== Cleanup done ==
- CHECKDIR is removed; one can use "MK_MAKE_CHECK_USE_SANDBOX=no" to enable
"legacy" (r295380) behavior.

MFC after: 2 months
Relnotes: yes (CHECKDIR removed; "make check" behavior changed)
Requested by: jhb
Reviewed by: arch (silence), testing (silence)
Differential Revision: D11905

Fix typo with missing line continuation added in r321956

Expose _TESTS_USE_OBJDIR as MK_MAKE_CHECK_USE_SANDBOX and anchor the default
value on MK_TESTS.

Use bsd.opts.mk in suite.test.mk to toggle the behavior.

Promote MK_TESTS from src.opts.mk to bsd.opts.mk

This knob is needed here to avoid polluting suite.tests.mk with src.opts.mk for
a knob that will be introduced in the following commit.

Delete comment above "__DEFAULT_DEPENDENT_OPTIONS" related to "meta mode options"

src.conf(5) should document which knobs are which and the dependency between each;
remove the comment so the variable can apply to non-"meta mode options".

MFC after: 2 weeks

Some minor doc fixups

- Tweak a sentence by placing the modifier before an adjective to make it flow
better.
- Fix a typo.

MFC after: 3 days

Use __DEFAULT_DEPENDENT_OPTIONS for articulating dependency relationship
between MK_STALE_STAGED and MK_STAGING instead of using equivalent ad hoc
logic.

MFC after: 1 month

MK_COVERAGE should rely on MK_DEBUG_FILES

Building programs and libraries with --coverage without having corresponding
symbol support built in to them doesn't make a whole lot of sense.

Add MK_COVERAGE support to bsd.opts.mk, etc

Remove the old depend (mkdep) code and make FAST_DEPEND the one true way.

Reviewed by: emaste, hselasky (partial), brooks (brief)
Discussed on: arch@
Sponsored by: EMC / Isilon Storage Division
Differential Revision: https://reviews.freebsd.org/D5742

Enable FAST_DEPEND by default.

Discussed on: arch
Sponsored by: EMC / Isilon Storage Division

Enable NO_ make variable deprecation warning

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D5495

Hook the meta/nofilemon build into using FAST_DEPEND.

FAST_DEPEND is intended to be the "skip 'make depend' and mkdep"
feature. Since DIRDEPS_BUILD does this already with some of its own
hacks, and filemon doesn't need this, and nofilemon does, teach it how
to handle each of these cases.

In meta+filemon mode filemon will handle dependencies itself via the
meta mode logic in bmake. We still want to set MK_FAST_DEPEND=yes to
enable some logic that indicates that 'make depend' is skipped in the
traditional sense. The actual .depend.* files will be skipped.

When nofilemon is set though we still need to track and generate dependencies.

Sponsored by: EMC / Isilon Storage Division

Move the inclusion of bsd.cpu.mk from sys.mk to bsd.opts.mk. However,
for historical behavior that ports depends on, include it if we're
inside the ports tree.

Differential Review: https://reviews.freebsd.org/D4383
Ports Exp run: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=205021

Build and install userland .debug files by default

Debug data files are now built by default with 'make buildworld' and
installed with 'make installworld'. This facilitates debugging but
requires more disk space both during the build and for the installed
world. Debug files may be disabled by setting WITHOUT_DEBUG_FILES=yes
in src.conf(5).

Reviewed by: bdrewery, eadler, vangyzen
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D4018

Add built-in ccache build support via WITH_CCACHE_BUILD option.

ccache is mostly beneficial for frequent builds where -DNO_CLEAN is not
used to achieve a safe pseudo-incremental build. This is explained in
more detail upstream [1] [2]. It incurs about a 20%-28% hit to populate the
cache, but with a full cache saves 30-50% in build times. When combined with
the WITH_FAST_DEPEND feature it saves up to 65% since ccache does cache the
resulting dependency file, which it does not do when using mkdep(1)/'CC
-E'. Stats are provided at the end of this message.

This removes the need to modify /etc/make.conf with the CC:= and CXX:=
lines which conflicted with external compiler support [3] (causing the
bootstrap compiler to not be built which lead to obscure failures [4]),
incorrectly invoked ccache in various stages, required CCACHE_CPP2 to avoid
Clang errors with parenthesis, and did not work with META_MODE.

The option name was picked to match the existing option in ports. This
feature is available for both in-src and out-of-src builds that use
/usr/share/mk.

Linking, assembly compiles, and pre-processing avoid using ccache since it is
only overhead. ccache does nothing special in these modes, although there is
no harm in calling it for them.

CCACHE_COMPILERCHECK is set to 'content' when using the in-tree bootstrap
compiler to hash the content of the compiler binary to determine if it
should be a cache miss. For external compilers the 'mtime' option is used
as it is more efficient and likely to be correct. Future work may optimize the
'content' check using the same checks as whether a bootstrap compiler is needed
to be built.

The CCACHE_CPP2 pessimization is currently default in our devel/ccache
port due to Clang requiring it. Clang's -Wparentheses-equality,
-Wtautological-compare, and -Wself-assign warnings do not mix well with
compiling already-pre-processed code that may have expanded macros that
trigger the warnings. GCC has so far not had this issue so it is allowed to
disable the CCACHE_CPP2 default in our port.

Sharing a cache between multiple checkouts, or systems, is explained in
the ccache manual. Sharing a cache over NFS would likely not be worth
it, but syncing cache directories between systems may be useful for an
organization. There is also a memcached backend available [5]. Due to using
an object directory outside of the source directory though you will need to
ensure that both are in the same prefix and all users use the same layout. A
possible working layout is as follows:
Source: /some/prefix/src1
Source: /some/prefix/src2
Source: /some/prefix/src3
Objdir: /some/prefix/obj
Environment: CCACHE_BASEDIR='${SRCTOP:H}' MAKEOBJDIRPREFIX='${SRCTOP:H}/obj'
This will use src*/../obj as the MAKEOBJDIRPREFIX and tells ccache to replace
all absolute paths to be relative. Using something like this is required due
to -I and -o flags containing both SRC and OBJDIR absolute paths that ccache
adds into its hash for the object without CCACHE_BASEDIR.

distcc can be hooked into by setting CCACHE_PREFIX=/usr/local/bin/distcc.
I have not personally tested this and assume it will not mix well with
using the bootstrap compiler.

The cache from buildworld can be reused in a subdir by first running
'make buildenv' (from r290424).

Note that the cache is currently different depending on whether -j is
used or not due to ccache enabling -fdiagnostics-color automatically if
stderr is a TTY, which bmake only does if not using -j.

The system I used for testing was:
WITNESS
Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_CCACHE_BUILD=yes
DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
The arc was fully populated with src tree files and ccache objects.
RAM: 76GiB
CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16

The WITH_FAST_DEPEND feature was used for comparison here as well to show
the dramatic time savings with a full cache.

buildworld:
x buildworld-before
+ buildworld-ccache-empty
* buildworld-ccache-full
% buildworld-ccache-full-fastdep
# buildworld-fastdep
+-------------------------------------------------------------------------------+
|% * # +|
|% * # +|
|% * # xxx +|
| |A |
| A|
| A |
|A |
| A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 3744.13 3794.31 3752.25 3763.5633 26.935139
+ 3 4519 4525.04 4520.73 4521.59 3.1104823
Difference at 95.0% confidence
758.027 +/- 43.4565
20.1412% +/- 1.15466%
(Student's t, pooled s = 19.1726)
* 3 1823.08 1827.2 1825.62 1825.3 2.0785572
Difference at 95.0% confidence
-1938.26 +/- 43.298
-51.5007% +/- 1.15045%
(Student's t, pooled s = 19.1026)
% 3 1266.96 1279.37 1270.47 1272.2667 6.3971113
Difference at 95.0% confidence
-2491.3 +/- 44.3704
-66.1952% +/- 1.17895%
(Student's t, pooled s = 19.5758)
# 3 3153.34 3155.16 3154.2 3154.2333 0.91045776
Difference at 95.0% confidence
-609.33 +/- 43.1943
-16.1902% +/- 1.1477%
(Student's t, pooled s = 19.0569)

buildkernel:
x buildkernel-before
+ buildkernel-ccache-empty
* buildkernel-ccache-empty-fastdep
% buildkernel-ccache-full
# buildkernel-ccache-full-fastdep
@ buildkernel-fastdep
+-------------------------------------------------------------------------------+
|# @ % * |
|# @ % * x + |
|# @ % * xx ++|
| MA |
| MA|
| A |
| A |
|A |
| A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 571.57 573.94 571.79 572.43333 1.3094401
+ 3 727.97 731.91 728.06 729.31333 2.2492295
Difference at 95.0% confidence
156.88 +/- 4.17129
27.4058% +/- 0.728695%
(Student's t, pooled s = 1.84034)
* 3 527.1 528.29 528.08 527.82333 0.63516402
Difference at 95.0% confidence
-44.61 +/- 2.33254
-7.79305% +/- 0.407478%
(Student's t, pooled s = 1.02909)
% 3 400.4 401.05 400.62 400.69 0.3306055
Difference at 95.0% confidence
-171.743 +/- 2.16453
-30.0023% +/- 0.378128%
(Student's t, pooled s = 0.954969)
# 3 201.94 203.34 202.28 202.52 0.73020545
Difference at 95.0% confidence
-369.913 +/- 2.40293
-64.6212% +/- 0.419774%
(Student's t, pooled s = 1.06015)
@ 3 369.12 370.57 369.3 369.66333 0.79033748
Difference at 95.0% confidence
-202.77 +/- 2.45131
-35.4225% +/- 0.428227%
(Student's t, pooled s = 1.0815)

[1] https://ccache.samba.org/performance.html
[2] http://www.mail-archive.com/ccache@lists.samba.org/msg00576.html
[3] https://reviews.freebsd.org/D3484
[5] https://github.com/jrosdahl/ccache/pull/30

PR: 182944 [4]
MFC after: 3 weeks
Sponsored by: EMC / Isilon Storage Division
Relnotes: yes

Add a FAST_DEPEND option, off by default, which speeds up the build significantly.

This speeds up buildworld by 16% on my system and buildkernel by 35%.

Rather than calling mkdep(1), which is just a wrapper around 'cc -E',
use the modern -MD -MT -MF flags to gather and generate dependencies during
compilation. This flag was introduced in GCC "a long time ago", in GCC 3.0,
and is also supported by Clang. (It appears that ICC also supports this but I
do not have access to test it). This avoids running the preprocessor *twice*
for every build, in both 'make depend' and 'make all'. This is especially
noticeable when using ccache since it does not cache preprocessor results from
mkdep(1) / 'cc -E', but still speeds up compilation with the -MD flags.

For 'make depend' a tree-walk is still done to ensure that all DPSRCS
are generated when expected, and that beforedepend/afterdepend and
_EXTRADEPEND are all still respected. In time this may change but for now
I've been conservative. The time for a tree-walk with -j combined with
SUBDIR_PARALLEL is not significant. For example, it takes about 9 seconds
with -j15 to walk all of src/ for 'make depend' now on my system.

A .depend file is still generated with the various rules that apply to
the final target, or custom rules. Otherwise there are now
per-built-object-file .depend files, such as .depend.filename.o. These
are included directly by make rather than populating .depend with a loop
and .depend lines, which only added overhead to the now almost-NOP 'make
depend' phase.

Before this I experimented with having mkdep(1) called in parallel per-file.
While this improved the kernel and lib/libc 'make depend' phase, it resulted
in slower build times overall.

The -M flags are removed from CFLAGS when linking since they have no effect.

Enabling this by default, for src or out-of-src, can be done once more testing
has been done, such as a ports exp-run, and with more compilers.

The system I used for testing was:
WITNESS
Build options: -j20 WITH_LLDB=yes WITH_DEBUG_FILES=yes WITH_FAST_DEPEND=yes
DISK: ZFS 3-way mirror with very slow disks using SSD l2arc/log.
The arc was fully populated with src tree files.
RAM: 76GiB
CPU: Intel(R) Xeon(R) CPU L5520 @2.27GHz
2 package(s) x 4 core(s) x 2 SMT threads = hw.ncpu=16

buildworld:
x buildworld-before
+ buildworld-fastdep
+-------------------------------------------------------------------------------+
|+ |
|+ |
|+ xx x|
| |_MA___||
|A |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 3744.13 3794.31 3752.25 3763.5633 26.935139
+ 3 3153.34 3155.16 3154.2 3154.2333 0.91045776
Difference at 95.0% confidence
-609.33 +/- 43.1943
-16.1902% +/- 1.1477%
(Student's t, pooled s = 19.0569)

buildkernel:
x buildkernel-before
+ buildkernel-fastdep
+-------------------------------------------------------------------------------+
|+ x |
|++ xx|
| A||
|A| |
+-------------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 3 571.57 573.94 571.79 572.43333 1.3094401
+ 3 369.12 370.57 369.3 369.66333 0.79033748
Difference at 95.0% confidence
-202.77 +/- 2.45131
-35.4225% +/- 0.428227%
(Student's t, pooled s = 1.0815)

Sponsored by: EMC / Isilon Storage Division
MFC after: 3 weeks
Relnotes: yes

Add dependent option STAGING_MAN/STAGING

Use a single mlinks set, since some libs (eg. libc) do cross section
links.

Remove leftovers from r276551

Fix typo in comments.

Noticed by: brooks@

Acutally, replaces NO_INCS with WITHOUT_INCLUDES and make
WITHOUT_TOOLCHAIN imply it instead.

Sponsored by: Netflix

Revert r267233 for now. PIE support needs to be reworked.

1. 50+% of NO_PIE use is fixed by adding -fPIC to INTERNALLIB and other
build-only utility libraries.
2. Another 40% is fixed by generating _pic.a variants of various libraries.
3. Some of the NO_PIE use is a bit absurd as it is disabling PIE (and ASLR)
where it never would work anyhow, such as csu or loader. This suggests
there may be better ways of adding support to the tree. Many of these
cases can be fixed such that -fPIE will work but there is really no
reason to have it in those cases.
4. Some of the uses are working around hacks done to some Makefiles that are
really building libraries but have been using bsd.prog.mk because the code
is cleaner. Had they been using bsd.lib.mk then NO_PIE would not have
been needed.

We likely do want to enable PIE by default (opt-out) for non-tree consumers
(such as ports). For in-tree though we probably want to only enable PIE
(opt-in) for common attack targets such as remote service daemons and setuid
utilities. This is also a great performance compromise since ASLR is expected
to reduce performance. As such it does not make sense to enable it in all
utilities such as ls(1) that have little benefit to having it enabled.

Reported by: kib

The GNU texinfo and GNU info pages are not built and installed
anymore, WITH_INFO knob has been added to allow to built and install
them again.

Reviewed by: imp

In preparation for ASLR [1] support add WITH_PIE to support building with -fPIE.

This is currently an opt-in build flag. Once ASLR support is ready and stable
it should changed to opt-out and be enabled by default along with ASLR.

Each application Makefile uses opt-out to ensure that ASLR will be enabled by
default in new directories when the system is compiled with PIE/ASLR. [2]

Mark known build failures as NO_PIE for now.

The only known runtime failure was rtld.

[1] http://www.bsdcan.org/2014/schedule/events/452.en.html
Submitted by: Shawn Webb <lattera@gmail.com>
Discussed between: des@ and Shawn Webb [2]

Better documentation for why KERBEROS and OPENSSH are here, and where
to look for all the other options that used to be here (well, in
bsd.own.mk)

Suggested by: des@

Move DOCCOMPRESS to MK variable.

Migrate NO_WARN to MK_WARN. Support legacy NO_WARN usage. Remove a
check for EARLY_BUILD because it isn't necessary (MK_WARN=no will
always be defined for that).

All the NO_foo options processed in src.opts.mk are really bsd.opts.mk
options, so move their processing there. This fixes issues with
Makefiles that define NO_MAN=t and only inlcude bsd.*.mk files. A few
ports fell into this category, and they should be fixed by this change.
Also, for now, disable the warning about NO_foo being deprecated. More
work is needed than anticipated before we can do that, so kill the
noise for now.

Use src.opts.mk in preference to bsd.own.mk except where we need stuff
from the latter.

Move the /usr/src specific options to src.opts.mk. Move inclusion of
/etc/src.conf to this file as well. Now, it will only affect builds of
/usr/src and not others that use the bsd.*.mk files. Specifically
don't install src.opts.mk so we can catch when it 'leaks' into
bsd.*.mk again and have there be errors when this happens. Future
commits will move to including src.opts.mk instead of bsd.own.mk when
all that's needed is one of the MK_FOO options from src.opts.mk.
Future options should be placed here, unless they directly affect a
bsd.*.mk file, in which case they should be placed in bsd.opts.mk.

NLS is used by the build system.

Split up the options into those used only by bsd.mk.* or sys.mk and
the rest. Add note of caution for the surprises that popped up. MK_SSP
is the only one used by sys.mk, and it likely needs even extra
attention.

Now that all the NO_foo expressed in this loop have been removed from
the tree, start warning that NO_foo is deprecated. It was supposed to
be gone from the tree as a user-setting a long time ago...

Add option WITHOUT_VI to not build/install vi. vi was the largest
binary without a knob to turn it off.

Use MK_OPENLDAP in preference to WITH_OPENLDAP and make it a default
NO option to match the opt-in nature of the historical nature of this
option.

Retire compatibility YES_HESIOD. If you haven't upgraded in the last
decade to WITH_HESIOD, your are in for a rude surprise...

Move the generic part of bsd.opts.mk into bsd.mkopts.mk to allow for
the WITH/WITHOUT_FOO -> MK_FOO={yes,no} stuff to be used elsewhere.

Allow setting of MK_FOO by anybody. There are too many edge cases that
we're preventing now with this policy. However, these edge cases
should be rare and all that set MK_FOO directly.
WITH_FOO and WITHOUT_FOO both being defined now result in the
non-default behavior happening silently. Users needing determinism
here fall into the edge case exception for MK_FOO setting.

Copy bsd.own.mk to bsd.opts.mk and separate out the option setting
code from the rest. Include bsd.opts.mk in bsd.own.mk to preserve
current behavior. Future revisions will replace the inclusion of
bsd.own.mk elsewhere with bsd.opts.mk or a more appropriate new
file that's still being finalized.