llvm: Reduce overlinking with the minimal llvm

We only need to link against libz and libzstd when linking against the
fill libllvm, libllvmminimal doesn't use either library. Move adding
libz and libzstd to the list of libraries to link against to where
we decide to use the full libllvm.

Reported by: Cristian Marussi <Cristian.Marussi@arm.com>
Reported by: Colin S. Gordon <csgordon@fastmail.com>
Reviewed by: dim
Sponsored by: Arm Ltd
Differential Revision: https://reviews.freebsd.org/D42528

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

Move LIBADD lines from usr.bin/clang/*/Makefile one level up

Some utilities under usr.bin/clang were only linked to libz, while most
others were linked to libz and libzstd. Make this consistent, and remove
repetition, by moving these LIBADD lines to usr.bin/clang/clang.prog.mk
and usr.bin/clang/clang.prog.mk.

MFC after: 3 days

Merge llvm-project main llvmorg-15-init-15358-g53dc0f10787

This updates llvm, clang, compiler-rt, libc++, libunwind, lld, lldb and
openmp to llvmorg-15-init-15358-g53dc0f10787.

PR: 265425
MFC after: 2 weeks

Allow bootstrapping llvm-tblgen on macOS and Linux

This is needed in order to build various LLVM binutils (e.g. addr2line)
as well as clang/lld/lldb.

Co-authored-by: Jessica Clarke <jrtc27@FreeBSD.org>
Test Plan: Compiles on ubuntu 18.04 and macOS 11.4
Reviewed By: dim
Differential Revision: https://reviews.freebsd.org/D31057

Update build glue for a minimal build of the clang executable.

Fix packaging for clang, lldb and lld 3.9.0

During the upgrade of clang/llvm etc to 3.9.0 in r309124, the PACKAGE
directive in the usr.bin/clang/*.mk files got dropped accidentally.

Restore it, with a few minor changes and additions:
* Correct license in clang.ucl to NCSA
* Add PACKAGE=clang for clang and most of the "ll" tools
* Put lldb in its own package
* Put lld in its own package

Reviewed by: gjb, jmallett
X-MFC-With: 309124
Differential Revision: https://reviews.freebsd.org/D8666

Centralize where LLVM_SRCS, CLANG_SRCS and LLDB_SRCS are defined.

Missed one .mk file where OBJTOP can be used.

Use SRCTOP and OBJTOP throughout the llvm/clang/lldb build.

Completely revamp the way llvm, clang and lldb are built.

* Bootstrap llvm-tblgen and clang-tblgen with a minimal llvm static
library, that has no other dependencies.
* Roll up all separate llvm libraries into one big static libllvm.
* Similar for all separate clang and lldb static libraries.
* For all these libraries, generate their .inc files only once.
* Link all llvm tools (including extra) against the big libllvm.
* Link clang and clang-format against the big libllvm and libclang.
* Link lldb against the big libllvm, libclang and liblldb.

N.B.: This is work in progress, some details may still be missing.

It also heavily depends on bsd.*.mk's support for SRCS and DPSRCS with
relative pathnames, which apparently does not always work as expected.
For building llvm, clang and lldb though, it seems to work just fine.

The main idea behind this restructuring is maintainability and build
peformance. The previous large number of very small libraries, each
with their own generated files and dependencies was slow to traverse
and hard to understand.

Possible future improvements:
* Only build certain targets, e.g. for most regular users having just
one target will be fine. This will shave off some build time.
* Building the big llvm, clang and lldb libraries as shared (private)
libraries.
* Adding other components from the LLVM project, such as lld.

Fix mis-merged LDADD/DPADD

Sponsored by: The FreeBSD Foundation

MFH r289384-r293170

Sponsored by: The FreeBSD Foundation

Fix LDADD/DPADD that should be LIBADD.

Sponsored by: EMC / Isilon Storage Division

Merge from head

Update llvm/clang build glue.

Move clang binaries into FreeBSD-clang package

Merge from head@274682

Revert r267233 for now. PIE support needs to be reworked.

1. 50+% of NO_PIE use is fixed by adding -fPIC to INTERNALLIB and other
build-only utility libraries.
2. Another 40% is fixed by generating _pic.a variants of various libraries.
3. Some of the NO_PIE use is a bit absurd as it is disabling PIE (and ASLR)
where it never would work anyhow, such as csu or loader. This suggests
there may be better ways of adding support to the tree. Many of these
cases can be fixed such that -fPIE will work but there is really no
reason to have it in those cases.
4. Some of the uses are working around hacks done to some Makefiles that are
really building libraries but have been using bsd.prog.mk because the code
is cleaner. Had they been using bsd.lib.mk then NO_PIE would not have
been needed.

We likely do want to enable PIE by default (opt-out) for non-tree consumers
(such as ports). For in-tree though we probably want to only enable PIE
(opt-in) for common attack targets such as remote service daemons and setuid
utilities. This is also a great performance compromise since ASLR is expected
to reduce performance. As such it does not make sense to enable it in all
utilities such as ls(1) that have little benefit to having it enabled.

Reported by: kib

Merge head from 7/28

Replace all uses of libncurses and libtermcap with their wide character
variants. This allows usable file system images (i.e. those with both a
shell and an editor) to be created with only one copy of the curses library.

Exp-run: antoine
PR: 189842
Discussed with: bapt
Sponsored by: DARPA, AFRL

In preparation for ASLR [1] support add WITH_PIE to support building with -fPIE.

This is currently an opt-in build flag. Once ASLR support is ready and stable
it should changed to opt-out and be enabled by default along with ASLR.

Each application Makefile uses opt-out to ensure that ASLR will be enabled by
default in new directories when the system is compiled with PIE/ASLR. [2]

Mark known build failures as NO_PIE for now.

The only known runtime failure was rtld.

[1] http://www.bsdcan.org/2014/schedule/events/452.en.html
Submitted by: Shawn Webb <lattera@gmail.com>
Discussed between: des@ and Shawn Webb [2]

Pay attention to MK_SHARED_TOOLCHAIN

Upgrade our copy of llvm/clang to 3.4 release. This version supports
all of the features in the current working draft of the upcoming C++
standard, provisionally named C++1y.

The code generator's performance is greatly increased, and the loop
auto-vectorizer is now enabled at -Os and -O2 in addition to -O3. The
PowerPC backend has made several major improvements to code generation
quality and compile time, and the X86, SPARC, ARM32, Aarch64 and SystemZ
backends have all seen major feature work.

Release notes for llvm and clang can be found here:
<http://llvm.org/releases/3.4/docs/ReleaseNotes.html>
<http://llvm.org/releases/3.4/tools/clang/docs/ReleaseNotes.html>

MFC after: 1 month

Sprinkle a bit of style.Makefile(5) across various clang Makefiles. No
functional changes.

MFC after: 3 days

Import the build makefiles for clang/LLVM.

Approved by: ed (mentor)

Allow bootstrapping llvm-tblgen on macOS and Linux

This is needed in order to build various LLVM binutils (e.g. addr2line)
as well as clang/lld/lldb.

Co-authored-by: Jessica Clarke <jrtc27@FreeBSD.org>
Test Plan: Compiles on ubuntu 18.04 and macOS 11.4
Reviewed By: dim
Differential Revision: https://reviews.freebsd.org/D31057

Update build glue for a minimal build of the clang executable.

Fix packaging for clang, lldb and lld 3.9.0

During the upgrade of clang/llvm etc to 3.9.0 in r309124, the PACKAGE
directive in the usr.bin/clang/*.mk files got dropped accidentally.

Restore it, with a few minor changes and additions:
* Correct license in clang.ucl to NCSA
* Add PACKAGE=clang for clang and most of the "ll" tools
* Put lldb in its own package
* Put lld in its own package

Reviewed by: gjb, jmallett
X-MFC-With: 309124
Differential Revision: https://reviews.freebsd.org/D8666

Centralize where LLVM_SRCS, CLANG_SRCS and LLDB_SRCS are defined.

Missed one .mk file where OBJTOP can be used.

Use SRCTOP and OBJTOP throughout the llvm/clang/lldb build.

Completely revamp the way llvm, clang and lldb are built.

* Bootstrap llvm-tblgen and clang-tblgen with a minimal llvm static
library, that has no other dependencies.
* Roll up all separate llvm libraries into one big static libllvm.
* Similar for all separate clang and lldb static libraries.
* For all these libraries, generate their .inc files only once.
* Link all llvm tools (including extra) against the big libllvm.
* Link clang and clang-format against the big libllvm and libclang.
* Link lldb against the big libllvm, libclang and liblldb.

N.B.: This is work in progress, some details may still be missing.

It also heavily depends on bsd.*.mk's support for SRCS and DPSRCS with
relative pathnames, which apparently does not always work as expected.
For building llvm, clang and lldb though, it seems to work just fine.

The main idea behind this restructuring is maintainability and build
peformance. The previous large number of very small libraries, each
with their own generated files and dependencies was slow to traverse
and hard to understand.

Possible future improvements:
* Only build certain targets, e.g. for most regular users having just
one target will be fine. This will shave off some build time.
* Building the big llvm, clang and lldb libraries as shared (private)
libraries.
* Adding other components from the LLVM project, such as lld.

Fix mis-merged LDADD/DPADD

Sponsored by: The FreeBSD Foundation

Fix LDADD/DPADD that should be LIBADD.

Sponsored by: EMC / Isilon Storage Division

Update llvm/clang build glue.

Move clang binaries into FreeBSD-clang package

Revert r267233 for now. PIE support needs to be reworked.

1. 50+% of NO_PIE use is fixed by adding -fPIC to INTERNALLIB and other
build-only utility libraries.
2. Another 40% is fixed by generating _pic.a variants of various libraries.
3. Some of the NO_PIE use is a bit absurd as it is disabling PIE (and ASLR)
where it never would work anyhow, such as csu or loader. This suggests
there may be better ways of adding support to the tree. Many of these
cases can be fixed such that -fPIE will work but there is really no
reason to have it in those cases.
4. Some of the uses are working around hacks done to some Makefiles that are
really building libraries but have been using bsd.prog.mk because the code
is cleaner. Had they been using bsd.lib.mk then NO_PIE would not have
been needed.

We likely do want to enable PIE by default (opt-out) for non-tree consumers
(such as ports). For in-tree though we probably want to only enable PIE
(opt-in) for common attack targets such as remote service daemons and setuid
utilities. This is also a great performance compromise since ASLR is expected
to reduce performance. As such it does not make sense to enable it in all
utilities such as ls(1) that have little benefit to having it enabled.

Reported by: kib

Replace all uses of libncurses and libtermcap with their wide character
variants. This allows usable file system images (i.e. those with both a
shell and an editor) to be created with only one copy of the curses library.

Exp-run: antoine
PR: 189842
Discussed with: bapt
Sponsored by: DARPA, AFRL

In preparation for ASLR [1] support add WITH_PIE to support building with -fPIE.

This is currently an opt-in build flag. Once ASLR support is ready and stable
it should changed to opt-out and be enabled by default along with ASLR.

Each application Makefile uses opt-out to ensure that ASLR will be enabled by
default in new directories when the system is compiled with PIE/ASLR. [2]

Mark known build failures as NO_PIE for now.

The only known runtime failure was rtld.

[1] http://www.bsdcan.org/2014/schedule/events/452.en.html
Submitted by: Shawn Webb <lattera@gmail.com>
Discussed between: des@ and Shawn Webb [2]

Pay attention to MK_SHARED_TOOLCHAIN

Sprinkle a bit of style.Makefile(5) across various clang Makefiles. No
functional changes.

MFC after: 3 days

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Import the build makefiles for clang/LLVM.

Approved by: ed (mentor)