Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

etcupdate: Remove redundant semicolons

Fixes: 03e62670c33c ("etcupdate: Consolidate nobuild cases and make more robust")

etcupdate: Use new buildetc and installetc targets when available

These new targets avoid the need to invoke internal build system targets
or set internal variables when building, and also have the added benefit
of working with BUILD_WITH_STRICT_TMPPATH. Old source trees lacking such
targets will not work with BUILD_WITH_STRICT_TMPPATH; they could be made
to work by copying the steps, but it's not worth doing so, as they never
have worked in the past. The primary goal of this is to support changing
the default of BUILD_WITH_STRICT_TMPPATH to enabled.

Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D41206

etcupdate: Consolidate nobuild cases and make more robust

The distrib-dirs and distribution steps are shared between the two, the
only difference is whether MAKEOBJDIRPREFIX is in the environment for
the latter. Having in the environment for the former is currently not
needed but does no harm and will be needed in future, so we can just
export it up-front in the subshell. When we do distrib-dirs relative to
_obj and everything also doesn't matter, so move it next to distribution
where it makes more sense. Finally, to avoid complicated && chains, use
"|| exit 1" everywhere to make the subshell fail, and add an extra one
on to the cd $SRCDIR to handle that failing (otherwise we'd go on and
try to build the current directory after cd prints its error, which is
unhelpful).

These changes will make it easier to bundle these steps up into new
top-level targets to allow the build system to manage the steps rather
than etcupdate, which will also handle BUILD_WITH_STRICT_TMPPATH, which
currently does not work with etcupdate.

Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D41204

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

etcupdate: Fix -N support for build command

Whilst febca0e64361 added -N for both build and extract, it only fully
worked for extract; build would perform the actual tree build with
-DNO_ROOT and construct the intended METALOG, but the subsequent tarball
creation did not take this into account and just tarred up the the
directory as-is rather than using the METALOG. This resulted in the
permissions and ownership not being correct, as well as there being a
stray METALOG file in the tarball's root.

Reported by: avg
Reviewed by: avg, jhb, imp
Tested by: avg
Fixes: febca0e64361 ("etcupdate: Add a -N flag to perform a NO_ROOT build")
Differential Revision: https://reviews.freebsd.org/D36845

etcupdate: Add a -N flag to perform a NO_ROOT build

This is in preparation for including an etcupdate tree when performing a
-DNO_ROOT release image build. Although -DNO_ROOT can be passed via -M,
to be useful we need to mangle the resulting METALOG to mirror the
various cleanups to the tree that are done after the build (removing
generated files, empty files and empty directories), so etcupdate needs
its own flag.

Reviewed by: jhb, pauamma
Obtained from: CheriBSD
Differential Revision: https://reviews.freebsd.org/D35857

etcupdate: Prefer POSIX -depth to BSD -d

This is in preparation for building an etcupdate tree on non-FreeBSD
when building release images. The -d option is documented as a
BSD-specific equivalent to the POSIX -depth primary. Whilst GNU find
sort of accepts it in an attempt to be compatible, it still doesn't
permit it coming before the paths, unlike BSD find, and prints a
deprecation warning either way. Thus, use the equivalent POSIX -depth to
ensure it works correctly and without warning everywhere.

Reviewed by: jhb
Obtained from: CheriBSD
Differential Revision: https://reviews.freebsd.org/D35856

etcupdate: Add a -m flag to change the make binary that's run

This will allow release/Makefile to forward on ${MAKE} to allow building
on non-FreeBSD systems where ${MAKE} is something other than make, as
make is typically GNU make in such situations.

Reviewed by: jhb, pauamma
Obtained from: CheriBSD
Differential Revision: https://reviews.freebsd.org/D35854

etcupdate: Preserve permissions when installing a resolved file.

Similar to the change in 1a04446f088c79cc2cf85fd86e60ebcc228d3075, use
cat to overwrite the contents of the existing file rather than cp so
that metadata of the existing file such as permissions and ownership
is preserved.

PR: 255514
Reported by: uqs
MFC after: 1 week

etcupdate: Don't rotate trees for a dry run.

When performing a dry run, remove the temporary tree created rather
than rotating the trees. Rotating the trees meant that etcupdate
thought the latest changes were already merged and would not merge
them on the next real run.

PR: 260281
Reported by: Tomoaki AOKI <junchoon@dec.sakura.ne.jp>
Fixes: 0611aec3cf3a etcupdate: Always extract to a temporary tree.
MFC after: 1 week

etcupdate: remove redundant diff3 flag

-A and -m select different output modes output modes for diff3. When
both flags are present gnu diff3 prefers -m, drop the extra -A flag in
etcupdate.

Reviewed by: pstef, 0mp
MFC after: 3 days
Sponsored by: Klara Inc
Differential Revision: <https://reviews.freebsd.org/D###>

secure/caroot, certctl: Rename secure/caroot/blacklisted

Old certctl commands still work for compatability, but are deprecated.

Approved by: secteam (gordon)
Differential Revision: https://reviews.freebsd.org/D30807

etcupdate: Add -D destdir to usage for 'extract'.

Reported by: Mark Millard <marklmi@yahoo.com>
MFC after: 1 week

etcupdate: Remove the old pre-world tree when rotating for -p.

This fixes a bug in an earlier change to move tree rotation to
the end of the update where the step to make room for the new
preworld tree was deleting the old "current" tree instead of
the old "preworld" tree.

Reported by: olivier, dhw
Fixes: 0611aec3cf3a373e6a06f103699dbc91c3d6d472
MFC after: 2 weeks

etcupdate: Add a revert mode to restore one or more stock files.

Reviewed by: imp
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D29846

etcupdate: Trim trailing whitespace.

Reviewed by: imp
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D29845

etcupdate: Gracefully handle SIGINT when building trees.

Run the 'build_tree' function inside of a subshell and trap SIGINT to
return an error to the caller. This allows callers to gracefully
cleanup a partially created tree.

While here, redirect stdout/stderr of the subshell to the log file
instead of applying redirections individually to each command executed
while building the tree.

Reviewed by: imp
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D29844

etcupdate: Always extract to a temporary tree.

etcupdate has had a somewhat nasty race condition since its creation
in that its state machine can get very confused if it is interrupted
while building the tree to compare against. This is exacerbated by
the fact that etcupdate doesn't emit any output while building the
tree which can take several seconds (especially in recent years with
the addition of the tree-wide buildconfig/installconfig passes).

To mitigate this, always install a new tree into a temporary directory
created via mktemp as was previously done only for dry-runs via -n.
The existing trees are only rotated and the new tree installed as
/var/db/etcupdate/current after the update command has completed.

Reported by: dim, np (and many others)
Reviewed by: imp
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D29843

Unbreak etcupdate(8) and mergemaster(8) after r352950

r352950 introduced improper case fall-through for shell scripts. Fix it with
a pipe.

Reported by: lwhsu, David Wolfskill

[3/3] etcupdate and mergemaster support for certctl

This commit add support for certctl in mergemaster and etcupdate. Both will
either rehash or prompt for rehash as new certificates are
trusted/blacklisted.

This work was done primarily by allanjude@, with minor contributions by
myself.

No objection from: secteam
Differential Revision: https://reviews.freebsd.org/D17389

Move back group, master.passwd and shells to etc directory

Use the .PATH mechanism instead so keep installing them from lib/libc/gen

While here revert 347961 and 347893 which are no longer needed

Discussed with: manu
Tested by: manu
ok manu@

etcupdate: Add missing directory when building the tree

Missed in 348151

Reported by: lwshu, ci

etcupdate: Fix -p after the move of the passwd related files

Since PREWORLD_FILES only contain files that needs to be copied into /etc
copy directly those files there.

Reported by: many

Fix mergemaster after r347638 and the master.passwd / group move.

Check the legacy directory and use it instead if present.

Install these first if using beinstall.

UPDATING entry to follow.

Approved by: allanjude (mentor, in person)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D20279

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

etcupdate: preserve the metadata of the destination file

When using diff3 to perform a three-way merge, etcupdate lost the destination
file's metadata. The metadata from the temporary file were used instead.
This was unpleasant for rc.d scripts, which require execute permission.
Use "cat >" to overwrite the destination file's contents while preserving its
metadata.

Reviewed by: bapt
Sponsored by: Dell Technologies
Differential Revision: https://reviews.freebsd.org/D7817

etcupdate: directly use diff3(1) instead of merge(1)

During the last attempt to rmeove GNU rcs, 2 blockers were spotted:
We need an ident(1) and etcupdate(8) uses merge(1).

Now nothing should prevent to remove rcs from base

Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D7401

META_MODE: Don't require filemon(4) for mergemaster(8)/etcupdate(8)

New .meta files will be created without filemon data, but any future build
that wants filemon data will force a rebuild due to the missing data
due to use of bmake's .MAKE.MODE=missing-filemon=yes feature.

Reported by: np
Sponsored by: EMC / Isilon Storage Division
MFC after: 3 days

tzsetup(1) -> tzsetup(8)

MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Reassign copyright statements on several files from Advanced
Computing Technologies LLC to Hudson River Trading LLC.

Approved by: Hudson River Trading LLC (who owns ACT LLC)
MFC after: 1 week

Generate /var/db/services.db during 'make distribution' so that it is
present during new installs. Update etcupdate and mergemaster to
ignore the generated file.

Tested by: gjb (release build)
MFC after: 1 month

- Refresh /etc/localtime after each update using tzsetup -r.
- Regenerate /var/db/services.db when /etc/services changes.

MFC after: 1 week

Fix a couple of issues with -F:
- Fix ALWAYS_INSTALL to take precedence over the FreeBSD ID checks.
In particular, always install a file where the only change was
the FreeBSD ID even if -F is specified.
- Fix the -F option in the case that the only upstream change is a
change in the FreeBSD ID and the local file is removed.
- Add tests for these two cases.

Escape a newline added in the previous commit when verifying the flags
passed to the build, diff, or status commands.

Add a pre-world mode of updating similar to the -p option that can be
passed to mergemaster. In this mode, only changes to /etc/master.passwd
and /etc/group are merged to /etc. In addition, it uses a temporary
tree to stage these changes rather than overwriting the existing
'current' and 'previous' trees so that a full update can be run after
a normal installworld has completed.

MFC after: 2 weeks

The etcupdate utility is a tool for managing updates to files that are
not updated as part of `make installworld' such as files in /etc. It
manages updates by doing a three-way merge of changes made to these files
against the local versions. It is also designed to minimize the amount
of user intervention with the goal of simplifying upgrades for clusters
of machines.

The primary difference from mergemaster is that etcupdate requires less
manual work. The primary difference from etcmerge is that etcupdate
updates files in-place similar to mergemaster rather than building a
separate /etc tree.

Requested by: obrien, kib, theraven, joeld (among others)