freebsd-update: Correctly check if pkg(8) is present

On systems without pkg(8) installed, `command -v pkg` will return
success and falsely report that pkg(8) is present. Fix that by checking
via the `pkg -N` form.

This is missing from the final revision of D39695.

Reported by: delphij
Reviewed by: fernape, delphij
Fixes: bc0c6c9cf3a9 freebsd-update: Add check for kernel modules
Differential Revision: https://reviews.freebsd.org/D45292

Remove remnants of portsnap(8)

This was prompted by noticing that '/var/db/portsnap' still exists on
newly-installed machines.

With this change, all mentions of portsnap(8) in the tree are gone,
except for the historical note in the AUTHORS section of manpage
phttpget(8).

locate(1) will thus start indexing again '/var/db/portsnap' on machines
where this directory still exists, which may be a good way to push
administrators to delete it.

Reviewed by: cperciva
Approved by: emaste (mentor)
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D45023

freebsd-update(8): Use kern.module_path

Instead of tailored configuration files to look for module_path entries.

Reported by: kevans@
Reviewed by: kevans, imp, zlei
Approved by: kevans,zlei
Differential Revision: https://reviews.freebsd.org/D44797

freebsd-update: Add check for kernel modules

People get confused when some software (VirtualBox, etc) does not work as
expected (or at all) after a major upgrade.

We have a nice way to deal with this when using sources, namely including
PORTS_MODULES in /etc/make.conf, but we lack something similar for binary
updates.

This patch retrieves a list of kernel modules installed from packages and
advises the user to recompile from ports to avoid problems.

Approved by: zlei@
Differential Revision: https://reviews.freebsd.org/D39695

freebsd-update: Fix typo in comment

s/patchname expansion/pathname expansion/

Signed-off-by: a.biardi@tiscali.it
Pull-request: https://github.com/freebsd/freebsd-src/pull/1152

freebsd-update: mark "cron" as fetched as "fetch" itself

The change in 33bd05c3187d7b49c80cf1b0132b405c105d0833 was incomplete
because it did not mark "cron" as ISFETCHED=1 although it performs the
same operations as "install", but less output and does not perform a
hard exit. Mark result as such and make "install" know that updates have
been fetched.

PR: 277699
Approved by: jrm (mentor), emaste, cperciva
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D44356

freebsd-update: Don't provide copiable commands in output

Previously, freebsd-update provided ready-to-go commands for copying and
pasting into the terminal. This causes problems as soon as options are
used and not supplied again by the user, e.g., '-b' or '-d'.
Stop making them copiable and force the user to construct a valid command
line by himself to avoid failures.

PR: 276102
Approved by: jrm (mentor), emaste
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D43700

freebsd-update: allow user to break out of conflict resolution loop

When a conflict marker is found during update allow the user to confirm
they want it to remain in the file.

Suggested by: Tim Hogard
Reviewed by: imp
Sponsored by: The FreeBSD Foundation
Fixes: ceb5f28ba5fc ("freebsd-update: re-edit files if merge confli...")
Differential Revision: https://reviews.freebsd.org/D38896

freebsd-update: for rollback, first create directories

rollback_files() tried to install files before creating the directories
for those files. In some cases this is due to special handling to
install certain classes of files earlier than others.

Just create all directories up front when performing rollback.

PR: 273950
Reviewed by: dim
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D42020

freebsd-update: handle directories changing to files

Further to f6d37c9ca13f ("freebsd-update: handle file -> directory on
upgrade"), handle the reverse case of a directory changing to a file.
We may not encounter this case on upgradess (before freebsd-update is
retired) but it is needed to support rollback.

PR: 273950
Reviewed by: dim
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41945

freebsd-update: create deep BEs by default

The -r flag to bectl needs to go away, and we need to just do the right
thing. In the meantime, we can apply an -r in freebsd-update as a
minimal fix to stop creating partial backups in these (non-default) deep
BE setups.

PR: 267535

freebsd-update: handle file -> directory on upgrade

Upgrading from FreeBSD 13.2 to 14.0 failed with
install: ///usr/include/c++/v1/__string exists but is not a directory
because __string changed from a file to a directory with an LLVM
upgrade.

Now, remove the existing file when the type conflicts. Note that this
is only an interim fix to facilitate upgrades from 13.2 for 14.0 BETA
testing. This change does not handle the directory -> file case and
further work is needed.

PR: 273661
Reviewed by: dim, gordon
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41893

freebsd-update: Allow for upper/lowercase y/n

Allow for upper/lowercase y/n in "Does this look right?" question.

Reviewed by: emaste
Approved by: emaste
Differential revision: https://reviews.freebsd.org/D40434

freebsd-update: do not restart sshd when updating jail/basedir

In 6cd1bc531609 for PR 263489 I changed freebsd-update to restart sshd
after upgrade, to avoid an upgrade-related incompatibility that made it
impossible to login.

This is intended to avoid losing access to remote hosts, and ought not
apply to upgrading jails (from outside).

PR: 263489, 272282
Reported by: otis
Reviewed by: otis, kevans
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Fixes: 6cd1bc531609 ("freebsd-update: restart sshd after upgrade")
Differential Revision: https://reviews.freebsd.org/D41890

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

freebsd-update: Fix merging already-updated files

When performing an "upgrade" (moving between FreeBSD releases, as
opposed to "update" which merely applies security/errata updates
to the installed release) FreeBSD Update:

1. Generates a list of "files needing to be merged", namely those
files which don't match the version installed in the "old" release
and have paths matching the MergeChanges configuration directive
(by default, /boot/device.hints and everything under /etc/).

and later on,

2. Compares the currently-installed files to the versions in the
"new" release, removing index entries for files which "don't need
to be updated because they're not changing".

Unfortunately if a file falls into both of these categories -- that
is to say, if a file in /etc/ is the same as the version in the new
release and not the same as the version in the old release -- the
resulting "merge" step saw that the file was no longer listed as
being part of the new release, resulting in the file being deleted.

For the first 18 years of FreeBSD Update's existence, this never
happened, since $FreeBSD$ tags resulted in "new release" files
always being different from any files systems would already have
installed.

This commit fixes this behaviour by only placing a file into the
"files needing to be merged" list if it does not match the version
in the old release *or* the version in the new release.

Reported by: des
Reviewed by: delphij (earlier version), des, emaste
MFC after: 7 days
X-EN-Candidate: yes
Differential Revision: https://reviews.freebsd.org/D39973

freebsd-update: Mention 13.2-RELEASE in usage.

freebsd-update: use grep -E instead of egrep

GNU egrep emits a warning that it is obsolescent and suggests grep -E
instead. Switch to grep -E in case we end up invoking GNU (e)grep (and
for consistency with other invocations in this file).

Reported by: Steffen Nurpmeso
Sponsored by: The FreeBSD Foundation

freebsd-update: re-edit files if merge conflict markers remain

freebsd-update will open ${EDITOR} if conflicts occur while merging
updates to config files. Inform the user if they've left conflict
markers behind, and go back to editing the file.

PR: 185546
PR: 229689
Reviewed by: delphij
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D37703

freebsd-update: avoid effects of user's environment on grep

A user had GREP_OPTIONS containing --color=always, which broke grep use
in freebsd-update. Unset this environment variable.

PR: 255990
Reviewed by: kevans
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

freebsd-update: restart sshd after upgrade

Sometimes the parent-child sshd protocol changes during an upgrade, and
when this happens sshd will not accept new connections until it is
restarted.

PR: 263489
Reviewed by: kevans, gjb
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D35109

freebsd-update: improve BE creation feature

This addresses one nit and one bug in the BE creation feature of
freebsd-update:

The nit addressed is that it currently only names the BEs after the
userland version, but the kernel version may be higher. After this
change, we request both and pass them through sort(1) to choose the
highest. This is especially helpful if a freebsd-update patch touched
one but not the other.

The bug fixed is that roots updated that are not located at '/', e.g.,
by using -b or -j, will no longer create boot environments
automatically. There's a very low chance these will actually change the
BE in any meaningful way, anyways. It could make sense in the future
to allow an argument-override to create the BE anyways if someone comes
up with a non-standard setup, e.g., where a jail is an important part of
their boot environment on an appliance or some such setup.

Half of this patch is submitted by delphij@, the other half kevans@.

PR: 261446
MFC after: 3 days
Reviewed by: delphij, emaste, Dave Fullard <dave_fullard.ca>
Differential Revision: https://reviews.freebsd.org/D34257

freebsd-update(8): Add -j flag to support jails

Make freebsd-update(8) support jails by adding the -j flag which takes
a jail jid or name as an argument. This takes advantage of the recently
added -j support to freebsd-version(8) in order to get the version of
the installed userland.

Reviewed by: dteske, kevans
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D25711

freebsd-update: create a ZFS boot environment on install

Updated freebsd-update to allow it to create boot environments using
bectl should the system support it. The bectl utility was updated in
r352211 (490e13c1403f) to support a 'check' to determine if the system
supports boot environments. If UFS is used, the bectl check will fail
then no attempt will be made to create the boot environment.

If freebsd-update is run inside a jail, no attempt will be made to
create a boot environment.

The boot environment function will create a new environment using the
format: current FreeBSD kernel version and date/timestamp, example:

12.0-RELEASE-p10_2019-10-03_185233

This functionality can be disabled by setting 'CreateBootEnv' in
freebsd-update.conf to 'no'.

Discussed with: allanjude
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D21892

freebsd-update: Update URL of supported platforms information

MFC after: 3 days

freebsd-update: improve mandoc db generation

freebsd-update compares the dates on man pages with mandoc.db, and if
any newer pages are found it regenerates mandoc.db.

Previously, if mandoc.db did not already exist the check failed and
freebsd-update then failed to create one. Now, check that mandoc.db
exists before performing the check for newer pages.

Reported by: bdrewery (in D10482)
Reviewed by: gordon
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29575

freebsd-update: unconditionally regenerate passwd/login.conf files

The existing logic is nice in theory, but in practice freebsd-update will
not preserve the timestamps on these files. When doing a major upgrade, e.g.
from 12.1-RELEASE -> 12.2-RELEASE, pwd.mkdb et al. appear in the INDEX and
we clobber the timestamp several times in the process of packaging up the
existing system into /var/db/freebsd-update/files and extracting for
comparisons. This leads to these files not getting regenerated when they're
most likely to be needed.

Measures could be taken to preserve timestamps, but it's unclear whether
the complexity and overhead of doing so is really outweighed by the marginal
benefit.

I observed this issue when pkg subsequently failed to install a package that
wanted to add a user, claiming that the user was removed in the process.
bapt@ pointed to this pre-existing bug with freebsd-update as the cause.

PR: 234014, 232921
Reviewed by: bapt, emaste
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D27635

Move finalize_components_config from get_params to cmd_*.

This allows us to redirect its output in cmd_cron, so that the
"src component not installed, skipped" message will be treated
the same way as other output from freebsd-update cron: Sent
in an email to root (or other address specified) if there are
updates to install, and silenced otherwise.

PR: 202492
MFC After: 1 week
Differential Revision: https://reviews.freebsd.org/D26432

Unbreak `freebsd-update updatesready'.

The command would only work if PWD happened to be WORKDIR.
Also, exit 1 in case WORKDIR exists, but isn't accessible
by the current user.

PR: 242709
Reported by: Max Fiedler
MFC after: 1 week

freebsd-update: rehash certs

With the inclusion of caroot bits, we'll need to also rehash on update as we
do in mergemaster/etcupdate.

If certctl's installed on the system, just unconditionally rehash. This
isn't an expensive operation, and we can refine it to compare
INDEX-{OLD,NEW} later if we really want to.

Reviewed by: emaste, allanjude
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D21805

freebsd-update: Add `updatesready' and `showconfig' commands

`freebsd-update updatesready' can be used to check if there are any pending
fetched updates that can be installed.

`freebsd-update showconfig' writes freebsd-update's configuration to
stdout.

This also changes the exit code of `freebsd-update install' to 2 in case
there are no updates pending to be installed and there wasn't a fetch phase
in the same invocation. This allows scripts to tell apart these error
conditions without breaking existing jail managers.

See freebsd-update(8) for details.

PR: 240757, 240177, 229346
Reviewed by: manpages (bcr), sectam (emaste), yuripv
Differential Revision: https://reviews.freebsd.org/D21473

Fix src component detection

Reviewed by: emaste
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D21579

freebsd-update: make usage output consistent

Drop trailing . which appeared only on description of IDS.

Submitted by: grembo
Event: EuroBSDCon Norway FreeBSD DevSummit

freebsd-update: restore old exit code when no updates are available locally

This unbreaks ezjail and iocell, which get into an endless loop trying to
figure out how many times "freebsd-update install" needs to be called.

PR: 229346
Submitted by: Mike Cole <mcole36@gmail.com>
Approved by: bapt
MFC after: 1 week

freebsd-update: Clarify help text

Improve help text to include example release numbers for reference
and clarify the -F option.

PR: 231185, 214619
Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
Reviewed by: delphij, rgrimes
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D18944

freebsd-update: regenerate man page database after update

These are currently not reproducible because they're built by the
makewhatis on the freebsd-update build host, not the one in the tree.
Regenerate after update, and later we can avoid including it in
freebsd-update data.

PR: 214545, 217389
Reviewed by: delphij
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D10482

Fix a typo/wordsmith a description modified in r343407

r343407 accidentally introduced a typo (folling -> following). While
reading the change out loud, I realized that the original sentence was
wordy. almost sounding like a run-on sentence.

Improve the flow by splitting up the two thoughts into two distinct sentence
fragments.

PR: 194547, 208497
Reviewed by: emaste
Approved by: emaste (mentor)
MFC after: 1 month
MFC with: r343407
Differential Revision: https://reviews.freebsd.org/D18947

freebsd-update: Clarify unsupported upgrade message

PR: 204115
Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
Reviewed by: delphij
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D18916

freebsd-update: fix style from r343271 change

PR: 234771
Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
MFC with: r343271
Sponsored by: The FreeBSD Foundation

freebsd-update: Stop installing empty component sets

Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
Reported by: rgrimes
Reviewed by: delphij
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D18927

freebsd-update: open $PAGER only if necessary

PR: 194547, 208497
Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
Reviewed by: delphij
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

freebsd-update: Allow upgrade from pre-release builds

Update r343122 to include -ALPHA, -BETA and -RC releases as
upgrade-able via freebsd-update.

PR: 234771
Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
Reported by: delphij, des
Reviewed by: delphij
MFC with: r343122
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D18881

freebsd-update: Update /etc/passwd after password db changes

Add -p to pwd_mkdb in order to ensure password db changes are also
included in /etc/passwd.

PR: 165954, 232921, 229487
Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
Reviewed by: jilles
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D18827

frebsd-update: fix --currently-running after r343122

PR: 234771
Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
Reported by: Brandon Schneider
MFC with: r343122
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D18896

freebsd-update: Use BASEDIR when checking for src component

src could potentially be installed under the based dir
and not under the root or vice versa.

PR: 224048
Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
Reviewed by: delphij
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D18849

freebsd-update: Clarify unsupported release upgrade error message

Notify users that upgrading from -CURRENT or -STABLE is unsupported by
freebsd-update.

Also ensure --currently-running provides a correctly formatted release
(as done by -r).

PR: 234771
Submitted by: Gerald Aryeetey <aryeeteygerald_rogers.com>
Reported by: yuri
Reviewed by: bcran
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D18803

freebsd-update: add a progress report for the "fetching files..."

After patching step is done.

PR: 232857
Submitted by: mat@
MFC after: a week

Switch the default pager for most commands to less

Finally, a pager for the nineties.

MFC after: Never
Relnotes: Yes
Differential Revision: https://reviews.freebsd.org/D13465
Poll: https://reviews.freebsd.org/V7

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

freebsd-update: do not duplicate patchlist entries

PR: 221079
Submitted by: Masachika ISHIZUKA
Submitted by: ota@j.email.ne.jp
Reviewed by: cperciva
MFC after: 1 week

Fix freebsd-update(8) erroneous message and exit status when "fetch install" used.

PR: 190660
Reviewed by: allanjude
Approved by: emaste
Differential Revision: https://reviews.freebsd.org/D12037

MFportsnap r264740: Use case insensitive match when parsing host(1) output.

Some DNS caches turn "FreeBSD.org" into "freebsd.org", which was causing
the printed SRV records to not match our regex.

PR: 170503
MFC after: 2 weeks

In preparation for removal of GNU rcs, replace merge(1) usage with direct
diff3(1) usage

Allow - in distribution names. This is needed for freebsd-update to work
with 11.0+, where the debugging symbols use a new naming scheme for release
distribution files.

Errata candidate.

Approved by: cperciva
Differential Revision: https://reviews.freebsd.org/D7170

Provide a future release as an example, instead of a historical one.

Add debug file extension to freebsd-update(8) after r288176

Reviewed by: delphij
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3784

Add --currently-running <release> option to freebsd-update.

This option tells freebsd-update to act as if it is running a specific
release instead of querying the kernel. In particular, this can be
useful when upgrading jails.

Requested by: EuroBSDCon devsummit jails session
Tested by: allanjude
MFC after: 1 week

Skip src component if /usr/src is empty.

Differential Revision: https://reviews.freebsd.org/D2364
Submitted by: kczekirda
Reviewed by: cperciva, delphij, nwhitehorn, allanjude
MFC after: 2 weeks

Fix syntax errors in conditions for new features in freebsd-update

Differential Revision: https://reviews.freebsd.org/D1550
Submitted by: kmoore
Approved by: delphij
Obtained from: PCBSD
MFC after: 1 week
X-MFC-With: 279571
Sponsored by: ScaleEngine Inc.

- Include commas in valid file-name chars freebsd-update will support

Approved by: cperciva

Add a new safetly belt to freebsd-update to prevent a user doing a minor update (-pX) while having an unfinished major upgrade (9.x to 9.y)
Safetly belt can be disabled with the -F flag

Additionally, add the --not-running-from-cron flag they bypasses the TTY requirement, and allows freebsd-update to be invoked by orchestration frameworks, scripts, or otherwise.

PR: 196760
Differential Revision: https://reviews.freebsd.org/D1550
Reviewed by: cperciva, delphij
Approved by: bcr (mentor), rodrigc (src)
MFC after: 1 month
Relnotes: yes
Sponsored by: ScaleEngine Inc.

(backup_kernel_finddir, backup_kernel, install_files):
Add missing references to $BASEDIR, in order to allow correct operation
when updating a system mounted at another location than / (e.g. when
updating an alternate Boot Environment).

Reviewed by: cperciva
MFC after: 1 week

When upgrading, install the ELF runtime linkers before libraries.

This is required to prevent problems with nss modules that use libthr when
upgrading from releases prior to 10.1.

PR: 197366
Sponsored by: DARPA, AFRL
Differential Revision: D1790
Reviewed by: cperciva
MFC after: 3 days

Use "RCS tag" instead of "$FreeBSD$ tag", since svn will obediently
expand the latter.

MFC after: 3 days

Strip trailing / characters from paths in "not present" index entries, not
just "directory" entries.

Prior to this commit, if / was added as part of a security update (how? In
the most recent case, because lib32 was accidentally omitted and was then
re-added, and every installer distribution set gets its own paths) then
the code which was supposed to filter out updates to deleted parts of the
base system (if someone decides to delete / then we shouldn't re-create it
for them) would instead get confused and decided that while / should exist,
// should not exist and needs to be removed.

This fixes the bug which caused freebsd-update to want to delete / (which is
harmless, since `rm /` fails, but scary nonetheless). A workaround is being
applied to the update bits in order to avoid triggering the bug on unpatched
systems.

PR: 196055, 196091, 196147

Re-work non-persistent filesystem detection as it was not possible to
detect /dev/md backed mfs filesystems that way.

Differential Revision: https://reviews.freebsd.org/D1163
Approved by: ian

Add logic for detecting non-persistent filesystems being utilized by
workdir which would break the upgrade process upon reboot.

Currently we check for tmpfs and mdmfs.

PR: 195006
Differential Revision: https://reviews.freebsd.org/D1163
Approved by: cperciva

Fix typo in r256646: We want to generate lists of directories in INDEX-OLD
and INDEX-NEW and compare them, not generate the same list of directories
from INDEX-OLD twice...

Pointy hats to: cperciva & everybody who didn't proofread EN-13:04 enough

Be more selective when filtering for lib*.so.N files. These are deleted
at the end of the upgrade process, after warning users to upgrade any
3rd party software (e.g., from the ports tree) which might link to the
libraries being removed.

Prior to this commit, the line
/usr/lib/libc.so|...|/lib/libc.so.7
matched the regex, which -- upgrading from 9.x to 10.x, where libc.so is
a regular file and thus was not part of a line which matched the regex --
resulted in freebsd-update thinking that /usr/lib/libc.so was a shared
library which was being removed as part of the upgrade. This had some
unfortunate consequences.

This will be part of an upcoming Errata Notice.

Allow ~ in file names so libtool droppings in contrib don't break updates.
It has happened twice now, and is likely to happen again.

Errata notice candidate.

When installing updates, install new directories first and remove old
directories last.

This is generally handled by the fact that the list of filesystem objects
is sorted, but this sorting is broken by code which moves .so files ahead
(so that they're present before any binaries which use them)... that code
also moved .so files ahead of directories, which is a problem for upgrading
to 10.0 where there's a new directory containing new .so files.

Errata Notice Candidate.

Speed up `freebsd-update IDS` by using IFS to split fields instead of
forking lots of processes to run echo|cut. In one test this reduced
the CPU time from 980s to 134s and the wallclock time from 806s to
132s.

Submitted by: Oleg Ginzburg

Add '%' and '@' to the set of characters which can appear in path names.
Without this change, freebsd-update refuses to accept 9.0 metadata files.

Make freebsd-update(8) smarter in how it handles $FreeBSD$ tags in
configuration files.

If the current file differs from the canonical version from the old release
only due to differences in the $FreeBSD$ tag (which can happen if the system
was installed from source code, depending on how the src tree was checked out)
then freebsd-update will treat the file as "unmodified" and silently update
it to the "clean" version in the new release.

If the only change being made to a configuration file is in the $FreeBSD$
tag (e.g., for any configuration files which have been modified locally, now
that we're using SVN and the $FreeBSD$ tag changes when a branch is created),
freebsd-update will no longer print the diff and prompt "Does this look
reasonable (y/n)?".

Nagged by: pgollucci
MFC after: 1 month

Honor $PAGER instead of always calling more.

Approved by: colin@
MFC after: 1 week

Typo fix in a comment.

Reviewed by: cperciva

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

In backup_kernel(), support backing up subdirectories and handle files
with spaces correctly.

Approved by: cperciva
MFC after: 1 month

"freebsd-update fetch -r NEWRELEASE" is meaningless and probably a typo
for "freebsd-update upgrade -r NEWRELEASE". Error out and suggest what
the user probably meant.

Submitted by: James Seward
MFC after: 1 month

Remind the user that he needs to run 'freebsd-update install' to install
new bits after downloading them using 'freebsd-update upgrade'.

Submitted by: bapt
MFC after: 1 month

Create the altix project branch. The altix project will add support
for the SGI Altix 350 to FreeBSD/ia64. The hardware used for porting
is a two-module system, consisting of a base compute module and a
CPU expansion module. SGI's NUMAFlex architecture can be an excellent
platform to test CPU affinity and NUMA-aware features in FreeBSD.

Disable SSL renegotiation in order to protect against a serious
protocol flaw. [09:15]

Correctly handle failures from unsetenv resulting from a corrupt
environment in rtld-elf. [09:16]

Fix permissions in freebsd-update in order to prevent leakage of
sensitive files. [09:17]

Approved by: so (cperciva)
Security: FreeBSD-SA-09:15.ssl
Security: FreeBSD-SA-09:16.rtld
Security: FreeBSD-SA-09:17.freebsd-udpate

Disable SSL renegotiation in order to protect against a serious
protocol flaw. [09:15]

Correctly handle failures from unsetenv resulting from a corrupt
environment in rtld-elf. [09:16]

Fix permissions in freebsd-update in order to prevent leakage of
sensitive files. [09:17]

Approved by: so (cperciva)
Security: FreeBSD-SA-09:15.ssl
Security: FreeBSD-SA-09:16.rtld
Security: FreeBSD-SA-09:17.freebsd-udpate

Special-case "-r X" where X is [0-9.]+ to mean "-r X-RELEASE".

Tripped over by: too many people to count
MFC after: 1 month

MFC 196392:

Add support for backing up the old kernel when installing a new kernel
using freebsd-update. This applies to using freebsd-update in "upgrade
mode" and normal freebsd-update on a security branch.

The backup kernel will be written to /boot/kernel.old, if the directory
does not exist, or the directory was created by freebsd-update in a
previous backup. Otherwise freebsd-update will generate a new directory
name for use by the backup. By default symbol files are not backed up
to save diskspace and avoid filling up the root partition.

This feature is fully configurable in the freebsd-update config file,
but defaults to enabled.

Reviewed by: cperciva

Approved by: re (kib)

Add support for backing up the old kernel when installing a new kernel
using freebsd-update. This applies to using freebsd-update in "upgrade
mode" and normal freebsd-update on a security branch.

The backup kernel will be written to /boot/kernel.old, if the directory
does not exist, or the directory was created by freebsd-update in a
previous backup. Otherwise freebsd-update will generate a new directory
name for use by the backup. By default symbol files are not backed up
to save diskspace and avoid filling up the root partition.

This feature is fully configurable in the freebsd-update config file,
but defaults to enabled.

MFC after: 1 week (stable/7)
Reviewed by: cperciva
Approved by: re (kib)

- Import the HEAD csup code which is the basis for the cvsmode work.

In `freebsd-update IDS`, strip out file flags before we look for
non-matching index lines. This fixes a bug where bogus warnings would
be printed file has the wrong file flags AND has been updated by
FreeBSD Update.

Reported by: Royce Williams

Add "IDS" command to freebsd-update. This was present in the original
version of freebsd-update, but I took it out when I rewrote everything
and added FreeBSD Update to the base system because I didn't think it
was useful. It turns out that quite a few people liked it and wanted
it back.

Requested by: Royce Williams + others
MFC after: 2 weeks

Adjust recognize-shared-libraries regex to avoid matching symlinks to
shared libraries.

This fixes a problem which resulted in 6.x->7.x upgrades having the
/usr/lib/libpthread.so -> libthr.so symlink missing; what happened was
that the old libpthread.so symlink pointed to /lib/libpthread.so.2 --
which matched the "/lib/*\.so\.[0-9]+" regex -- but the new symlink
didn't, so FreeBSD Update got confused and deleted the symlink as part
of its "remove old shared libraries" step.

To recreate the symlink (which I understand is necessary for ports like
KDE to build) on a 7.x system which FreeBSD Update upgraded from 6.x:
# ln -s libthr.so /usr/lib/libpthread.so

Reported by: Dmitry RCL Rekman
Help diagnosing bug from: kris
MFC after: 7 days

When updating the install list for files which have had local changes
merged with upgrade changes, don't try to compute the SHA256 hash of
files which don't exist.

Reported by: Jaakko Heinonen
MFC after: 1 week

Add change missing from previous commit: Remove temporary file.

Fix "freebsd-update rollback" applied to minor-version upgrades: Old
shared object files which have the same name as currently-installed
shared object files should be reinstalled after binaries are rolled
back. The order for rolling back updates is therefore
1. Install any old shared object files which can be installed without
overwriting a new shared object file.
2. Rollback everything which isn't a shared object or kernel file.
3. Rollback any shared object files which we didn't deal with in (1).
4. Rollback to the old kernel.

Bug reported by: Jan Henrik Sylvester
MFC after: 3 days

Add support for "freebsd-update -r newrelease upgrade" -- binary
upgrading to new releases. Important parts of this code include
* automatically determining which optional components (e.g., src,
info, proflibs) are installed.
* merging changes in files which are modified locally and have
changed between the currently running and new release.
* prompting the user to rebuild all 3rd party software before
deleting old shared libraries.

Yes, this is compatible with "freebsd-update rollback" -- you can
test a new -BETA and roll back to the old release if you don't
like it.

Subject to re@ approval, this will be MFCed before 7.0-BETA3 and
6.3-RC1.

MFC after: 2 days

Two minor improvements uncovered by work on upgrading between releases:
* When installing updates, make sure that securelevel <= 0. Otherwise
we can't remove the schg flag from files.
* When preparing to download updates, check to see if we already have
them sitting in the /files/ directory. This saves bandwidth if users
run "freebsd-update fetch" more than once without installing updates
in between.

While I'm here, bump the copyright date.

MFC after: 3 days

Change the time of the first "EoL is coming soon, you should upgrade" warning
from EoL minus 6 months to EoL minus 3 months, in order to increase the odds
of there actually being a more recent release to which users can upgrade.
(In particular, for releases which are only supported for 12 months, it's
quite likely that the next release will occur between 6 and 9 months later.)

Discussed with: kensmith
Approved by: re (bmah)
MFC after: 3 days

When storing old versions of files for use in generating new files via
patching and for rolling back updates, don't copy a file if it has already
been stored. This provides a significant speedup to the "Preparing to
download files" stage of "freebsd-update fetch" if many updates have already
been applied or if a file being updated is linked many times (such as
/rescue/*).

Reported by: Paul Dekkers
MFC after: 1 week
Approved by: re (bmah)

Add some missing '${BASEDIR}/'s. Prior to this commit, FreeBSD Update
operating with the "-b basedir" option would not correctly update files
which had flags set or were hardlinked.

Submitted by: Karsten Schmidt
Pointy hat to: cperciva
MFC after: 1 week

Fix problems resulting from SMP kernels (mis-)identifying themselves as
"SMP-GENERIC" (i386) or "GENERIC" (amd64).

FreeBSD 6.2 Errata candidate.

MFC after: 3 days
Pointy hat to: cperciva

Three minor bug fixes:
1. When downloading metadata files, make sure we only download each
file once; without this fix, "freebsd-update fetch" will fail the first
time it is run if there have been no updates yet for the installed
release.
2. If the FOO kernel is installed in /boot/kernel instead of /boot/FOO
and the /boot/FOO directory does not exist, don't try to update
/boot/FOO. This is an issue only where an update involves adding a new
kernel module.
3. When removing files and directories, operate in reverse
lexographical order, in order to ensure that files are removed before
the directory which contains them.

MFC after: 3 days

Set LC_ALL=C in order to avoid problems with character ranges and
sorting.

PR: bin/104505
MFC after: 3 days

Conform to wider English usage.

Submitted by: Royce Williams

Add FreeBSD Update 2.0 client code. The build code is in the projects
repository.

Sponsored by: FreeBSD security development fundraiser