MFC r326082: freebsd-update: do not duplicate patchlist entries

PR: 221079
Submitted by: Masachika ISHIZUKA
Submitted by: ota@j.email.ne.jp
Reviewed by: cperciva

MFC r319954:
Modernize FreeBSD version numbers in freebsd-update(8).
While here, expand a contraction to make textproc/igor happy.

Sponsored by: The FreeBSD Foundation

MFC r302534:

Allow - in distribution names. This is needed for freebsd-update to work
with 11.0+, where the debugging symbols use a new naming scheme for release
distribution files.

Errata candidate.

MFC r289065: Add --currently-running <release> option to freebsd-update.

MFC r279901 (kmoore):

- Include commas in valid file-name chars freebsd-update will support

MFC r256594 (cperciva):

Speed up `freebsd-update IDS` by using IFS to split fields instead of
forking lots of processes to run echo|cut. In one test this reduced
the CPU time from 980s to 134s and the wallclock time from 806s to
132s.

Submitted by: Oleg Ginzburg

MFC r274630,r274653 (feld):

Add logic for detecting non-persistent filesystems being utilized by
workdir which would break the upgrade process upon reboot.

Currently we check for tmpfs and md.

MFC r276095 (des)

Use "RCS tag" instead of "$FreeBSD$ tag", since svn will obediently
expand the latter.

MFC r284425:

Skip src component if /usr/src is empty.

Submitted by: kczekirda
Reviewed by: cperciva, delphij, nwhitehorn, allanjude

MFC r279571,281563 (allanjude):

Add a new safetly belt to freebsd-update to prevent a user doing a minor
update (-pX) while having an unfinished major upgrade (9.x to 9.y)
Safetly belt can be disabled with the -F flag

Additionally, add the --not-running-from-cron flag they bypasses the TTY
requirement, and allows freebsd-update to be invoked by orchestration
frameworks, scripts, or otherwise.

PR: 196760
Relnotes: yes
Sponsored by: ScaleEngine Inc.

MFC rev. 278728:
(backup_kernel_finddir, backup_kernel, install_files):
Add missing references to $BASEDIR, in order to allow correct operation
when updating a system mounted at another location than / (e.g. when
updating an alternate Boot Environment).

Reviewed by: cperciva

When upgrading, install the ELF runtime linkers before libraries.

This is required to prevent problems with nss modules that use libthr when
upgrading from releases prior to 10.1.

PR: 197366
Differential Revision: D1790
Reviewed by: cperciva
Sponsored by: DARPA, AFRL

Strip trailing / characters from paths in "not present" index entries.

Errata: FreeBSD-EN-14:13.freebsd-update
Approved by: so@

Hoist all the mergeinfo up to the root in preparation for enforcing merges
to the root only. All MFC's were rerecorded to the root.

Going forward, if an MFC includes mergeinfo, it will need to be made to
the root and committed from the root. Merges with --ignore-ancestry
or diff | patch can go anywhere.

The mergeinfo in HEAD is in a bad state from years of neglect and manual
tampering and this was branched into 10.x. This confuses the coalescing
code and prevents it from doing its job.

Approved by: re (gjb, implicit)

MFC r257879:

Fix typo in r256646: We want to generate lists of directories in
INDEX-OLD and INDEX-NEW and compare them, not generate the same
list of directories from INDEX-OLD twice...

Pointy hats to: cperciva & everybody who didn't proofread
EN-13:04 enough
Errata Notice: FreeBSD-EN-13:05.freebsd-update
Approved by: re (gjb)

MFC r256646, r256767, r257038:
When installing updates, install new directories first and remove old
directories last.

Allow ~ in file names so libtool droppings in contrib don't break updates.
It has happened twice now, and is likely to happen again.

Be more selective when filtering for lib*.so.N files. These are deleted
at the end of the upgrade process, after warning users to upgrade any
3rd party software (e.g., from the ports tree) which might link to the
libraries being removed.

Approved by: re (gjb)
Errata Notice: FreeBSD-EN-13:04.freebsd-update

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

mdoc sweep.

Fixes to man8 groff mandoc style, usage mistakes, or typos.

PR: 168016
Submitted by: Nobuyuki Koganemaru
Approved by: gjb
MFC after: 3 days

Remove trailing whitespace per mdoc lint warning

Disussed with: gavin
No objection from: doc
Approved by: joel
MFC after: 3 days

Add '%' and '@' to the set of characters which can appear in path names.
Without this change, freebsd-update refuses to accept 9.0 metadata files.

Make freebsd-update(8) smarter in how it handles $FreeBSD$ tags in
configuration files.

If the current file differs from the canonical version from the old release
only due to differences in the $FreeBSD$ tag (which can happen if the system
was installed from source code, depending on how the src tree was checked out)
then freebsd-update will treat the file as "unmodified" and silently update
it to the "clean" version in the new release.

If the only change being made to a configuration file is in the $FreeBSD$
tag (e.g., for any configuration files which have been modified locally, now
that we're using SVN and the $FreeBSD$ tag changes when a branch is created),
freebsd-update will no longer print the diff and prompt "Does this look
reasonable (y/n)?".

Nagged by: pgollucci
MFC after: 1 month

Honor $PAGER instead of always calling more.

Approved by: colin@
MFC after: 1 week

Typo fix in a comment.

Reviewed by: cperciva

mdoc: drop redundant .Pp and .LP calls

They have no effect when coming in pairs, or before .Bl/.Bd

In backup_kernel(), support backing up subdirectories and handle files
with spaces correctly.

Approved by: cperciva
MFC after: 1 month

"freebsd-update fetch -r NEWRELEASE" is meaningless and probably a typo
for "freebsd-update upgrade -r NEWRELEASE". Error out and suggest what
the user probably meant.

Submitted by: James Seward
MFC after: 1 month

Add warning about freebsd-update upgrade needing lots of disk space.

Submitted by: jpaetzel
MFC after: 1 month

Remind the user that he needs to run 'freebsd-update install' to install
new bits after downloading them using 'freebsd-update upgrade'.

Submitted by: bapt
MFC after: 1 month

Fix typos, spelling, formatting and mdoc mistakes found by Nobuyuki while
translating these manual pages. Minor corrections by me.

Submitted by: Nobuyuki Koganemaru <n-kogane@syd.odn.ne.jp>

Update some release version numbers. The patch is based on the one
from the PR, but the version numbers reflect the newer ones from
http://security.freebsd.org/#sup

PR: docs/145227
Submitted by: Glen Barber (glen dot j dot barber at gmail dot com)
Reviewed by: cperciva
Mentored by: jkois
MFC after: 1 week

Fix several typos in macros or macro misusage.

Found by: make manlint
Reviewed by: ru
Approved by: philip (mentor)

Disable SSL renegotiation in order to protect against a serious
protocol flaw. [09:15]

Correctly handle failures from unsetenv resulting from a corrupt
environment in rtld-elf. [09:16]

Fix permissions in freebsd-update in order to prevent leakage of
sensitive files. [09:17]

Approved by: so (cperciva)
Security: FreeBSD-SA-09:15.ssl
Security: FreeBSD-SA-09:16.rtld
Security: FreeBSD-SA-09:17.freebsd-udpate

Special-case "-r X" where X is [0-9.]+ to mean "-r X-RELEASE".

Tripped over by: too many people to count
MFC after: 1 month

Add support for backing up the old kernel when installing a new kernel
using freebsd-update. This applies to using freebsd-update in "upgrade
mode" and normal freebsd-update on a security branch.

The backup kernel will be written to /boot/kernel.old, if the directory
does not exist, or the directory was created by freebsd-update in a
previous backup. Otherwise freebsd-update will generate a new directory
name for use by the backup. By default symbol files are not backed up
to save diskspace and avoid filling up the root partition.

This feature is fully configurable in the freebsd-update config file,
but defaults to enabled.

MFC after: 1 week (stable/7)
Reviewed by: cperciva
Approved by: re (kib)

fix grammar...

Reviewed by: cperciva

In `freebsd-update IDS`, strip out file flags before we look for
non-matching index lines. This fixes a bug where bogus warnings would
be printed file has the wrong file flags AND has been updated by
FreeBSD Update.

Reported by: Royce Williams

Add "IDS" command to freebsd-update. This was present in the original
version of freebsd-update, but I took it out when I rewrote everything
and added FreeBSD Update to the base system because I didn't think it
was useful. It turns out that quite a few people liked it and wanted
it back.

Requested by: Royce Williams + others
MFC after: 2 weeks

Adjust recognize-shared-libraries regex to avoid matching symlinks to
shared libraries.

This fixes a problem which resulted in 6.x->7.x upgrades having the
/usr/lib/libpthread.so -> libthr.so symlink missing; what happened was
that the old libpthread.so symlink pointed to /lib/libpthread.so.2 --
which matched the "/lib/*\.so\.[0-9]+" regex -- but the new symlink
didn't, so FreeBSD Update got confused and deleted the symlink as part
of its "remove old shared libraries" step.

To recreate the symlink (which I understand is necessary for ports like
KDE to build) on a 7.x system which FreeBSD Update upgraded from 6.x:
# ln -s libthr.so /usr/lib/libpthread.so

Reported by: Dmitry RCL Rekman
Help diagnosing bug from: kris
MFC after: 7 days

When updating the install list for files which have had local changes
merged with upgrade changes, don't try to compute the SHA256 hash of
files which don't exist.

Reported by: Jaakko Heinonen
MFC after: 1 week

Add change missing from previous commit: Remove temporary file.

Fix "freebsd-update rollback" applied to minor-version upgrades: Old
shared object files which have the same name as currently-installed
shared object files should be reinstalled after binaries are rolled
back. The order for rolling back updates is therefore
1. Install any old shared object files which can be installed without
overwriting a new shared object file.
2. Rollback everything which isn't a shared object or kernel file.
3. Rollback any shared object files which we didn't deal with in (1).
4. Rollback to the old kernel.

Bug reported by: Jan Henrik Sylvester
MFC after: 3 days

Add support for "freebsd-update -r newrelease upgrade" -- binary
upgrading to new releases. Important parts of this code include
* automatically determining which optional components (e.g., src,
info, proflibs) are installed.
* merging changes in files which are modified locally and have
changed between the currently running and new release.
* prompting the user to rebuild all 3rd party software before
deleting old shared libraries.

Yes, this is compatible with "freebsd-update rollback" -- you can
test a new -BETA and roll back to the old release if you don't
like it.

Subject to re@ approval, this will be MFCed before 7.0-BETA3 and
6.3-RC1.

MFC after: 2 days

Two minor improvements uncovered by work on upgrading between releases:
* When installing updates, make sure that securelevel <= 0. Otherwise
we can't remove the schg flag from files.
* When preparing to download updates, check to see if we already have
them sitting in the /files/ directory. This saves bandwidth if users
run "freebsd-update fetch" more than once without installing updates
in between.

While I'm here, bump the copyright date.

MFC after: 3 days

Change the time of the first "EoL is coming soon, you should upgrade" warning
from EoL minus 6 months to EoL minus 3 months, in order to increase the odds
of there actually being a more recent release to which users can upgrade.
(In particular, for releases which are only supported for 12 months, it's
quite likely that the next release will occur between 6 and 9 months later.)

Discussed with: kensmith
Approved by: re (bmah)
MFC after: 3 days

When storing old versions of files for use in generating new files via
patching and for rolling back updates, don't copy a file if it has already
been stored. This provides a significant speedup to the "Preparing to
download files" stage of "freebsd-update fetch" if many updates have already
been applied or if a file being updated is linked many times (such as
/rescue/*).

Reported by: Paul Dekkers
MFC after: 1 week
Approved by: re (bmah)

Add some missing '${BASEDIR}/'s. Prior to this commit, FreeBSD Update
operating with the "-b basedir" option would not correctly update files
which had flags set or were hardlinked.

Submitted by: Karsten Schmidt
Pointy hat to: cperciva
MFC after: 1 week

Fix problems resulting from SMP kernels (mis-)identifying themselves as
"SMP-GENERIC" (i386) or "GENERIC" (amd64).

FreeBSD 6.2 Errata candidate.

MFC after: 3 days
Pointy hat to: cperciva

"block is set to UTC" -> "clock is set to UTC"

Submitted by: csjp

Add a sentence to the DESCRIPTION noting that updates aren't available
for -STABLE or -CURRENT.

Inspired by submission from: Scott Robbins
MFC after: 3 days

Three minor bug fixes:
1. When downloading metadata files, make sure we only download each
file once; without this fix, "freebsd-update fetch" will fail the first
time it is run if there have been no updates yet for the installed
release.
2. If the FOO kernel is installed in /boot/kernel instead of /boot/FOO
and the /boot/FOO directory does not exist, don't try to update
/boot/FOO. This is an issue only where an update involves adding a new
kernel module.
3. When removing files and directories, operate in reverse
lexographical order, in order to ensure that files are removed before
the directory which contains them.

MFC after: 3 days

Set LC_ALL=C in order to avoid problems with character ranges and
sorting.

PR: bin/104505
MFC after: 3 days

Conform to wider English usage.

Submitted by: Royce Williams

It's spelled MAN.

Add FreeBSD Update 2.0 client code. The build code is in the projects
repository.

Sponsored by: FreeBSD security development fundraiser