Reapply "release.sh: Add -jN to `make release`"

With the latest changes to release/Makefile, it is now possible to
run `make release -jN` without the build failing (at least in my
latest tests).

This reverts commit 7b707e797b2cd6265ba8f6215e59445e9efb9e97.

MFC after: 1 week

release.sh: Don't install git if already present

Prior to this commit, we install git from ports if there is a ports
tree available and git is not installed, and we install git from pkg
otherwise -- including the case where git is already installed.

Rework the logic to not (re)install git at all if it is already
installed.

MFC after: 3 days

Revert "release.sh: Add -jN to `make release`"

The release-building code is not fully parallel-safe yet.

This reverts commit 9c1bad2da5ae2a2a3d957c97dcf33c09b34e1c47.

Reported by: jrtc27

release.sh: Add -jN to `make release`

The `make release` command now creates VM and cloudware images (if
enabled) in addition to disk images; this results in a very large
number of 'make installworld' commands running sequentially. Adding
-jN should speed this up significantly.

MFC after: 1 month
X-MFC-to: stable/14

release: fix ports checkout if /usr/ports does not exist

Reported by: Yasuhiro Kimura
Sponsored by: Rubicon Communications, LLC ("Netgate")

release: remove logic to locate the svn{,lite} binary

Sponsored by: Rubicon Communications, LLC ("Netgate")

release: follow-up to previous commit to use Git for the ports tree

MFC after: immediately (pending re approval)
Sponsored by: Rubicon Communications, LLC ("Netgate")

release: update release.sh to use Git for ports tree checkouts

MFC after: immediately (pending re approval)
Sponsored by: Rubicon Communications, LLC ("Netgate")

release: do not set __MAKE_CONF and SRCCONF for the chroot build

PR: 254319
Submitted by: truckman
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")

release: permanently remove the 'reldoc' target and associates

Following 7b1d1a1658ffb69eff93afc713f9e88ed8b20eac, the structure
for the reldoc target has significantly changed as result of the
ASCIIDoctor/Hugo migration. As the release notes related files
on the installation medium are inherently out of date, purge them
entirely.

Discussed within: re, doceng
No objection: re (silence), doceng (silence)
Timeout: 2 weeks
MFC after: 1 week
MFC to: stable/13, stable/12, and stable/11 only
Sponsored by: Rubicon Communications, LLC ("Netgate")

release.sh: fix OSVERSION and UNAME_r overrides

Add PBUILD_FLAGS and UNAME_r overrides to extra_chroot_setup()
to fix building ports for 14-CURRENT builds on 13.x.

MFC after: 3 days
MFC with: c511a5ab53ba335dc85f7ac93843872eb5763602
Sponsored by: Rubicon Communications, LLC ("Netgate")

release.sh: Skip installing textproc/docproj when NODOC is set

PR: 253192
Submitted by: Yasuhiro Kimura
MFC after: 3 days
Sponsored by: Rubicon Communications, LLC ("Netgate")

release: update workaround during transition to git

PR: 253181
Submitted by: Yasuhiro Kimura
MFC after: 3 days
Sponsored by: Rubicon Communications, LLC ("Netgate")

release: fix PORT variable

Reported by: Yasuhiro Kimura (yasu xat utahime dot org)
Sponsored by: Rubicon Communications, LLC ("Netgate")

release.sh: bump Copyright year after previous commit

Sponsored by: Rubicon Communications, LLC (netgate.com)

release: Add workaround to use SVN for ports

The ports tree is scheduled to be converted from Subversion to Git
after the currently-scheduled 13.0-RELEASE, so the source of truth
will be Subversion for the ports tree.

Implement a hack specifically for this case.

Sponsored by: Rubicon Communications, LLC (netgate.com)

release.sh: Update GITROOT URL

Hard-code the GITROOT for the ports tree to use cgit-beta
until the ports repository is converted.

While here, remove $FreeBSD$ RCS IDs.

Sponsored by: Rubicon Communications, LLC (netgate.com)

release.sh: add support for RISC-V embedded builds

Since the few existing RISC-V hardware platforms are single board
computers, we can piggyback off of arm/arm64's embedded build support
for generating SD card images.

I don't see a pressing need to change the naming in this file at this
time.

Reviewed by: gjb, manu
Differential Revision: https://reviews.freebsd.org/D27043

Fix setting RELENGDIR when release.sh is called from an external
script.

Sponsored by: Rubicon Communications, LLC (netgate.com)

Fix indentation following r365638. No functional changes.

Sponsored by: Rubicon Communications, LLC (netgate.com)

Add a NOGIT option to avoid explicitly requiring devel/git if the
sources are obtained through a different mechanism.

Reported and tested by: Ruslan Garipov
Sponsored by: Rubicon Communications, LLC (netgate.com)

Fix ordering of the 'pull' subcommand and the '-q' flag.

Pointyhat to: gjb (myself)
Sponsored by: Rubicon Communications, LLC (netgate.com)

Restore the '-q' flag to the 'git pull' command.

Sponsored by: Rubicon Communications, LLC (netgate.com)

Remove the VCSUPDATE command, because git is too stupid to have
the '-C <directory>' after the subcommand.

Meanwhile, hard-code 'git -C <...> pull' for now.

Reported by: Michael Butler
Sponsored by: Rubicon Communications, LLC (netgate.com)

Refine the VCSUPDATE logic further:
- Look for the .git directory instead of top-level directory.
- Use 'git -C' instead of cd(1).

Sponsored by: Rubicon Communications, LLC (netgate.com)

Add a VCSUPDATE command to run 'git pull' instead of 'git clone'
if the tree already exists.

Reported by: Michael Butler
Sponsored by: Rubicon Communications, LLC (netgate.com)

Install devel/git from packages if NOPORTS is set in the release.sh
configuration file.

Reported by: Michael Butler
Sponsored by: Rubicon Communications, LLC (netgate.com)

Correct the conditional evaluating if git exists.

Pointyhat to: gjb (myself)
Sponsored by: Rubicon Communications, LLC (netgate.com)

Do not unconditionally build git from ports if it already exists.

Sponsored by: Rubicon Communications, LLC (netgate.com)

Fix passing OPTIONS_UNSET with multiple options to make(1).
While here, adjust the copyright.

Sponsored by: Rubicon Communications, LLC (netgate.com)

Remove an escape that does not belong.

Sponsored by: Rubicon Communications, LLC (netgate.com)

release: update to use git instead of svn:

Makefile.* (cloudware):
- Consistify setting the BUILDDATE for snapshots.

release.conf.sample/release.sh:
- Run 'git clone' in 'quiet' mode.

Makefile.inc1:
- Set BUILDDATE and export the variable.

Sponsored by: Rubicon Communications, LLC (netgate.com)

release: update to use git instead of svn:

release.sh:
- Bump the version number.
- Default VCSCMD to '/usr/local/bin/git clone'.
- Rename SVN* variables to GIT* equivalents.
- Remove dead code to inject a trailing '/' between two variables.
- Remove SRC_FORCE_CHECKOUT.
- Exit if the build host does not have devel/git installed.
- Install devel/git in the build chroot(8) for newvers.sh.

release.conf.sample:
- Update sample configuration variables to the git equivalent.

relnotes-search.sh:
- Remove script. It is specifically for use with svn.

Sponsored by: Rubicon Communications, LLC (netgate.com)

release: amd64 efi boot name is bootx64

efi_boot_name is just used for arm image so no harm done.

Reported by: gonzo
MFC after: 3 days

Fix passing ${CONF_FILES} (which contains MAKE_CONF and
SRC_CONF, __MAKE_CONF and SRCCONF, respectively) through
to arm_install_base() and chroot_arm_build_release().
This prevents failures when the target image is intended
to be build with make.conf(5) and src.conf(5) overrides,
which are correctly handled for non-embedded image builds.

Reported and tested by: Daniel Engberg
PR: 238615
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Set OPTIONS_UNSET in the argument list to env(1), and add
AVAHI to the list. This fixes the textproc/docproj build
seemingly following FLAVORS being added.

Specifically, the problem with the dependency chain here is:
- textproc/docproj depends on print/cups, which sets AVAHI=on
by default;
- net/avahi-app depends on devel/gobject-introspection, which
requires python3+;
- graphics/netpbm depends on graphics/mesa-libs, which can
only be built with python2.7;
- textproc/docproj depends on a number of graphics ports for
font rendering, etc.

MFC after: 3 days
MFC before: 12.0-BETA3
Sponsored by: The FreeBSD Foundation

release: Deinstall u-boot ports before installing

FORCE_PKG_REGISTER is broken so multiple invocation of release.sh for the
same board will fails if /scratch isn't cleaned.
Leave it but deinstall the package first.

Reviewed by: gjb
Differential Revision: https://reviews.freebsd.org/D16513

release: arm: Use BOARDNAME in the image name

Using KERNEL made sense when all boards had different kernel configuration.
Now that all of them are using GENERIC use the board name instead

Reviewed by: gjb
Differential Revision: https://reviews.freebsd.org/D16512

release: Add arm_install_boot to install the commit boot bits

This reduce the per-board arm_install_uboot to just install u-boot.
While here remove the installation of rpi.dtb and rpi2.dtb as we load
them from the UFS partition via ubldr.

Reviewed by: gjb, imp (older version)
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D16239

Bump Copyright year following r328283.

MFC after: 3 days
MFC with: r328283
Sponsored by: The FreeBSD Foundation

When CHROOTBUILD_SKIP is set, evaluate the existence of /bin/sh
within the CHROOTDIR. If it does not exist, unset CHROOTBUILD_SKIP
to prevent build failures.

Requested by: swills
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Fix port build flags passed to make(1) after r326315, where
it was missed for embedded image builds.

MFC after: 3 days
MFC with: r326315
Sponsored by: The FreeBSD Foundation

Set DISTDIR and WRKDIRPREFIX when building ports within the
chroot(8) to avoid mtime changes within the ports checkout,
which can cause checksum differences.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Only copy /etc/resolv.conf to ${CHROOTDIR} if /etc/resolv.conf does
not already exist within ${CHROOTDIR}. This allows re-using a build
chroot with CHROOTBUILD_SKIP set to a non-empty value and CHROOTDIR
set to '/' in release.conf.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Update the GUMSTIX image build to use arm/arm TARGET/TARGET_ARCH.
Update the TARGET/TARGET_ARCH matching in release/release.sh and
release/Makefile.mirrors for simplification.

Discussed with: mmel
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Specify WITH_UNIFIED_OBJDIR in chroot_arm_build_release() to
ensure the output image is created in the correct .OBJDIR.

Sponsored by: The FreeBSD Foundation

Include TARGET and TARGET_ARCH in chroot_arm_build_release()
make(1) invocations following the OBJDIR restructuring to
ensure the output arm SoC image is in the correct directory.

Sponsored by: The FreeBSD Foundation

Add arm/armv7 checks to env_check().

Sponsored by: The FreeBSD Foundation

Always expand the full path to the configuration file specified
with the '-c' flag.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Include WORLD_FLAGS in CHROOT_IMAKEFLAGS and CHROOT_DMAKEFLAGS, to
allow passing '-jN' to the installworld and distributeworld targets.

Submitted by: bdrewery
Sponsored by: The FreeBSD Foundation

In release/release.sh:
- Rename chroot_arm_armv6_build_release() to chroot_arm_build_release()
and make it hardware agnostic (such as armv6 -vs- armv7 -vs- arm64).

- Evaluate EMBEDDED_TARGET differently so release/tools/arm.subr can
be used for arm/armv6 and arm64/aarch64.

- Update comments and copyright.

In release/tools/arm.subr:
- In arm_create_disk(), change the default alignment from 63 to 512k,
fixing a boot issue on arm64 and EFI. [1]

- Update comments and copyright.

Add a RPI3 configuration file, pieces obtained from Crochet.

Obtained from: Crochet [1]
MFC after: 5 days
X-MFC-Note: maybe
Sponsored by: The FreeBSD Foundation

Pass overrides to make(1) when building ports for arm/armv6
targets, similar to what is done for the run-autotools-fixup
override for non-arm targets.

MFC after: 3 days
Tested on: 12-CURRENT building 10-STABLE
Sponsored by: The FreeBSD Foundation

One of the already implemented options in release/Makefile is NOSRC. When
it's defined, installation image is shipped without source distribution
(src.txz)

Add the hability of defining NOSRC in release.conf and pass it to
'make release' argument

Approved by: gjb
Sponsored by: Rubicon Communications (Netgate)
Differential Revision: https://reviews.freebsd.org/D6710

Set FORCE_PKG_REGISTER=1 when installing packages to avoid failures
when re-using build chroot(8) environments.

This is based on the patch in the PR referenced below, but instead
of using 'reinstall' in two locations (one of which already uses
FORCE_PKG_REGISTER=1), changes the non-embedded behavior.

PR: 205998
Submitted by: ngie
MFC after: 5 days
Sponsored by: The FreeBSD Foundation

Deprecate MD5 checksum generation in favor of SHA512.

This was discussed during the 10.2-RELEASE cycle, however
since we were nearing the end of the cycle, we decided to
defer this change until after 10.2-RELEASE.

Reminded by: so (delphij), jmg
MFC after: 5 days
Sponsored by: The FreeBSD Foundation

Append VMFORMATS and VMSIZE to RELEASE_RMAKEFLAGS only if
WITH_VMIMAGES is set.

MFC after: 3 days
X-MFC-With: r288341, r288345
Sponsored by: The FreeBSD Foundation

In followup to r288341, ensure VMFORMATS and VMSIZE are not
set to empty values, which would result in nonintuitive build
errors.

MFC after: 3 days
X-MFC-With: r288341
PR: 203420 (related to)
Sponsored by: The FreeBSD Foundation

Honor VMFORMATS and VMSIZE if set in release.conf.

PR: 203420
Submitted by: luca <dot> pizzamiglio <at> gmail <dot> com
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Sigh. Really fix CHECKSUM file creation.

Sponsored by: The FreeBSD Foundation

Fix CHECKSUM file creation.

Sponsored by: The FreeBSD Foundation

Do not include '/R' in the path when generating checksums, since
that will cause '/R' to be included in the CHECKSUM files.

Sponsored by: The FreeBSD Foundation

Move the finished image build to the '/R' directory (the
standard 'install' location for other architectures), then
compress the image with xz(1), and generate the CHECKSUM
files.

Sponsored by: The FreeBSD Foundation

Fix the name of the final image.
Sigh. Thou shalt not commit before coffee.

Sponsored by: The FreeBSD Foundation

Rename KERNEL.img -> OSRELEASE-KERNEL.img when complete.

Sponsored by: The FreeBSD Foundation

Fix paths.
Remove the temporary directory when done with it.
Destroy the md(4) device when done with it.

Sponsored by: The FreeBSD Foundation

In release.sh, create the /dev directory within the chroot
before attempting to mount(8) devfs. Also, create the
.OBJDIR for the 'release' target, so files end up in the
correct location.

In tools/arm.subr, fix the target device when creating the
gpart partition scheme.

Sponsored by: The FreeBSD Foundation

Use the correct variable for the tools/TARGET.conf file.

Sponsored by: The FreeBSD Foundation

Add tools/arm.subr to contain common subroutines used for
building arm images. This is similar to tools/vmimage.subr
used for building virtual machine disk images. By default,
only arm_create_disk() and arm_install_base() contain real
functionality here, and arm_install_uboot() must be overridden
in the arm/KERNEL.conf file.

In release.sh, make create_arm_armv6_build_release() do
something now.

In arm/BEAGLEBONE.conf, set IMAGE_SIZE, PART_SCHEME, FAT_SIZE,
FAT_TYPE, and MD_ARGS, as well as make arm_install_uboot()
functional.

Parts of this were taken from disecting a previous BEAGLEBONE
image, and other parts obtained from Crochet sources.

Sponsored by: The FreeBSD Foundation

Evaluage chroot_build_release_cmd based on the TARGET/TARGET_ARCH
of the build target.

Move arm/armv6 'release' bits to chroot_arm_armv6_build_release().

Sponsored by: The FreeBSD Foundation

buildenv_setup(), if overridden in the configuration file,
only needs to be invoked once.

Sponsored by: The FreeBSD Foundation

Remove Crochet invocation.

Sponsored by: The FreeBSD Foundation

Update the arm/*.conf configuration files to only set
EMBEDDEDBUILD, EMBEDDED_TARGET, EMBEDDED_TARGET_ARCH,
EMBEDDEDPORTS, and KERNEL.

In release.sh, set TARGET and TARGET_ARCH to the
EMBEDDED_* variants from the configuration file.

Sponsored by: The FreeBSD Foundation

In extra_chroot_setup(), use 'uname -U' to determine OSVERSION.

Move the port build for EMBEDDEDPORTS (if set) from arm/release.sh
to release.sh.

Remove arm/release.sh.

Sponsored by: The FreeBSD Foundation

Add a return to chroot_build_target().
In chroot_build_release(), reverse the order evaluating
EMBEDDEDBUILD.

Sponsored by: The FreeBSD Foundation

Wrap long lines in comments after previous commits that indented
the file.

Sponsored by: The FreeBSD Foundation

Ensure resolv.conf exists on the build host before attempting
to copy it to the build chroot.

Move where the ldconfig(8) startup script is invoked.

Sponsored by: The FreeBSD Foundation

Move where the configuration file is sourced if '-c'
is specified. If the configuration file does not exist,
proceed with the default options instead of exiting.

Sponsored by: The FreeBSD Foundation

Move the trap after env_check(), since CHROOTDIR may be
redefined.

Sponsored by: The FreeBSD Foundation

Shorten long lines.

Sponsored by: The FreeBSD Foundation

Move PATH and export to the same line.
Handle svn or svnlite being installed in /usr/bin or
/usr/local/bin.

Sponsored by: The FreeBSD Foundation

Add documentation for newly-added functions.
While here, move CHROOT_* and RELEASE_* variables from
env_setup() to env_check() since they may change if
a release.conf file is used.

Sponsored by: The FreeBSD Foundation

Add the following functions to release.sh to organize and make
clear the workflow:
- env_setup()
- env_check()
- chroot_setup()
- extra_chroot_setup()
- chroot_build_target()
- chroot_build_release()

There should be no functional changes at this point.

Sponsored by: The FreeBSD Foundation

Add initial support for building RPI2 images.

In release.sh, allow overriding buildenv_setup() before
the handoff to arm/release.sh.

Copy arm/RPI-B.conf -> arm/RPI2.conf, set UBOOT_PORT and
the correct KERNEL, and add the buildenv_setup() override
to install the sysutils/u-boot-rpi2 port/package.

Copy tools/arm/crochet-RPI-B.conf -> tools/arm/crochet-RPI2.conf,
and set the correct entries for the RaspberryPi2 board.

Thanks to: loos@
Sponsored by: The FreeBSD Foundation

Default XZ_THREADS=0, and add to RELEASE_RMAKEFLAGS
so it is passed to the 'release' make(1) target if
it needs to be adjusted to a non-zero integer.

Sponsored by: The FreeBSD Foundation

Bump copyright after r277458.

MFC after: 2 weeks
X-MFC-with: r277458, r277536, r277606, r277609,
r277836, r278118, r278119
Sponsored by: The FreeBSD Foundation

Set a VERSION value.

Sponsored by: The FreeBSD Foundation

Add examples for WITH_CLOUDWARE to release.conf.sample.
Add WITH_CLOUDWARE evaluation to RELEASE_RMAKEFLAGS.

Sponsored by: The FreeBSD Foundation

Add WITH_COMPRESSED_VMIMAGES variable, which when set enables
xz(1) compression of the virtual machine images.

This is intentionally separate to allow more fine-grained
tuning over which images are compressed, especially in cases
where compressing 20GB sparse images can take hours.

Sponsored by: The FreeBSD Foundation

release/Makefile:
Connect the virtual machine image build to the release
target if WITH_VMIMAGES is set to a non-empty value.

release/release.sh:
Add WITH_VMIMAGES to RELEASE_RMAKEFLAGS.

release/release.conf.sample:
Add commented entries for tuning the release build if the
WITH_VMIMAGES make(1) environment variable is set to
a non-empty value.

Sponsored by: The FreeBSD Foundation

Wrap a long line.

MFC after: 3 days
X-MFC-with: r270417, r270418, r270455
Sponsored by: The FreeBSD Foundation

Set OSREL and UNAME_r in release/release.sh when building
ports to prevent ports build failures from killing the
release build.

MFC after: 3 days
X-MFC-with: r270417, r270418
Sponsored by: The FreeBSD Foundation

Merge the following from ^/projects/release-noxdev:
r269549, r269551, r269552, r269553, r269554, r269555, r269558, r269559,
r269560, r269561, r269628, r269629, r269630, r269635, r269637:

r269549:
Create a new project branch, release-noxdev, for
a sandbox workspace outside of head/ to update the
release bits for arm builds since the deprecation of
the XDEV and XDEV_ARCH make(1) variables.

r269551:
Define load_chroot_env() and load_target_env()
prototypes.

r269552:
Call load_chroot_env() and load_target_env() where
they can be mutually conflicting with regard to
TARGET, TARGET_ARCH, XDEV, and XDEV_ARCH.

r269553:
Add shebang line to arm/*.conf files since these
should be considered to be executable (albeit not
on their own) shell scripts.

r269554:
Redefine load_chroot_env() and load_target_env() in
the arm/*.conf files, and reindent.

r269555:
Simplify where load_chroot_env() and load_target_env() are
called.

r269558:
Provide example in release.conf.sample for overriding the
load_chroot_env() and load_target_env() prototypes.

r269559:
Remove a gratuitous newline.

r269560:
Unset potentially conflicting variables in load_chroot_env()
and load_target_env().

r269561:
Make global variables global, and accessible outside of
the functions within which they were once defined.

r269628:
Remove XDEV/XDEV_ARCH evaluation if EMBEDDEDBUILD is set.

r269629:
In arm/release.sh, switch TARGET/TARGET_ARCH back to the
original XDEV/XDEV_ARCH make(1) variables.

In theory, this should have been a no-op, but the TARGET and
TARGET_ARCH are now unset in load_target_env() to avoid
collision with the chroot userland.

r269630:
Export variables in the arm/*.conf files because they
need to be passed through release.sh to arm/release.sh.

Set MK_TESTS=no for the xdev target.

r269635:
As part of the XDEV/XDEV_ARCH deprecation, the
'xdev-links' target was intentionally no longer
invoked automatically.

Invoke the xdev-links target after xdev, which
creates, for example, /usr/bin/armv6-freebsd-cc
symlink to /usr/armv6-freebsd/usr/bin/cc.

r269637:
Set TARGET and TARGET_ARCH to the XDEV and XDEV_ARCH
counterparts for the xdev and xdev-links make(1)
targets.

Sponsored by: The FreeBSD Foundation

Add SRC_UPDATE_SKIP, DOC_UPDATE_SKIP, and PORTS_UPDATE_SKIP
variables. These are intended to allow bypassing the
'svn co /usr/{src,doc,ports}' step in the chroot when the
tree exists from external means.

The use case here is that /usr/src, /usr/doc, and /usr/ports
in the chroot exist as result of zfs dataset clones, so it
is possible (and happens quite often) that the included
distributions may not be consistent. (This is not the case
for -RELEASE builds, but does happen for snapshot builds.)

Tested on: stable/9@r264319
MFC After: 3 days
Sponsored by: The FreeBSD Foundation

Add a new release build variable, WITH_COMPRESSED_IMAGES.

When set to a non-empty value, the installation medium is
compressed with gzip(1) as part of the 'install' target in
the release/ directory.

With gzip(1) compression, downloadable image are reduced in
size quite significantly. Build test against head@263927
shows the following:

bootonly.iso: 64% smaller
disc1.iso: 44% smaller
memstick.img: 47% smaller
mini-memstick.img: 65% smaller
dvd1.iso: untested

This option is off by default, I would eventually like to
turn it on by default, and remove the '-k' flag to gzip(1)
so only compressed images are published on FTP.

Requested by: wkoszek
MFC After: 1 week
Sponsored by: The FreeBSD Foundation

Add a hack-ish attempt to provide backwards-compatibility with
release.conf SVNROOT entries that do not have a trailing '/'.

In order to provide a mechanism to use !svn, the trailing '/'
was added to SVNROOT so there were no assumptions being made
about '/' being in the URL between SVNROOT and SRCBRANCH.

Sponsored by: The FreeBSD Foundation

Rework how WORLD_FLAGS and KERNEL_FLAGS are set, to remove an
unnecessary expr(1) call.

Inspired by: hrs (a few months ago)
MFC after: 1 week
X-MFC-With: r262499, r262505
Sponsored by: The FreeBSD Foundation

release.sh:
- Add a VCSCMD variable that defaults to 'svn checkout',
and update places 'svn co' is used directly.
- After sourcing a configuration file, prefix SRCBRANCH,
PORTBRANCH, and DOCBRANCH with the SVNROOT.
- Properly capitalize 'FreeBSD.org' in the default SVNROOT.
- Update Copyright.

release.conf.sample:
- Add an example to use git instead of svn, by nullifying
SVNROOT, and setting SRCBRANCH, DOCBRANCH, and PORTBRANCH
to the URL fo a git repository.

release.7:
- Document VCSCMD.

Submitted by: Rick Miller (based on)
Sponsored by: The FreeBSD Foundation

Remove EMBEDDED_WORLD_FLAGS from release.sh, and move the gcc(1)
build to arm/release.sh.

This prevents needing to build a separate chroot environment for
the arm/armv6 builds when it is not absolutely necessary. This
is useful for situations where a single userland build is used
to populate more than one chroot.

Sponsored by: The FreeBSD Foundation

When bootstrapping the build chroot, the default environment is
set up for the amd64/amd64 TARGET/TARGET_ARCH combination.

Some of the build tools used here (u-boot in particular) require
use of gcc(1). The lang/gcc* from ports/ will install as 'gccNN'
instead of 'gcc', so they cannot be used without extra hacks.

Add an EMBEDDED_WORLD_FLAGS variable to be used to properly set
up the build chroot. For the RPI-B case, EMBEDDED_WORLD_FLAGS
is set to 'WITH_GCC=1', which is used to set up the build chroot.

While here, in followup to r260895, do not rely on the necessary
configuration files and/or scripts to exist in the build target
src/ tree.

To work around cases where files do not exist, copy (from the
local release/ checkout) the tools/${XDEV}/crochet-${KERNEL}.conf
to ${CHROOTDIR}/tmp/external/crochet-${KERNEL}.conf to make them
accessible to the external utilities that need them (i.e., crochet).

Sponsored by: The FreeBSD Foundation

Implement cross build support for embedded devices into
release.sh:

- Modify release/release.sh to recognize an 'EMBEDDEDBUILD'
variable. When set, release/release.sh will build the chroot
userland as is done for big-iron builds.

- Instead of running the 'buildworld', 'buildkernel', and
'release' targets in the chroot, a separate script is run.
The script tools/release/${XDEV}/release.sh sets up the rest
of the chroot workspace, such as installing ports that will
be needed, and building the 'xdev' target.

- crochet (by kientzle@) is checked out from the upstream
repository (and if needed, u-boot, in the case of Raspberry
Pi).

- tools/release/${XDEV}/release.sh then runs crochet with
a configuration file specific to the ${KERNCONF} being built.

Note that this is effectively a first-pass attempt to integrate
this into the release process.

Tested against: head@r259961
Sponsored by: The FreeBSD Foundation

Move build_doc_ports() to the if...fi block from which it is called.

Sponsored by: The FreeBSD Foundation

Put the release objdir inside the chroot.

When building the system from outside the chroot as part of the release
process, ensure the objdir is within the chroot so that the whole output
of the release.sh script is self-contained in a single directory.

Use CHROOTDIR/tmp/obj instead of CHROOTDIR/usr/obj to minimize possible
interferences with the output of the build itself.

Reviewed by: gjb
Approved by: rpaulo (mentor)

Delay copying of resolv.conf into the chroot until /etc exists.

With an unpopulated chroot dir, release building was failing because
the script was attempting to copy resolv.conf into a non-existent /etc
directory of the chroot. Fix this by copying the file only after the
installworld has happened, which will create this directory.

Reviewed by: gjb
Approved by: rpaulo (mentor)

Add WITH_DVD to RELEASE_RMAKEFLAGS, otherwise it is not actually
passed to 'make release'.

MFC after: 3 days
X-Before-RC2: yes
Sponsored by: The FreeBSD Foundation

Move OPTIONS_UNSET outside of the PBUILD_FLAGS variable, otherwise the
textproc/docproj port build fails.

Pointyhat: gjb
MFC after: 3 days
X-Before-RC1: yes
Sponsored by: The FreeBSD Foundation

Unconditionally copy the build host /etc/resolv.conf into
the chroot directory, since hostname resolution may be
needed in the case of building a dvd image (with packages)
and also setting 'NOPORTS=1'.

MFC after: 3 days
X-MFC-With: r258305, r258307
Sponsored by: The FreeBSD Foundation

Since the doc/ toolchain conversion to docbook 5.0, JADETEX
is no longer a valid option, so remove 'WITHOUT_JADETEX=yes'
from PBUILD_FLAGS.

While here, also remove 'WITHOUT_X11=yes', since it will
cause a dependency conflict by requiring both print/ghostscript9
and print/ghostscript9-nox11.

Switch to OPTIONSNG format (WITHOUT_* -> OPTIONS_UNSET='FOO'),
and unset the FOP option, which requires Java (although it is
disabled by default). Also unset the 'IGOR' option, since
textproc/igor is a validation tool, not necessarily a dependency
of the doc/ build itself.

While here, reduce the line length of PBUILD_FLAGS.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Fix a few style nits.

MFC after: 3 days
X-MFC-After: 10.0-RELEASE
Sponsored by: The FreeBSD Foundation

When building the textproc/docproj port, the ports-mgmt/pkg port
needs /var/run/ld-elf*.so.hints, which is not automatically created.

Fix reldoc build by running the ldconfig(8) startup script in the
chroot directory before starting the port build phase.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Remove hash generation from release.sh, as it is run as part of
the 'install' target in the release/Makefile.

MFC after: 3 days
Submitted by: jhb
Approved by: re (hrs)
Sponsored by: The FreeBSD Foundation

Remove extra 'MAKE_FLAGS' line.

Submitted by: jhb
Approved by: re (marius)
Sponsored by: The FreeBSD Foundation

Remove get_rev_branch(), functionality exists in the release/Makefile.

Submitted by: hrs

- Only set ARCH_FLAGS (TARGET/TARGET_ARCH) if specified, otherwise
allow the toolchain to detect the correct values.

- Remove {SRC,DOC,PORT}REVISION variables, and use 'branch@rNNNNNN'
as the {SRC,DOC,PORT}BRANCH variables.

- Only set default KERNEL_FLAGS and WORLD_FLAGS make(1) jobs if the
number of CPUs is greater than 1.

Submitted by: hrs

- Add SRC_FORCE_CHECKOUT configuration option to force svn to checkout
the src/ tree into a directory that contains files/directories, such
as a case where a custom kernel configuration file is specified.

- Allow specification of multiple KERNCONFs to pass to 'make release'.

- Move evaluation of NODOCS/NOPORTS earlier, and set based on how the
release process expects these options to be evaluated.

- Wrap KERNCONF specification in double quotes, and use 'eval' so multiple
kernel configurations do not cause the build to fail in strange ways.

- Set WITHOUT_X11 for the port build flags for the documentation toolchain
build. Also run 'clean distclean' targets during port build.

PR: 180192
Submitted by: Anes Mukhametov
MFC after: 3 days
Approved by: kib (mentor, implicit)

If MAKE_CONF and/or SRC_CONF are set and not character devices,
copy to chroot to use with release build.

Submitted by: Garrett Cooper (original version)
Approved by: kib (mentor)
MFC after: 1 week

Fix OSVERSION variable within PBUILD_FLAGS.

Submitted by: Garrett Cooper (in part)
Approved by: kib (mentor)
MFC After: 1 week
X-MFC-With: r251652
X-MFC-To: stable/9 only

Add new script to run release builds. The script is heavily based upon
the release/generate-release.sh script by nwhitehorn.

This script can use optional configuration file to override defaults,
making running multiple serialzied builds with different specific
configurations or architecture-specific tunings quite easy.

Sample overrideable options are included in the release.conf.sample
file.

Approved by: kib (mentor)
MFC After: 2 weeks
X-MFC-To: stable/9 only