release: make dependencies, not recursive make

For historical reasons, the vm-release and cloudware-release targets
were written as recursive makes of ${VMTARGETS} and ${CLOUDTARGETS};
this worked fine until we started running `make release -jN` and
had both VM and cloud targets depending on ${QEMUTGT}, at which
point things exploded when we tried to build the port twice at the
same time.

Switch vm-release and cloudware-release to have make dependencies on
their individual components; this way a single make process runs and
is aware of the duplicate dependency (and only runs it once).

MFC after: 5 days

release: Use qemu when cross-building vm images

For a bit over 5 years, we have used qemu when cross-building cloudware
images; in particular, it's necessary when installing packages which
might include post-install scripts.

Use qemu in the vm-images target too; while "generic" vm images don't
install packages, they still run newaliases and /etc/rc.d/ldconfig,
both of which fail without appropriate emulation.

MFC after: 1 week

Makefile.vm: Set CLEANDIRS correctly for vm-image

With multiple filesystems and disk image formats, we now use different
staging directories; we need to include them all in CLEANDIRS.

Fixes: 97bd53ef4d20b "Makefile.vm: Fix duplicate rc.conf files"
MFC after: 3 days

cloudware: allow disk format to be a list

Make basic-cloudinit available both in qcow2 and raw formats

MFC After: 3 days
Co-Authored-By: bapt
Sponsored by: OVHCloud (original patch from bapt)
Differential Revision: https://reviews.freebsd.org/D44747

release/Makefile.vm: Support read-only ports tree

Build qemu (if needed) with WRKDIRPREFIX=/tmp/ports DISTDIR=/tmp/distfiles
so that we can have a read-only /usr/ports and don't contaminate it. This
became an issue when I enabled parallel release building, since one image
might be creating its ports.txz file at the same time as we're building
qemu as a prerequisite for building another image.

MFC after: 5 days

Revert "cloudware: allow disk format to be a list"

This reverts commit 6ec9aaf63c81a68881cb6312f777349a0ac82ad5.

Requested by: cperciva

cloudware: allow disk format to be a list

Make basic-cloudinit available both in qcow2 and raw formats

MFC After: 1 week
Reviewed by: Allanjude
Sponsored by: OVHCloud
Differential Revision: https://reviews.freebsd.org/D44747

cloudinit: fix creation of cloudinit ready VMs

The framework do not support multiple format images, so this last minute
addition of qcow2 format was clearly not a good idea.

while here fix a typo

Reported by: cperciva

release: add basic cloudinit images

Provide both zfs and ufs images which a 1MB partition reserved for the
config drive wearing a GPT Label "config-drive" to allow consumer to
know where they should push the config drive on the provided image.

2 formats available: qcow2 and raw

This has been tested on OVHCloud baremetal via "bring your own image"
Also tested on openstack

Reviewed by: emaste
Sponsored by: OVHCloud
Differential Revision: https://reviews.freebsd.org/D44369

release: Add AZURE to CLOUDWARE

Let Azure use the image directly built by the release engineering team.

Reviewed by: emaste
Approved by: cperciva (re)
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41906

release/Makefile.vm: Add cloudware overrides

Allow the cloudware *_FLAVOURS and *_FSLIST values to be overridden
at the command line, to assist users who want to e.g. build only one
of the many EC2 AMIs available.

release/Makefile.vm: Rework emulator-portinstall

The emulator-portinstall target now unconditionally ensures that qemu
is installed; but is only invoked if needed (aka. when cross building
VM images).

MFC After: 3 days
MFC With: 97bd53ef4d20 ("Fix duplicate rc.conf files")

Makefile.vm: Fix duplicate rc.conf files

Two bugs in Makefile.vm resulted in disk images being "built" multiple
times, resulting in lines added to /etc/rc.conf being duplicated:

1. The vm-image target reused the same "staging tree" directory for all
of its builds (multiple disk image types and multiple filesystem types).

2. The cw-type-flavour-fs target depends on emulator-portinstall, which
did not have a 'touch ${.TARGET}' and thus re-ran every time -- and
caused the cw-type-flavour-fs target to be re-run. This was triggered
by release builds running `make cloudware-release` (creating the disk
images) followed by `make ec2amis` (which re-created the disk images
prior to uploading them).

MFC After: 1 week
Sponsored by: https://www.patreon.com/cperciva

EC2: Add experimental cloud-init images

Known issues:

1. The ec2-user user is created with a homedir of /usr/home/ec2-user
instead of /home/ec2-user; this appears to be a bug in cloud-init's
FreeBSD support.

2. Cloud-init configures IPv4 networking but not IPv6 networking.

releng/14.0 candidate.

Discussed with: gjb
Reviewed by: imp
MFC after: 5 days
Relnotes: yes
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41793

EC2: Flavour existing AMIs as "base"

Using the recently-added "cloudware flavours" mechanism, turn the
existing EC2 AMIs into a new "base" flavour. The only user-visible
change is that AMI names now include the word "base".

releng/14.0 candidate.

Discussed with: gjb
Reviewed by: imp
MFC after: 5 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41791

release/Makefile.vm: Add support for CW flavours

There are two "Vagrant" images right now: "Vagrant Image for VirtualBox"
and "Vagrant Image for VMWare". Rather than listing these separately in
a CLOUDWARE list, place "VAGRANT" into a CLOUDWARE_TYPES list and then
use a VAGRANT_FLAVOURS variable to identify the two versions. Add make
logic to allow defaults (in this case, image format and filesystem) to
be specified once for VAGRANT and inherited by both flavours.

This will make future work to add flavoured EC2 images simpler.

releng/14.0 candidate.

Discussed with: gjb
Reviewed by: imp
MFC after: 5 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41790

release/Makefile.vm: Garbage collect unused bits

The cw*-package targets were introduced in February 2015 as part of
adding support for building GCE images; but GCE support was reworked
in June 2015 and the cw*-package targets were in fact never used.
Remove them.

The *_DISK variables were introduced in February 2015 as part of
adding the cloudware-install target; this was removed in May 2016 as
the cloudware images are published via the respective cloud systems
and not published as disk images via the FreeBSD FTP site. As such,
the *_DISK variables are not unused; remove them.

releng/14.0 candidate.

Discussed with: gjb
Reviewed by: imp
MFC after: 5 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41789

releng-gce: Add ZFS support to GCE image buliding and uploading

Approved by: delphij (re), gjb (re)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41527

releng-azure: Add ZFS support to Azure image buliding and uploading

Approved by: gjb (re)
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41454

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

EC2: Build and upload AMIs for both UFS and ZFS

The pre-existing "ec2ami" target builds and uploads a single AMI
(with filesystem determined by ${VMFS}) as before; a new "ec2amis"
target does both UFS and ZFS.

Reviewed by: gjb
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41343

release: Build cloudware with multiple FS types

New ${CW}_FSLIST settings control the list of filesystem types with which
each cloudware image will be built; currently these are all set to "ufs",
i.e. no change from previous.

The cloudware images have their filesystem type as part of their file
name; for backwards compatibilty the ${VMFS} image is linked to the
previously used file name. This compatibility can be removed once all
the cloudware uploading/publishing code has been updated to use the new
image names (possibly more than one of them).

Reviewed by: gjb
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41341

release: Build UFS and ZFS VM images

A new option 'VMFSLIST' controls the list of filesystems for which VM
images will be built; the default is to build both UFS and ZFS.

The vm-install target installs these as ${OSRELEASE}-${FS}.${FORMAT},
e.g. FreeBSD-14.0-CURRENT-amd64-zfs.vmdk. For backwards compatibility,
the ${VMFS} image is linked to the previously used ${OSRELEASE}.${FORMAT}
name.

Cloudware building will be updated in a later commit.

Reviewed by: gjb
Reviewed by: emaste, markj (previous version)
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41340

release: Add code to make FreeBSD/Firecracker

This just stages the kernel and builds a stripped-down rootfs for use
with the Firecracker VMM. At some point in the future the release
engineering team might start publishing these, but initially it's
just here to simplify FreeBSD/Firecracker development and testing.

Note that the rootfs generated:
* Uses an IP address of 10.0.0.2 with a gateway of 10.0.0.1,
* Has sshd enabled,
* Has user "freebsd" with password "freebsd" and a root password
of "root", and
* Is 1 GB in size (but has growfs enabled).

All of those are subject to change without notice; anyone intending to
use FreeBSD/Firecracker in anything remotely resembling a production
environment should talk to cperciva first.

Reviewed by: gjb
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D40956

release/oci: add Oracle Cloud image builder

Provides an OCI (Oracle Cloud Infrastructure) release target for
Oracle's KVM-based VM implementation. Tested using 13.1-RELEASE,
primarily on Ampere CPU on A1.Flex VM shapes, but also works on
amd64 shapes.

- supports cloud-init and custom scripts
- provides a freebsd@ sudo-enabled user
- root user disabled over ssh & console

Approved by: gjb
Reviewed by: emaste
MFS after: 1 week
Sponsored by: The FreeBSD Foundation
Sponsored by: SkunkWerks, GmbH
Technical assistance from: Oracle
Differential Revision: https://reviews.freebsd.org/D34746

release: Add support for creating ZFS-based VM images

The change extends vmimage.subr to handle a new parameter, VMFS, which
should be equal to either "ufs" or "zfs". When it is set to ZFS, we use
makefs to create a bootable pool populated using the same dataset layout
as bsdinstall and "poudriere image" use. The pool can be grown using
the growfs rc.d script, just as in UFS images.

This will make it easy to provide VM and cloud images with ZFS as the
root filesystem. So far I did not do extensive testing of cloud images;
I merely verified that creation of ZFS-based AWS AMIs works and allows
me to create amd64 and arm64 EC2 instances with ZFS as the root
filesystem.

Reviewed by: emaste, gjb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34426

release: increase IMAGE_SIZE for arm, arm64, riscv

Related to: PR 264032
MFC after: 5 minutes
Sponsored by: Rubicon Communications, LLC ("Netgate")

Update Azure release bits

Imports the changes for building official images on Azure Marketplace,
which fulfill the requirements of Azure and FreeBSD cloud images like
disk layout and UEFI for Gen2 VM, along with some minor improvements like
configurations to speed up booting.

"CLOUDWARE" list will be updated after some more collaborations with re
completed.

Reviewed by: re (gjb)
Sponsored by: The FreeBSD Foundation
Technical assistance from: Microsoft
Differential Revision: https://reviews.freebsd.org/D23804

release: fix VMSIZE following 1ca8842f3ad9

truncate(1) is not case-sensitive with regard to setting the size
of a file. makefs(8), however, does not honor upper-case values.
Update release-specific files and the release(7) manual page to
reflect this.

MFC with: 1ca8842f3ad9
Submitted by: ehem_freebsd_m5p.com (original)
Differential Review: https://reviews.freebsd.org/D28979
Sponsored by: Rubicon Communications, LLC ("Netgate")

Include Makefile.inc1 in Makefile.vm, missed in previous change

MFC after: 3 days
MFC with: 80ab50e1de19, 0be274d37379
Sponsored by: Rubicon Communications, LLC ("Netgate")

Adjust where UNAME_r is defined for ports builds

In followup to 80ab50e1de19ca125f05a13937c796d48c4edd4a,
export UNAME_r in Makefile.inc1 instead of Makefile.vm.

MFC after: 3 days
MFC with: 80ab50e1de19
Sponsored by: Rubicon Communications, LLC ("Netgate")

pass UNAME_r to fix building 14.x ports on 13.x

MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")

release: Add an image for CI

A lot of projects CI can't do FreeBSD tests currently.
The main reason is that the project CI infrastructure is runned on Linux
and that our images aren't modifiable from a Linux hosts.
Add a basic image specific for this case (called BASIC-CI for a lack of a
better name).
The image have no package pre-installed.
It only have a few modification to have dhcp client runned on the default
interface and sshd started with option to be able to log on without a password
as root.

Sponsored by: The FreeBSD Foundation

Reviewed by: re (gjb@)
Differential Revision: https://reviews.freebsd.org/D25598

Ensure the TYPE, BRANCH, and REVISION variables are set in
cloudware targets when OSRELEASE is overridden.

Submitted by: Trond Endrestol
PR: 243287
MFC after: 3 days
Sponsored by: Rubicon Communications, LLC (netgate.com)

Increase the default VMSIZE for raw, qcow2, vhd, and vmdk virtual
machine images due to 'filesystem full' failures.

Sponsored by: Rubicon Communications, LLC (Netgate)

Reduce the default image size for virtual machine disk images from
30GB to 3GB. The raw images can be resized using truncate(1), and
other formats can be resized with tools included with other tools
included with other hypervisors.

Enable the growfs(8) rc(8) at firstboot if the disk was resized
prior to booting the virtual machine for the first time.

Discussed with: several
PR: 232313 (requested in other context)
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Add support for cross-building cloudware images.

If MACHINE_ARCH doesn't match TARGET_ARCH, and we're not in the special
case of building i386 images on an amd64 host, we need to pull in the
qemu-user-static package; this allows us to run some commands inside
the VM disk image chroot, most notably to install packages.

Reviewed by: gjb
MFC after: 2 weeks
Sponsored by: FreeBSD/EC2 patreon (https://www.patreon.com/cperciva)

Bump the size of virtual machine disk images from 20G to 30G,
providing more space for a local buildworld to succeed without
attaching separate disks for /usr/src and /usr/obj.

Reported by: mckusick
MFC after: 3 days

Sponsored by: The FreeBSD Foundation

Ensure ${_CW} is uppercase when passing '-c' to mk-vmimage.sh,
otherwise with 'CLOUDWARE=vagrant-virtualbox', the path to the
configuration file may be incorrect.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Force a few variables to be upper case when they are evaulated.

This allows specifying, for example: CLOUDWARE=vagrant-vmware

Approved by: re (gjb)

Pass SWAPSIZE in env(1) when invoking mk-vmimage.sh for the
vm-image target, missed in r308737.

Sponsored by: The FreeBSD Foundation

Pass SWAPSIZE in env(1) when invoking mk-vmimage.sh, otherwise
mkimg(1) does not create the second partition after r307008.

Sponsored by: The FreeBSD Foundation

Add the ability to override the size of the swap partition when building
VM images. The default continues to be 1G.

Reviewed by: gjb@
Sponsored by: Chelsio Communications

Remove vestigial cloudware-install bits missed in r300767.

Sponsored by: The FreeBSD Foundation

Remove the cw*-install target.

All operations are performed on the original output file,
and nothing is ever done with the copied file (as in, no
modifications and/or publication from the copied file).

Sponsored by: The FreeBSD Foundation

Disconnect the OPENSTACK target from the CLOUDWARE list.

This was reported to not work as expected, and feedback
following the report was never received. In addition, we
do not publish these anywhere publicly.

The configuration file is not (yet) removed, in hopes that
the remaining details on the original issues can be resolved,
and we can publish the resulting images in the future.

Sponsored by: The FreeBSD Foundation

Disconnect the AZURE target from the CLOUDWARE list.

Microsoft has graciously overtaken publication of FreeBSD
on Azure since 10.3-RELEASE.

Many thanks to Microsoft for their support of FreeBSD on
Azure.

The configuration file and Makefile.azure are not (yet)
planned to be removed, as they may be useful in the future.

Sponsored by: The FreeBSD Foundation

Deprecate MD5 checksum generation in favor of SHA512.

This was discussed during the 10.2-RELEASE cycle, however
since we were nearing the end of the cycle, we decided to
defer this change until after 10.2-RELEASE.

Reminded by: so (delphij), jmg
MFC after: 5 days
Sponsored by: The FreeBSD Foundation

Add support for building VirtualBox Vagrant images.
Abstract the build, package and upload to handle building either type.

Approved by: re (gjb)

Add initial support for building Vagrant images for VMWare. Next steps will be adding Virtualbox support and uploading to Hashicorp Atlas for others to consume.

Approved by: re (gjb)

Add initial support for automatically uploading GCE virtual
machine images to the Google Compute Engine platform.

By default, gcutil/gsutil requires an Oauth2 login generated
from a URL that must be opened in a browser, a verification
code copied back to the terminal from which it was invoked,
etc., etc., making it near impossible for automation.

I've hacked together an evil solution to work around this,
so unless GCE_LOGIN_SKIP is set to a non-empty value, this
Makefile will not do anything useful.

As a result of this commit, remove the gce-package.sh script
that was never, nor will ever be, used.

MFC after: 3 days
X-MFC-Note: (hopefully)
Sponsored by: The FreeBSD Foundation

Add Makefile.azure, used to upload VHD images to the
Microsoft Azure environment, similar to the EC2 AMIs.

MFC after: 3 days
Tested with: stable/10@r284517 (patched)
Sponsored by: The FreeBSD Foundation

Add code for creating an EC2 AMI.

Add bits for building EC2 disk images. Make logic for uploading these
to create EC2 AMIs will come in a later commit.

Add a CLOUDPACKAGE variable to contain a list of targets for
post-install packaging cloud provider images.

Add a 'gce-package.sh' script to generate the final output
image ready for upload to the GCE platform. Right now, this
is the only image that has a specific output format (GNU-tar),
and this implementation is expected to be temporary.

This is not directly connected to the other release targets.

MFC after: 1 week
X-MFC-with: r279249, r279250
Sponsored by: The FreeBSD Foundation

Remove a stale comment.

MFC after: 1 week
X-MFC-with: r279249
Sponsored by: The FreeBSD Foundation

Add 'cloudware-install' target to handle copying cloud
disk images to a directory within DESTDIR.

Add CLOUDINSTALL variable to automatically generate the
actual list of install targets. While here, rename the
cloud-specific names from 'vm-' to 'cw-' to avoid naming
collisions.

Add AZURE_DISK, GCE_DISK, and OPENSTACK_DISK output file
variables.

Add 'cloudware-install' to the 'install' target.

MFC after: 1 week
Sponsored by: The FreeBSD Foundation

release: use xz via pipe when compressing the tarballs.

libarchive(3) doesn't support the new liblzma API yet, but this change
allows us to enable multi-threaded xz compression.
``make release'' should now finish in half the time on a machine with
several cores and fast disks (our typical build server).

This behaviour only applies when building a release and it doesn't
affect buildworld/installworld. To disable threaded xz compression,
set XZ_THREADS=1.

Reviewed by: gjb
Tested by: gjb

Enable multi-threaded xz(1) compression, after r278433.
Allow multi-threaded xz(1) to be turned off by specifying
NO_XZTHREADS, and allow number of threads to be overridden
by specifying XZ_THREADS=N.

MFC after: 1 week
X-MFC-needs: r278433
Sponsored by: The FreeBSD Foundation

When iterating through VMFORMATS, the VMBASE file is
not removed or truncated to a zero-size file, which
if used to create more than one disk image format, can
result in accidental pollution of the target formatted
disk image.

Instead of using a single VMBASE image (vm.img, by
default), use a single base file for each format, named
as VMFORMAT.img, which produces VMBASE.VMFORMAT as the
final formatted image.

Reported by: cperciva
MFC after: 1 month
X-MFC-with: r277458, r277536
Sponsored by: The FreeBSD Foundation

Rename the 'release' target to 'real-release', and
add two targets, 'vm-release' and 'cloudware-release',
that are invoked if WITH_VMIMAGES and WITH_CLOUDWARE
are not empty.

This fixes an issue where 'make release' would not
build the cloud provider targets because CLOUDWARE was
not yet set. [1]

Move the WITH_VMIMAGES and WITH_CLOUDWARE targets to
Makefile.vm.

Note: There is no 'cloudware-install' target yet, since
some of the disk image names may need to be specific to
the provider, so this is probably best handled by the
build scripts.

Reported by: cperciva [1]
MFC after: 1 month
X-MFC-with: r277458
Sponsored by: The FreeBSD Foundation

Add initial support for the GCE (Google Compute Engine)
cloud hosting provider image.

Many thanks to swills@ for his work on getting this to
this point.

Submitted by: swills
Sponsored by: The FreeBSD Foundation

Add 'list-vmtargets' target, which produces a list of
all supported VM and cloud provider images.

Add VHD_DESC, VMDK_DESC, QCOW2_DESC, RAW_DESC image
descriptions.

Format the output to make a bit more readable.

Update release(7) to document the list-vmtargets target.

Sponsored by: The FreeBSD Foundation

Add a 'list-cloudware' target to print the list of
supported CLOUDWARE values and a description.

Add the AZURE_DESC and OPENSTACK_DESC descriptions.

Sponsored by: The FreeBSD Foundation

Merge duplicative vm-CLOUDTYPE targets before additional duplication gets
added by the impending arrival of ec2 and gcloud.

Remove a stray directory from CLEANFILES.

Sponsored by: The FreeBSD Foundation

Change path for mk-vmimage.sh from ${TARGET}/ to
scripts/ now that it is consolidated into one file.

Fix paths for the base image and output disk image
files.

Sponsored by: The FreeBSD Foundation

Fix DESTDIR for installworld, and make sure it is created
before use.

Sponsored by: The FreeBSD Foundation

Add CLEANFILES entry for VM targets

Sponsored by: The FreeBSD Foundation

Initial rewrite to consolidate VM image build scripts into one.
There may be some very sharp edges here while refactoring.

- Move amd64/mk-vmimage.sh -> scripts/mk-vmimage.sh.
- Remove vm-base target from Makefile.vm.
- In vm-image target, use getopts flags for argument passing.
- Create tools/vmimage.subr, containing default and prototype
for the following functions that are used to drive the build,
run in this order:

vm_install_base()
vm_extra_install_base()
vm_extra_install_packages()
vm_extra_install_ports()
vm_extra_enable_services()
vm_extra_pre_umount()
vm_create_disk()
vm_extra_create_disk()

- In tools/azure.conf, override:

vm_extra_install_base()
vm_extra_pre_umount()
vm_extra_create_disk()

- In tools/openstack.conf, override:

vm_extra_install_base()
vm_extra_pre_umount()

Sponsored by: The FreeBSD Foundation

Add a 'vm-cloudware' target, used to drive all
targets in CLOUDTARGETS.

Sponsored by: The FreeBSD Foundation

Add line continuation so OPENSTACKCONF is actually included
in the env(1).

Sponsored by: The FreeBSD Foundation

Initial commit providing a mechanism to create
openstack images as part of the release build.

This mimics the way Microsoft Azure images are
built, with the addition of installing the
net/cloud-init package and adding a (commented)
rc.conf(5) entry for cloudinit.

Sponsored by: The FreeBSD Foundation

Avoid hard-coding the Azure image file format.
While here, avoid using OSRELEASE for the output
file name.

Sponsored by: The FreeBSD Foundation

Add glue to allow enabling building cloud provider
VM images by default.

When WITH_CLOUDWARE is not empty, add CLOUDTARGETS
to the release/Makefile 'release' target.

CLOUDTARGETS is generated from the contents of
CLOUDWARE, which should be a list of all supported
target providers.

Sponsored by: The FreeBSD Foundation

Move virtual machine / cloud provider targets and
options from release/Makefile to their own Makefile.

Sponsored by: The FreeBSD Foundation