release: Rework vm_extra_pre_umount

The vm_extra_pre_umount function in vmimage.subr served two purposes:
It removed /etc/resolv.conf and /qemu (if cross-building), and it
provided a function for cloudware to override in order to make cloud
specific changes to the filesystem before constructing a disk image.

This resulted in a number of bugs:
1. When cross-building, the emulator binary was left as /qemu in the
Azure, GCE, Openstack and Vagrant images.
2. The build host's resolv.conf was left as /etc/resolv.conf in the
basic-ci and basic-cloudinit images.
3. When building GCE images, a Google-specific resolv.conf file was
constructed, and then deleted before the disk image was created.

Move the bits needed for running code inside a VM staging directory
from vm_install_base into a new vm_emulation_setup routine, and move
the corresponding cleanup bits from vm_extra_pre_umount to a new
vm_emulation_cleanup routine.

Remove the /qemu and /etc/resolv.conf cleanups from the cloudware
configuration files (where they exist) since we will now be running
vm_emulation_cleanup to remove those even when vm_extra_pre_umount
has been overridden.

Override vm_emulation_cleanup in gce.conf since in that one case (and
*only* that one case) we don't want to clean up resolv.conf (since it
was constructed for the VM image rather than copied from the host).

releng/14.1 candidate.

MFC after: 1 week
Sponsored by: https://www.patreon.com/cperciva

release: Change vmimage EFI GPT label

This matches the default bsdinstall nomenclature.

PR: 278480
Reviewed by: imp
Pull Request: https://github.com/freebsd/freebsd-src/pull/1188

release: add basic cloudinit images

Provide both zfs and ufs images which a 1MB partition reserved for the
config drive wearing a GPT Label "config-drive" to allow consumer to
know where they should push the config drive on the provided image.

2 formats available: qcow2 and raw

This has been tested on OVHCloud baremetal via "bring your own image"
Also tested on openstack

Reviewed by: emaste
Sponsored by: OVHCloud
Differential Revision: https://reviews.freebsd.org/D44369

release/tools: Add support for building armv7 vm

Currently there is no support for generating armv7 vm images in the
release artifacts. In fact in terms of release artifacts and
architecture there is no good reason to have a vm release artifact for
armv7 as those are mostly used in SOCs or embedded boards. However
considering that developers actually do need an easy way to test armv7
with a vm running this is really important. As part of pre-commit ci for
developers this can be really helpful for the end developers.

Approved by: cperciva, imp, re
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D43952

release: Add missing /var/crash ZFS dataset

This matches the layout from bsdinstall.

MFC after: 1 week
Fixes: 89585511cc05 ("release: Add support for creating ZFS-based VM images")
Pull Request: https://github.com/freebsd/freebsd-src/pull/981

release: stop generating mergemaster databases

mergemaster was deprecated some time ago and will be removed from
FreeBSD 15.

Reviewed by: imp
Relnotes: Yes
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D41797

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

release/tools/vmimage.subr: switch zfs dataset from /usr/home to /home

Change the vmimage script for zfs to create /home as a dataset
rather than /usr/home, ala change to bsdinstall's zfs script.

Reviewed by: markj
Differential Revision: <https://reviews.freebsd.org/D40111

release: Report disk image filename

For someone new to the release bits it's not always clear what files are
being created. Report the disk image name explicitly.

Reviewed by: gjb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39953

release(7): Enable zpoolupgrade rc script in ZFS based VM images

This will enable VM access to all ZFS feature automatically, only on a
newly installed or provisioned VM or cloud instance.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D37283

release: Add support for creating ZFS-based VM images

The change extends vmimage.subr to handle a new parameter, VMFS, which
should be equal to either "ufs" or "zfs". When it is set to ZFS, we use
makefs to create a bootable pool populated using the same dataset layout
as bsdinstall and "poudriere image" use. The pool can be grown using
the growfs rc.d script, just as in UFS images.

This will make it easy to provide VM and cloud images with ZFS as the
root filesystem. So far I did not do extensive testing of cloud images;
I merely verified that creation of ZFS-based AWS AMIs works and allows
me to create amd64 and arm64 EC2 instances with ZFS as the root
filesystem.

Reviewed by: emaste, gjb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34426

Update Azure release bits

Imports the changes for building official images on Azure Marketplace,
which fulfill the requirements of Azure and FreeBSD cloud images like
disk layout and UEFI for Gen2 VM, along with some minor improvements like
configurations to speed up booting.

"CLOUDWARE" list will be updated after some more collaborations with re
completed.

Reviewed by: re (gjb)
Sponsored by: The FreeBSD Foundation
Technical assistance from: Microsoft
Differential Revision: https://reviews.freebsd.org/D23804

release: fix copypasta

Approved by: gjb
MFC after: 1 week
X-MFC-With: fd17ea8c1849039c436f7192ca407db70561df03
Sponsored by: Rubicon Communications, LLC ("Netgate")

release: make pkg installs more robust

Currently pkg(8) will fail to install any package if one is missing, so
make this a loop to prevent one missing package from preventing the rest
from installing. Seen building an AWS AMI for aarch64 on main and
ebsnvme-id is not available in the repo at the moment.

Approved by: gjb
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")

Use makefs(8) in release VM-image generation instead of md(4) and newfs.

Using makefs instead reduces the privileges needed to build VM images,
simplifies the script (no need to copy files to a fresh image at the end),
and improves portability by allowing generation of cross-endian images.
As a result of the last, this patch also adds support for generation of
powerpc64 and powerpc64le VM images.

No other changes to the output. Tested and working for both amd64 and
powerpc64 targets.

Reviewed by: gjb
Differential Revision: https://reviews.freebsd.org/D28912

Mount the ESP on /boot/efi in VM images.

This follows an earlier change (0b7472b3d8d2) for mounting the ESP from
systems set up through the installer and should be MFC'ed with it.

riscv: allow building virtual machine images

RISC-V has the same booting requirements as arm64 (loader.efi, no legacy
boot options), so generated images for both architectures have the same
partition layout.

Reviewed by: gjb
Differential Revision: https://reviews.freebsd.org/D27044

arm64: convert virtual machine images to GPT

These images were switched to MBR in r281876 as a way to cope with a
hard-coded partition GUID in QEMU's default EFI firmware. Enough time
has passed that this is no longer a problem; QEMU versions >= 4.0
include a copy of edk2 EFI firmware that can detect the root filesystem
properly. Alternatively, sysutils/u-boot-qemu-arm64 can be used.

Switch back to building these images with a GPT partition table, and
re-enable the swap partition.

Reviewed by: gjb, emaste
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D26986

vmimage.subr: noisier failure for unsupported targets

The return code of write_partition_layout() doesn't bubble up, so an
invocation of make vm-release for an incorrect/unsupported target will
appear to succeed while make vm-install will fail due to missing
files. This isn't a common point of failure, but is worth handling
properly.

Upgrade this case to print a message to stderr, and exit in place. This
is okay to do since at this point in the execution of mk-vmimage.sh,
cleanup() has already been run.

Reviewed by: gjb
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D26985

Slight refactor in vmimage.subr

De-duplicate the invocation of mkimg(1). No functional change.

Reviewed by: gjb
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D26984

Enable EFI system partition on amd64 and i386 VM images

EFI support is a hard requirement for generating Hyper-V Gen2 VM images.

Reviewed by: gjb
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D25655

Revert r348438.

The fix to override the default python version when building
the sysutils/py-google-compute-engine did not work, and there
are still issues that need to be addressed in the port itself.

See bugzilla 238267 for additional details.

MFC after: 6 days
MFC with: r348438
MFC note: no-op to appease the merge tracker
Sponsored by: The FreeBSD Foundation

Fix GCE virtual machine startup.

The ports/head branch recently switched to python3 as the default,
which breaks the sysutils/py-google-compute-engine startup scripts,
as lang/python installs lang/python3{,.x} where lang/python2{,.x}
are needed.

Set DEFAULT_VERSIONS in release/tools/gce.conf to python=2.7, and
remove the lang/python3 inclusion in VM_EXTRA_PACKAGES.

Additionally, unset DEFAULT_VERSIONS in release/tools/vmimage.subr
to prevent persistence of DEFAULT_VERSIONS=python=2.7 in subsequent
VM/cloud image builds.

Note: at present, this affects only 13-CURRENT and 12-STABLE, as
the stable/11 branch had already switched to using the 2019Q2 branch
at the start of the 11.3-RELEASE cycle, so this does not immediately
affect 11.3-BETA, hence the 1-week merge timeout. This had been
manually tested on 13-CURRENT.

Reported by: ler (privately)
MFC after: 1 week
Sponsored by: The FreeBSD Foundation

Reduce the default image size for virtual machine disk images from
30GB to 3GB. The raw images can be resized using truncate(1), and
other formats can be resized with tools included with other tools
included with other hypervisors.

Enable the growfs(8) rc(8) at firstboot if the disk was resized
prior to booting the virtual machine for the first time.

Discussed with: several
PR: 232313 (requested in other context)
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Add support for cross-building cloudware images.

If MACHINE_ARCH doesn't match TARGET_ARCH, and we're not in the special
case of building i386 images on an amd64 host, we need to pull in the
qemu-user-static package; this allows us to run some commands inside
the VM disk image chroot, most notably to install packages.

Reviewed by: gjb
MFC after: 2 weeks
Sponsored by: FreeBSD/EC2 patreon (https://www.patreon.com/cperciva)

Rework UEFI ESP generation

Currently, the installer uses pre-created 800KB FAT12 filesystems that
it dd's onto the ESP partition.
This changeset improves that by having the installer generate a FAT32
filesystem directly onto the ESP using newfs_msdos and then copying
loader.efi into /EFI/freebsd.
For live installs it then runs efibootmgr to add a FreeBSD boot entry
in the BIOS.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D17947

Fix an indentation nit.

Sponsored by: The FreeBSD Foundation

Move sys/boot to stand. Fix all references to new location

Sponsored by: Netflix

Add option UNIFIED_OBJDIR, on by default, which moves the default build OBJDIR.

This changes the build OBJDIR from the older style of /usr/obj/<srcdir> for
native builds, and /usr/obj/<target>.<target_arch>/<srcdir> for cross builds to
a new simpler format of /usr/obj/<srcdir>/<target>.<target_arch>. This
new format is used regardless of cross or native build. It allows
easier management of multiple source tree object directories.

The UNIFIED_OBJDIR option will be removed and its feature made permanent
for the 12.0 release.

Relnotes: yes (don't note UNIFIED_OBJDIR option since it will be removed)
Prior work: D3711 D874
Reviewed by: gjb, sjg
Discussed at: https://lists.freebsd.org/pipermail/freebsd-arch/2016-May/017805.html
Discussed with: emaste
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D12840

Set a default hostname for virtual machine images.

A recent bug in security/sudo causes segmentation faults when
the system is not configured with a hostname, which causes issues
with some virtual machine setups, notably Vagrant. Set the default
hostname to the output of 'uname -o'.

Submitted by: Nicholas Fiorentini
Sponsored by: The FreeBSD Foundation

Bootstrap etcupdate(8) and mergemaster(8) databases when creating
virtual machine images and embedded images, similar to what is
done when extracting base.txz to the target root filesystem in
an new installation.

Noticed by: marius
Tested with: head@r323729
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Enable DHCP and IPv6 autoconfig on non-cloud VM images.

PR: 203653
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Trim trailing '/release/..' when setting _OBJDIR so arm64/aarch64
boot1.efifat is properly located when creating virtual machine images.

Sponsored by: The FreeBSD Foundation

Add the ability to override the size of the swap partition when building
VM images. The default continues to be 1G.

Reviewed by: gjb@
Sponsored by: Chelsio Communications

In vm_copy_base(), turn off SU+J on the resultant filesystem,
leaving only SU enabled.

Discussed with: kib (a few weeks ago)
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Fix virtual machine disk format creating by passing VMFORMAT
to mkimg(1)

PR: 200068
Submitted by: Jeremy Norris
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Fix a comment.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Create the arm64/aarch64 VM disk image as MBR instead of
GPT scheme. UEFI needs to know the unique partition GUID
with GPT, which changes each time. Specifically, the QEMU
EFI BIOS file has this hard-coded.[1]

Since the GPT labels are now unavailable, unconditionally
label the root filesystem as 'rootfs' with newfs(8), since
it does not hurt anything anywhere else. For the arm64 case,
'/' is mounted from /dev/ufs/rootfs; for all other VM images,
'/' is mounted from /dev/gpt/rootfs.

Unfortunately, since the /dev/gpt/swapfs label is also lost,
set NOSWAP=1 for the arm64/aarch64 images. This is temporary,
until I figure out a scalable solution to this. But, a certain
piece of softare was written "very fast", and ended up living
for 15 years. We can deal with this for a week or so.

Information from: andrew, emaste [1]
Sponsored by: The FreeBSD Foundation

Revert r281809, which did more harm than good, and apply
a more proper fix when attempting to locate the /boot
files.

Sponsored by: The FreeBSD Foundation

Simplify variable expansion in attempt to fix the vm-image build.

Sponsored by: The FreeBSD Foundation

Add support for building arm64/aarch64 virtual machine images.

Sponsored by: The FreeBSD Foundation

When building VM disk images, vm_copy_base() uses tar(1) to
copy the userland from one md(4)-mounted filesystem to a clean
filesystem to prevent remnants of files that were added and
removed from resulting in an unclean filesystem. When newfs(8)
creates the first filesystem with journaled soft-updates enabled,
the /.sujournal file in the new filesystem cannot be overwritten
by the /.sujournal in the original filesystem.

To avoid this particular error case, do not enable journaled
soft-updates when creating the md(4)-backed filesystems, and
instead use tunefs(8) to enable journaled soft-updates after
the new filesystem is populated in vm_copy_base().

While here, fix a long standing bug where the build environment
/boot files were used by mkimg(1) when creating the VM disk
images by using the files in .OBJDIR.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Improve check for whether ${DESTDIR}/dev is mounted.

Submitted by: gcooper

Clean up filesystem unmounting in vmimage builds:
* Remove vm_umount_base function which is currently unused.
* Add umount_loop function which loops attempting to unmount one filesystem.
* Replace calls to umount with calls to umount_loop.
* Don't attempt to unmount ${DESTDIR}/dev if it isn't mounted.

The looping is necessary because sometimes umount fails due to filesystems
being busy. The most common cause of such busyness is periodic(8) jobs
running `find / ...`.

Reviewed by: gjb

When creating VM images, copy the contents of the created filesystem into
a new filesystem before packaging it into a disk image. This prevents
"remnants" of deleted files from showing up in the VM images, and reduces
their compressed size (by about 10% for the cloudware images) as a result.

Looks good to: gjb

Add a comment to note that setting hw.vga.textmode=1 is
temporary.

Sponsored by: The FreeBSD Foundation

Enable the textmode console by default for VM images,
since there is no way to tell if the environment will
be able to use the graphics-mode console.

Sponsored by: The FreeBSD Foundation

Fix indentation nit.

Sponsored by: The FreeBSD Foundation

In vm_extra_install_packages(), only bootstrap pkg(8)
if VM_EXTRA_PACKAGES is empty.

In vm_extra_pre_umount(), cleanup downloaded packages
if pkg(8) was bootstrapped earlier.

Inspired by: cperciva
Sponsored by: The FreeBSD Foundation

Revert r274773, after I *specifically* objected to this change
numerous times.

Sponsored by: The FreeBSD Foundation

Change how packages are installed into VM images: Rather than chrooting
into the image and running 'pkg install' from there, use 'pkg fetch' to
download packages into a temporary location and then 'pkg add' to install
them into the image.

This simplifies the code by avoiding the need to copy /etc/resolv.conf
into the image and then delete it later, and makes it possible to cross
build (e.g., to create an amd64 image when running on i386 hardware; or
in the future for building disk images for embedded platforms).

Because pkg was implicitly installed when VM_EXTRA_PACKAGES was non-empty,
add it to VM_EXTRA_PACKAGES in azure.conf and openstack.conf to maintain
the current behaviour.

By default repo-FreeBSD.sqlite is copied into the image, (a) to match
previous behaviour, where the file would be downloaded by the chrooted
pkg invocation; and (b) because it may be useful for testing purposes,
e.g., to see why a package didn't get installed. Because this file is
large (46 MB) and not likely to be useful in -RELEASE images which are
being launched into Clouds several months later, it can be disabled by
setting NOREPOSQLITE.

As far as I know this commit does not change the disk images produced in
any filesystem-visible way.

Add NOSWAP option which can be set by a vmimage.conf file to specify that
no swap space should be created in the image. This will be used by EC2
builds, since FreeBSD/EC2 allocates swap space on "ephemeral" disks which
are physically attached to the Xen host node.

Silence errors when umounting the chroot's /dev, since it probably doesn't
exist when we're running this.

Unmount filesystems before attempting to destroy the md which holds them.

In vm_install_base(), copy the host resolv.conf into
the build chroot before attempting to do anything that
requires working DNS (i.e., pkg bootstrap).

In vm_extra_pre_umount(), remove the resolv.conf before
the disk image is unmounted from the backing md(4).

Reported by: cperciva
Sponsored by: The FreeBSD Foundation

Set the boot partition type to 'apple-boot' for powerpc.

Submitted by: jhibbits
Sponsored by: The FreeBSD Foundation

mount(8) and umount(8) devfs(5) as needed.

Sponsored by: The FreeBSD Foundation

Fix scheme flag to mkimg(1).

Sponsored by: The FreeBSD Foundation

Fix line continuation in write_partition_layout().
Remove variable test that is no longer needed.

Sponsored by: The FreeBSD Foundation

Move usage() from vmimage.subr to mk-vmimage.sh,
in case vmimage.subr has not been sourced.

Sponsored by: The FreeBSD Foundation

Add vm_extra_pre_umount() prototype to vmimage.subr.

Sponsored by: The FreeBSD Foundation

Add write_partition_layout() used to populate the
final image.

Fix duplicated mkimg(1) call in vm_create_disk().

Add primitive (untested) PowerPC/PowerPC64 VM image
support.

Note: As it is currently written, the /boot/pmbr and
/boot/{gptboot,boot1.hfs} use the build host and not
the target build. Fixing this is likely going to be
a hack in itself.

Sponsored by: The FreeBSD Foundation

Initial rewrite to consolidate VM image build scripts into one.
There may be some very sharp edges here while refactoring.

- Move amd64/mk-vmimage.sh -> scripts/mk-vmimage.sh.
- Remove vm-base target from Makefile.vm.
- In vm-image target, use getopts flags for argument passing.
- Create tools/vmimage.subr, containing default and prototype
for the following functions that are used to drive the build,
run in this order:

vm_install_base()
vm_extra_install_base()
vm_extra_install_packages()
vm_extra_install_ports()
vm_extra_enable_services()
vm_extra_pre_umount()
vm_create_disk()
vm_extra_create_disk()

- In tools/azure.conf, override:

vm_extra_install_base()
vm_extra_pre_umount()
vm_extra_create_disk()

- In tools/openstack.conf, override:

vm_extra_install_base()
vm_extra_pre_umount()

Sponsored by: The FreeBSD Foundation