MFC r336420,336433,336593,336621,336622,336624,337394,337401,338141

Teach the EC2 AMI upload code to:
* Mark EBS snapshots as public,
* Announce new AMIs to an SNS topic, and
* Place the SVN branch and revision into the AMI "description".

MFC r284893, r284895-r284897, r284942, r284968, r284996, r285005:

r284893 (brd):
Add initial support for building Vagrant images for VMWare.
Next steps will be adding Virtualbox support and uploading
to Hashicorp Atlas for others to consume.

r284895:
Add default VAGRANT_IMG variable.

r284896:
Remove _ACCOUNT and add _USERNAME, _NAME, _VERSION for the
VAGRANT_${VAR} variables extracted from VAGRANT_UPLOAD_CONF.

Set ATLAS_${VAR} to VAGRANT_${VAR} if VAGRANT_UPLOAD_CONF
is set. There is intent to intentionally have separate
variants of configuration entries, but the defaults do not
yet have any reason to be different.

r284897:
Instead of hard-coding the PROVIDERS for upload, add the
VAGRANT_PROVIDERS variable. Right now, it defaults to only
vmware_desktop, virtualbox support is to follow at some point.

While here, fix the hashicorp URL: s/vagrant/atlas/, which
was result of a sed(1) replace (and my fault).

r284942 (brd):
Add Support for uploading Vagrant images to Hashicorp Atlas.

r284968:
Default the VAGRANT_VERSION to ${REVISION}-${BRANCH} if
not set, which expands to '11.0-CURRENT', for example.

If the branch is -CURRENT, -STABLE, or -PRERELEASE, suffix
the VAGRANT_VERSION with the snapshot date.

r284996:
Fix the gcloud port/package name.

r285005:
Remove the HH-MM suffix from the build date suffix.

It was useful when working out several kinks when testing
automated image uploading when retrying was necessary, but
now it is making things much too messy.

Sponsored by: The FreeBSD Foundation

MFC r284780, r284811, r284812:

r284780:
Ensure ASSUME_ALWAYS_YES is set when deleting pkg(8) from the
DESTDIR, otherwise it can get stuck in ttyin.

r284811:
Consider PRERELEASE builds snapshots, in addition to CURRENT
and STABLE.

r284812:
Change the hour:minute delimiter from ':' to '-', since
the former is an invalid character in EC2 images.

Approved by: re (glebius, insta-MFC)
Sponsored by: The FreeBSD Foundation

MFC r284660, r284676:

r284660:
Remove the Azure-local vm_extra_create_disk(), since we no longer
need qemu-img to convert the final VHD image to an Azure-compatible
format.

Although the waagent utility is installed from ports, create the
symlink to /usr/sbin, pending investigation on where this is
hard-coded, so it can be reported upstream.

r284676:
Append the hour and minute to the snapshot suffix for EC2
AMIs and Azure VM images. This is particularly helpful for
testing to avoid name collisions, but also useful for cases
where a necessary rebuild is done before the date changes.

Sponsored by: The FreeBSD Foundation

MFC r283919:
Remove trailing tabs to avoid spamming stdout with long lines.

Sponsored by: The FreeBSD Foundation

MFC r283252:
Include the TARGET in the EC2 AMI name.

Without this, AWS rejects subsequent image uploads of a different
architecture because the name conflicts.

Sponsored by: The FreeBSD Foundation

MFC r282419, r282435:
r282419:
Add logic to detect if the net/bsdec2-image-upload port needs
to be installed. [1]

For the cw-ec2-portinstall and ec2ami targets, touch the
.TARGET file after completion to prevent duplicate invocations.

Add cw-ec2-portinstall and ec2ami to CLEANFILES.

r282435:
Remove a debugging line that snuck in with the previous commit.

Sponsored by: The FreeBSD Foundation

MFC r282116:
Add the cw-ec2-portinstall target to always install the
net/bsdec2-image-upload port.

Sponsored by: The FreeBSD Foundation

MFC r280299, r280840, r280846, r280879, r280881, r280884, r280928,
r281783, r281809, r281832:

r280299 (cperciva):
When creating VM images, copy the contents of the created filesystem
into a new filesystem before packaging it into a disk image. This
prevents "remnants" of deleted files from showing up in the VM images,
and reduces their compressed size (by about 10% for the cloudware
images) as a result.

r280840 (cperciva):
Clean up filesystem unmounting in vmimage builds:
- Remove vm_umount_base function which is currently unused.
- Add umount_loop function which loops attempting to unmount one
filesystem.
- Replace calls to umount with calls to umount_loop.
- Don't attempt to unmount ${DESTDIR}/dev if it isn't mounted.

The looping is necessary because sometimes umount fails due to
filesystems being busy. The most common cause of such busyness is
periodic(8) jobs running `find / ...`.

r280846 (cperciva):
Improve check for whether ${DESTDIR}/dev is mounted.

r280879 (cperciva):
Add bits for building EC2 disk images.

r280881 (cperciva):
Partially revert r278118 now that the required logic for deciding
whether freebsd-update can be useful has moved into the
firstboot_freebsd_update script.

r280884 (cperciva):
Fix pkg(8) command line in vm_extra_pre_umount().

r280928 (cperciva):
Add code for creating an EC2 AMI.

r281783:
When building VM disk images, vm_copy_base() uses tar(1) to copy
the userland from one md(4)-mounted filesystem to a clean filesystem
to prevent remnants of files that were added and removed from
resulting in an unclean filesystem. When newfs(8) creates the first
filesystem with journaled soft-updates enabled, the /.sujournal file
in the new filesystem cannot be overwritten by the /.sujournal in the
original filesystem.

To avoid this particular error case, do not enable journaled
soft-updates when creating the md(4)-backed filesystems, and instead
use tunefs(8) to enable journaled soft-updates after the new
filesystem is populated in vm_copy_base().

While here, fix a long standing bug where the build environment /boot
files were used by mkimg(1) when creating the VM disk images by using
the files in .OBJDIR.

r281809:
Simplify variable expansion in attempt to fix the vm-image build.

r281832:
Fix locating the /boot files.

Sponsored by: The FreeBSD Foundation

MFC r284893, r284895-r284897, r284942, r284968, r284996, r285005:

r284893 (brd):
Add initial support for building Vagrant images for VMWare.
Next steps will be adding Virtualbox support and uploading
to Hashicorp Atlas for others to consume.

r284895:
Add default VAGRANT_IMG variable.

r284896:
Remove _ACCOUNT and add _USERNAME, _NAME, _VERSION for the
VAGRANT_${VAR} variables extracted from VAGRANT_UPLOAD_CONF.

Set ATLAS_${VAR} to VAGRANT_${VAR} if VAGRANT_UPLOAD_CONF
is set. There is intent to intentionally have separate
variants of configuration entries, but the defaults do not
yet have any reason to be different.

r284897:
Instead of hard-coding the PROVIDERS for upload, add the
VAGRANT_PROVIDERS variable. Right now, it defaults to only
vmware_desktop, virtualbox support is to follow at some point.

While here, fix the hashicorp URL: s/vagrant/atlas/, which
was result of a sed(1) replace (and my fault).

r284942 (brd):
Add Support for uploading Vagrant images to Hashicorp Atlas.

r284968:
Default the VAGRANT_VERSION to ${REVISION}-${BRANCH} if
not set, which expands to '11.0-CURRENT', for example.

If the branch is -CURRENT, -STABLE, or -PRERELEASE, suffix
the VAGRANT_VERSION with the snapshot date.

r284996:
Fix the gcloud port/package name.

r285005:
Remove the HH-MM suffix from the build date suffix.

It was useful when working out several kinks when testing
automated image uploading when retrying was necessary, but
now it is making things much too messy.

Sponsored by: The FreeBSD Foundation

MFC r284780, r284811, r284812:

r284780:
Ensure ASSUME_ALWAYS_YES is set when deleting pkg(8) from the
DESTDIR, otherwise it can get stuck in ttyin.

r284811:
Consider PRERELEASE builds snapshots, in addition to CURRENT
and STABLE.

r284812:
Change the hour:minute delimiter from ':' to '-', since
the former is an invalid character in EC2 images.

Approved by: re (glebius, insta-MFC)
Sponsored by: The FreeBSD Foundation

MFC r284660, r284676:

r284660:
Remove the Azure-local vm_extra_create_disk(), since we no longer
need qemu-img to convert the final VHD image to an Azure-compatible
format.

Although the waagent utility is installed from ports, create the
symlink to /usr/sbin, pending investigation on where this is
hard-coded, so it can be reported upstream.

r284676:
Append the hour and minute to the snapshot suffix for EC2
AMIs and Azure VM images. This is particularly helpful for
testing to avoid name collisions, but also useful for cases
where a necessary rebuild is done before the date changes.

Sponsored by: The FreeBSD Foundation

MFC r283919:
Remove trailing tabs to avoid spamming stdout with long lines.

Sponsored by: The FreeBSD Foundation

MFC r283252:
Include the TARGET in the EC2 AMI name.

Without this, AWS rejects subsequent image uploads of a different
architecture because the name conflicts.

Sponsored by: The FreeBSD Foundation

MFC r282419, r282435:
r282419:
Add logic to detect if the net/bsdec2-image-upload port needs
to be installed. [1]

For the cw-ec2-portinstall and ec2ami targets, touch the
.TARGET file after completion to prevent duplicate invocations.

Add cw-ec2-portinstall and ec2ami to CLEANFILES.

r282435:
Remove a debugging line that snuck in with the previous commit.

Sponsored by: The FreeBSD Foundation

MFC r282116:
Add the cw-ec2-portinstall target to always install the
net/bsdec2-image-upload port.

Sponsored by: The FreeBSD Foundation

MFC r280299, r280840, r280846, r280879, r280881, r280884, r280928,
r281783, r281809, r281832:

r280299 (cperciva):
When creating VM images, copy the contents of the created filesystem
into a new filesystem before packaging it into a disk image. This
prevents "remnants" of deleted files from showing up in the VM images,
and reduces their compressed size (by about 10% for the cloudware
images) as a result.

r280840 (cperciva):
Clean up filesystem unmounting in vmimage builds:
- Remove vm_umount_base function which is currently unused.
- Add umount_loop function which loops attempting to unmount one
filesystem.
- Replace calls to umount with calls to umount_loop.
- Don't attempt to unmount ${DESTDIR}/dev if it isn't mounted.

The looping is necessary because sometimes umount fails due to
filesystems being busy. The most common cause of such busyness is
periodic(8) jobs running `find / ...`.

r280846 (cperciva):
Improve check for whether ${DESTDIR}/dev is mounted.

r280879 (cperciva):
Add bits for building EC2 disk images.

r280881 (cperciva):
Partially revert r278118 now that the required logic for deciding
whether freebsd-update can be useful has moved into the
firstboot_freebsd_update script.

r280884 (cperciva):
Fix pkg(8) command line in vm_extra_pre_umount().

r280928 (cperciva):
Add code for creating an EC2 AMI.

r281783:
When building VM disk images, vm_copy_base() uses tar(1) to copy
the userland from one md(4)-mounted filesystem to a clean filesystem
to prevent remnants of files that were added and removed from
resulting in an unclean filesystem. When newfs(8) creates the first
filesystem with journaled soft-updates enabled, the /.sujournal file
in the new filesystem cannot be overwritten by the /.sujournal in the
original filesystem.

To avoid this particular error case, do not enable journaled
soft-updates when creating the md(4)-backed filesystems, and instead
use tunefs(8) to enable journaled soft-updates after the new
filesystem is populated in vm_copy_base().

While here, fix a long standing bug where the build environment /boot
files were used by mkimg(1) when creating the VM disk images by using
the files in .OBJDIR.

r281809:
Simplify variable expansion in attempt to fix the vm-image build.

r281832:
Fix locating the /boot files.

Sponsored by: The FreeBSD Foundation