release: Rework vm_extra_pre_umount

The vm_extra_pre_umount function in vmimage.subr served two purposes:
It removed /etc/resolv.conf and /qemu (if cross-building), and it
provided a function for cloudware to override in order to make cloud
specific changes to the filesystem before constructing a disk image.

This resulted in a number of bugs:
1. When cross-building, the emulator binary was left as /qemu in the
Azure, GCE, Openstack and Vagrant images.
2. The build host's resolv.conf was left as /etc/resolv.conf in the
basic-ci and basic-cloudinit images.
3. When building GCE images, a Google-specific resolv.conf file was
constructed, and then deleted before the disk image was created.

Move the bits needed for running code inside a VM staging directory
from vm_install_base into a new vm_emulation_setup routine, and move
the corresponding cleanup bits from vm_extra_pre_umount to a new
vm_emulation_cleanup routine.

Remove the /qemu and /etc/resolv.conf cleanups from the cloudware
configuration files (where they exist) since we will now be running
vm_emulation_cleanup to remove those even when vm_extra_pre_umount
has been overridden.

Override vm_emulation_cleanup in gce.conf since in that one case (and
*only* that one case) we don't want to clean up resolv.conf (since it
was constructed for the VM image rather than copied from the host).

releng/14.1 candidate.

MFC after: 1 week
Sponsored by: https://www.patreon.com/cperciva

EC2: Bump AMI size to 6 GB

This week's cloud-init ZFS snapshots ran out of disk space.

EC2: Split off reusable configuration bits

Split ec2-base.conf into ec2-base.conf and a reusable ec2.conf,
similar to how Vagrant flavours share a common vagrant.conf.

releng/14.0 candidate.

Discussed with: gjb
MFC after: 5 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D41792

release/ec2: Remove references to portsnap

We no longer install a default portsnap.conf, so the sed invocation just
generates an error.

Reviewed by: cperciva
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D42003

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

EC2: Chase awscli package rename

MFC after: 1 week

release: Add support for creating ZFS-based VM images

The change extends vmimage.subr to handle a new parameter, VMFS, which
should be equal to either "ufs" or "zfs". When it is set to ZFS, we use
makefs to create a bootable pool populated using the same dataset layout
as bsdinstall and "poudriere image" use. The pool can be grown using
the growfs rc.d script, just as in UFS images.

This will make it easy to provide VM and cloud images with ZFS as the
root filesystem. So far I did not do extensive testing of cloud images;
I merely verified that creation of ZFS-based AWS AMIs works and allows
me to create amd64 and arm64 EC2 instances with ZFS as the root
filesystem.

Reviewed by: emaste, gjb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34426

EC2: Enable ephemeral swap by default

The latest version of the ec2-scripts package includes a completely
rewritten "use EC2 ephemeral disks for swap space" script. Now that
we have something which works on recent versions of FreeBSD, turn it
on since it's a great way to use the ephemeral disks.

Note that the option for controlling this, ec2_ephemeral_swap_enable,
is not the same as the option (ec2_ephemeralswap_enable) used with the
previous ephemeral-swap script; this change was deliberate to avoid
astonishment for users who upgraded their ec2-scripts package and had
a setting left behind in rc.conf.

release: Remove references to ChallengeResponseAuthentication

This sshd_config keyword was replaced by KbdInteractiveAuthentication in
openssh 8.7, though ChallengeResponseAuthentication is silently accepted
as an alias. However, this means that the code in ec2.conf which
modifies a commented-out line no longer does anything. Apply a minimal
fix.

Reviewed by: cperciva, emaste
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D34400

EC2: Turn off IPv6 DAD

Disable Duplicate Address Detection in EC2 instances. The networking
configuration in EC2, with IPv6 addresses assigned by DHCPv6 and
host egress filtering, makes "duplicate addresses" impossible.

This speeds up the boot process in EC2 by 2 seconds.

Reviewed by: kp, imp, bz
MFC after: 2 weeks
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D33091

release: allow VM_EXTRA_PACKAGES to be specified in the environment

This is useful for adding extra packages to the build of an AMI.
For example:
env VM_EXTRA_PACKAGES="zsh" make -C release ec2ami

Approved by: gjb
MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")

EC2: Tell gptboot to skip its 3 second wait

Nobody is going to hit a key to tell gptboot to select another disk,
so there's no point waiting.

release: fix VMSIZE following 1ca8842f3ad9

truncate(1) is not case-sensitive with regard to setting the size
of a file. makefs(8), however, does not honor upper-case values.
Update release-specific files and the release(7) manual page to
reflect this.

MFC with: 1ca8842f3ad9
Submitted by: ehem_freebsd_m5p.com (original)
Differential Review: https://reviews.freebsd.org/D28979
Sponsored by: Rubicon Communications, LLC ("Netgate")

Spawn the DHCPv6 client in EC2 instances via rtsold.

Prior to this commit, EC2 AMIs used a "dual-dhclient" tool which was
launched in place of dhclient and spawned both the base system dhclient
for IPv4 and the ISC dhclient from ports for IPv6.

Now that rtsold supports the "M bit" (managed configuration), we can go
back to having the base system dhclient spawned normally, and provide a
script to rtsold which spawns the ISC dhclient from ports when rtsold
decides that it is appropriate.

Thanks to: bz
MFC after: 1 week
Sponsored by: https://www.patreon.com/cperciva

Bump the size of EC2 AMIs up to 5 GB.

The FreeBSD base system continues to expand. 4GB is now insufficient;
we passed 3 GB in May 2019; we passed 2 GB in August 2017. Over half
of the disk space used is in /usr/lib/debug/.

Without this change, instances boot but are unusable, since the first
thing which breaks when VM filesystems are too small is the "pkg install"
in the VM building process.

Make EC2 AMIs use portsnap and freebsd-update mirrors hosted in AWS

This adjusts freebsd-update.conf and portsnap.conf files in EC2 AMIs to
point at the new AWS-hosted mirror network.

Approved by: re (delphij)
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D25498

Add ebsnvme-id to EC2 AMIs and enable /dev/aws/disk

The ebsnvme-id utility exposes information about EC2 disks -- for
Elastic Block Store volumes, their volume IDs and "linux device
names", and for Instance Store (aka "Ephemeral") disks, their
serial numbers.

The dev_aws_disk rc.d script and associated devd.conf rule maintains
a tree under /dev/aws/disk:
/dev/aws/disk/ebs/<volume ID>
/dev/aws/disk/linuxname/<linux device name>
/dev/aws/disk/ephemeral/<serial number>
which are symlinks to the corresponding nda or nvd devices.

MFC after: 1 week
Sponsored by: https://www.patreon.com/cperciva

Add /etc/autofs/special_efs to EC2 AMIs

Since Amazon Elastic File System is only available within AWS, it seems
more appropriate to have this added only in EC2 AMIs rather than
"polluting" non-EC2 images with it.

Reviewed by: gjb
MFC after: 7 days
Relnotes: Amazon EFS filesystems can be automounted by enabling autofs
and placing "/efs -efs" into /etc/auto_master.
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D24791

Set use_nvd=0 in EC2 AMIs.

FreeBSD is in the process of switching from nvd(4) to nda(4) as the disk
device front-end to NVMe. Changing the default in the kernel is tricky
since existing systems may have /dev/nvd* hard-coded e.g. in /etc/fstab;
however, there's no reason to not change the default in HEAD for *new*
systems.

At present I have no intention of MFCing this to stable branches, since
someone might reasonably expect scripts they use for launching and
configuring FreeBSD 12.1 instances to work with FreeBSD 12.2 AMIs, for
example.

Reviewed by: gjb, imp
Relnotes: NVMe disks in EC2 instances launched from 13.0 and later
now show up as nda(4) devices.
Differential Revision: https://reviews.freebsd.org/D24583

Mount /dev while deinstalling pkg inside EC2 AMIs.

This gets rid of (harmless) warnings:
"pkg: Cannot open /dev/null:No such file or directory".

MFC after: 2 weeks

Remove /qemu from EC2 ARM AMIs

I forgot to do this as part of r345858 -- I added it to the
vm_extra_pre_umount in vmimage.subr but forgot that function
was overridden in the EC2 build.

MFC after: 2 weeks

Switch EC2 AMIs from using the dual-dhclient script to using the new
dual-dhclient-daemon daemon. This makes it possible to stop/restart
the dhclients.

MFC after: 1 month

Bump EC2 AMI filesystem size up to 4000 MB.

AMIs have been breaking for the past month due to insufficient disk space.

Due to the small amount of overhead in the disk image, the EC2 AMIs end
up with the same (4GB) minimum disk size.

Reported by: Michal Krawczyk

Add support for cross-building cloudware images.

If MACHINE_ARCH doesn't match TARGET_ARCH, and we're not in the special
case of building i386 images on an amd64 host, we need to pull in the
qemu-user-static package; this allows us to run some commands inside
the VM disk image chroot, most notably to install packages.

Reviewed by: gjb
MFC after: 2 weeks
Sponsored by: FreeBSD/EC2 patreon (https://www.patreon.com/cperciva)

Only install amazon-ssm-agent into amd64 AMIs.

This package does not exist on aarch64 at present.

Fix sed script to insert Amazon NTP server into ntp.conf once rather
than twice.

Reported by: Rafal Lukawiecki
MFC after: 1 week

Turn off ec2_ephemeralswap for now

This script broke around FreeBSD 11.0 as a result of SWAPMETA no longer
being reported by vmstat -z; but it also needs to be reworked due to the
arrival in EC2 of nvme ephemeral disks.

I'll turn this option back on after I've found time to rewrite the
script in question.

PR: 234686
Reported by: meta@
MFC after: 1 week

Disable atkbd0 and atkdbc0 in EC2 AMIs. This has the effect of skipping
the probing and attaching of the PS/2 mouse (not present on EC2) and
keyboard (emulated, but not accessible via EC2).

Note that we disable atkbd0 separately even though during device probing
it shows up as a child of atkbdc0; this is necessary because the device
is also initialized during the early console setup from hammer_time.

This change cuts the kernel boot time on an EC2 c5.4xlarge instance from
7259ms down to 4727 ms.

Approved by: re (marius)

Move debug.{trace,debugger}_on_panic and kern.panic_reboot_wait_time in
EC2 instances from sysctl.conf to loader.conf; these can all be set as
loader tunables, and setting them in loader.conf gives us the right
behaviour in the event of a kernel panic taking place prior to when
sysctl.conf is processed.

MFC after: 1 week

Make EC2 instances use Amazon's NTP service for time synchronization.

Since Amazon provides NTP servers within their network, this should
be far superior to using the default NTP pools; and since the service
is provided by Amazon there's very little risk in enabling it by
default. (If someone is able to compromise Amazon's NTP servers and
exploit them to attack EC2 instances, they would almost certainly be
able to compromise EC2 instances even without ntpd running...)

MFC after: 1 week
Relnotes: EC2 instances now keep their clocks synchronized using
the Amazon Time Sync Service (aka. NTP).

Resurrect r321659: Turn off ChallengeResponseAuthentication for EC2 AMIs.

EC2 instances are normally launched with an SSH public key specified,
which is then used for logging in (by default, as 'ec2-user'). Having
ChallengeResponseAuthentication enabled (as FreeBSD's default sshd_config
does) has no functional effect in a new EC2 instance, since you can't log
in using a password until a password has been set -- but having this
enabled results in alerts from automated scanning tools which can detect
that sshd advertises support for keyboard-interactive logins (since they
can't detect that accounts have no password set).

EC2 users who want to use passwords to log in to their instances will need
to set 'ChallengeResponseAuthentication yes' in FreeBSD 12.0 and later.

Discussed with: gjb, gtetlow, emaste, des
Requested by: Amazon
X-MFC: No
Relnotes: ChallengeResponseAuthentication is turned off by default in
Amazon EC2 AMIs.

Remove /etc/resolv.conf from virtual machine images, which is
copied from the build host. It is renamed to /etc/resolv.conf.bak
on boot, so never used anyway.

Noticed by: peter
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Add the amazon-ssm-agent package to EC2 AMI builds. This makes it
immediately available on instances which are running without internet
access (or which can't rely on firstboot_pkgs to install it for some
other reason).

Note that this agent is not enabled by default; to enable it, add
amazon_ssm_agent_enable="YES" to /etc/rc.conf, e.g., by placing the lines
>>/etc/rc.conf
amazon_ssm_agent_enable="YES"
into the EC2 user-data. In addition to being enabled, the agent requires
keys to be provided via IAM Roles; users are encouraged to be very careful
in using this functionality due to the inherent vulnerability in the idea
of providing credentials via a service accessible to any process which can
open an HTTP connection.

Requested by: Amazon
No objection from: re@
Relnotes: FreeBSD/EC2 AMIs now include the Amazon EC2 Systems Manager
(SSM) Agent.

Increase the Amazon EC2 AMI image size from 2GB to 3GB to prevent
image build failures due to a full md(4)-backed filesystem.

Sponsored by: The FreeBSD Foundation

Revert r321659, re-enabling ChallengeResponseAuthentication, which was
discussed a while back between cperciva@ and so@, and I forgot.

Reported by: cperciva
Sponsored by: The FreeBSD Foundation

Turn off ChallengeResponseAuthentication for EC2 AMIs, one of EC2's
requirements.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Turn on support for the Amazon "Elastic Network Adapter" in EC2 AMIs.

X-MFC-after: 318647 + fixes for some lock ordering warnings

ec2.conf and vmimage.subr can be used from the installation livecd after
install to prepare an AMI image. This can be used to create a ZFS AMI disk
image using a virtual machine.

Change ec2.conf to use the pkg tool from a chroot rather than trying to
bootstrap it and fail from the livecd readonly filesystem.

Reviewed by: gjb

Increase the EC2 image size for 12-CURRENT. The recent snapshot
builds of EC2 images for 12-CURRENT failed due to a full filesystem
on the md(4) device during creation.

Sponsored by: The FreeBSD Foundation

Enable IPv6 networking on Amazon EC2.

MFC after: 1 week

Now that potentially buggy versions of Xen are automatically detected
(see r302635), there is no need to force msix interrupt migration off
via loader.conf.

Reverts: r302184
MFC after: 3 days

Enable indirect segment I/Os by default in the Xen blkfront driver when
running on EC2. Due to improvements in EC2, the performance penalty which
was present on some EC2 instances no longer exists, and enabling this
feature now consistently yields ~20% higher throughput with equal or lower
latency.

Reverts: r286063
Approved by: re (gjb)
MFC after: 2 weeks
Relnotes: Improved disk throughput on EC2

Turn off MSI-X interrupt migration by default in EC2 instances; this works
around a bug in older versions of Xen and unbreaks SR-IOV (aka. "EC2
Enhanced Networking").

Approved by: re (gjb)
Thanks to: jhb, Jeremiah Lott

Switch from console="comconsole" to boot_multicons="YES" in EC2.

Amazon recently introduced an API for capturing screenshots of an
emulated VGA device; this commit makes that (somewhat) useful.

MFC after: 3 weeks

Disable blkif indirect segment I/Os in EC2 by default due to performance
issues on some EC2 instance types. Users may want to experiment with
removing this from loader.conf and measuring the performance impact on
the EC2 instances they are using.

Ensure ASSUME_ALWAYS_YES is set when deleting pkg(8) from the
DESTDIR, otherwise it can get stuck in ttyin.

This means no 10-STABLE EC2 images this week.

MFC after: 3 days
Sponsored by: The FreeBSD Foundation

It would help if I committed the right patch...

Add bits for building EC2 disk images. Make logic for uploading these
to create EC2 AMIs will come in a later commit.