bhyvectl: Prepare to add arm64 support

Move MD code into a separate directory and add a simple interface which
lets the MD bits register options and handle them.

No functional change intended.

Reviewed by: jhb
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D44932

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

bhyvectl: Address compiler warnings and bump WARNS

Avoid unaligned accesses in cpu_vendor_intel() and address a few other
nits. No functional change intended.

Reviewed by: corvink, rew, jhb
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D38839

bhyve/snapshot: switch to nvlist for snapshot requests

Switch to using an nvlist with nvlist_send()/nvlist_recv() to
communicate from bhyvectl(8) to bhyve(8).

The idea is that a bhyve process receives a command with with a set of
arguments. The nvlist here is structured to reflect that premise.

For example, to snapshot the vm, the expected nvlist looks like:

{ cmd=START_CHECKPOINT, filename="filename" }

Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D33977

libvmm: clean up vmmapi.h

struct checkpoint_op, enum checkpoint_opcodes, and
MAX_SNAPSHOT_VMNAME are not vmm specific, move them out of the vmmapi
header.

They are used for the save/restore functionality that bhyve(8)
provides and are better suited in usr.sbin/bhyve/snapshot.h

Since bhyvectl(8) requires these, the Makefile for bhyvectl has been
modified to include usr.sbin/bhyve/snapshot.h

Reviewed by: kevans, grehan
Differential Revision: https://reviews.freebsd.org/D28410

Initial support for bhyve save and restore.

Save and restore (also known as suspend and resume) permits a snapshot
to be taken of a guest's state that can later be resumed. In the
current implementation, bhyve(8) creates a UNIX domain socket that is
used by bhyvectl(8) to send a request to save a snapshot (and
optionally exit after the snapshot has been taken). A snapshot
currently consists of two files: the first holds a copy of guest RAM,
and the second file holds other guest state such as vCPU register
values and device model state.

To resume a guest, bhyve(8) must be started with a matching pair of
command line arguments to instantiate the same set of device models as
well as a pointer to the saved snapshot.

While the current implementation is useful for several uses cases, it
has a few limitations. The file format for saving the guest state is
tied to the ABI of internal bhyve structures and is not
self-describing (in that it does not communicate the set of device
models present in the system). In addition, the state saved for some
device models closely matches the internal data structures which might
prove a challenge for compatibility of snapshot files across a range
of bhyve versions. The file format also does not currently support
versioning of individual chunks of state. As a result, the current
file format is not a fixed binary format and future revisions to save
and restore will break binary compatiblity of snapshot files. The
goal is to move to a more flexible format that adds versioning,
etc. and at that point to commit to providing a reasonable level of
compatibility. As a result, the current implementation is not enabled
by default. It can be enabled via the WITH_BHYVE_SNAPSHOT=yes option
for userland builds, and the kernel option BHYVE_SHAPSHOT.

Submitted by: Mihai Tiganus, Flavius Anton, Darius Mihai
Submitted by: Elena Mihailescu, Mihai Carabas, Sergiu Weisz
Relnotes: yes
Sponsored by: University Politehnica of Bucharest
Sponsored by: Matthew Grooms (student scholarships)
Sponsored by: iXsystems
Differential Revision: https://reviews.freebsd.org/D19495

usr.sbin: normalize paths using SRCTOP-relative paths or :H when possible

This simplifies make logic/output

MFC after: 1 month
Sponsored by: Dell EMC Isilon

Add a basic bhyvectl manpage.

Reviewed by: neel
MFC after: 2 weeks

Restructure memory allocation in bhyve to support "devmem".

devmem is used to represent MMIO devices like the boot ROM or a VESA framebuffer
where doing a trap-and-emulate for every access is impractical. devmem is a
hybrid of system memory (sysmem) and emulated device models.

devmem is mapped in the guest address space via nested page tables similar
to sysmem. However the address range where devmem is mapped may be changed
by the guest at runtime (e.g. by reprogramming a PCI BAR). Also devmem is
usually mapped RO or RW as compared to RWX mappings for sysmem.

Each devmem segment is named (e.g. "bootrom") and this name is used to
create a device node for the devmem segment (e.g. /dev/vmm/testvm.bootrom).
The device node supports mmap(2) and this decouples the host mapping of
devmem from its mapping in the guest address space (which can change).

Reviewed by: tychon
Discussed with: grehan
Differential Revision: https://reviews.freebsd.org/D2762
MFC after: 4 weeks

bhyvectl does not need to link to libutil

Make FreeBSD-bhyve an indivual package

Convert usr.sbin to LIBADD
Reduce overlinking

NO_MAN= has been deprecated in favor of MAN= for some time, go ahead
and finish the job. ncurses is now the only Makefile in the tree that
uses it since it wasn't a simple mechanical change, and will be
addressed in a future commit.

Parse the memory size parameter using expand_number() to allow specifying
the memory size more intuitively (e.g. 512M, 4G etc).

Submitted by: rodrigc
Reviewed by: grehan
Approved by: re (blanket)

Rename vmmctl to bhyvectl. 'vmmctl' came from a pre-bhyve
internal codebase at NetApp. No need for it to have an
unrelated name to the other userspace utils.

Reviewed by: neel
Obtained from: NetApp