ctl: Support NVMe requests in debug trace functions

Reviewed by: imp
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D44719

bhyve: Support legacy PCI interrupts on arm64

This allows us to remove various #ifdef hacks and enable building more
PCI devices.

Note that a hole is left in the interrupt mapping for the RTC rather
than having the two core devices straddle the PCIe interrupts. QEMU's
virt machine also takes this approach.

Reviewed by: jhb
MFC after: 2 weeks
Obtained from: CheriBSD

bhyve: Do not compile PCI passthrough support on arm64

Some required kernel functionality is not yet implemented.

For now this means that one cannot specify host PCI register values, but
that functionality is only used by amd64-specific device models for now.
Note that this limitation is rather artificial; it arises only because
pci_host_read_config() lives in pci_passthru.c.

Reviewed by: corvink, andrew, jhb
MFC after: 2 weeks
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D41738

bhyve: Provide optional libfdt linking

The arm64 port currently does not support ACPI, it instead builds up an
FDT which is exported to the guest. This mechanism will not be used on
amd64 but isn't really arm64-specific either, so provide an opt-in
mechanism to link libfdt.

No functional change intended.

Reviewed by: corvink, jhb
MFC after: 2 weeks
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40995

bhyve: Move device model-independent UART code into a separate file

Currently bhyve implements a ns16550-compatible UART in uart_emul.c.
This file also contains generic code to manage RX FIFOs and to handle
reading from and writing to a TTY. bhyve instantiates UARTs to
implement COM devices (via pci_lpc.c) and PCI UART devices.

The arm64 port will bring with it a PL011 device model which is used as
the default console (i.e., no COM ports). To simplify its integration,
add a UART "backend" layer which lets UART device models allocate an RX
FIFO and interact with TTYs without duplicating code. In particular,
code in uart_backend.* is to be shared among device models, and the
namespace for uart_emul.* is changed to uart_ns16550_*.

This is based on andrew@'s work in
https://github.com/zxombie/freebsd/tree/bhyvearm64 but I've made a
number of changes, particularly with respect to naming and source code
organization.

No functional change intended.

Reviewed by: corvink, jhb
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40993

bhyve: Add support for XML register definitions

This is useful for exposing additional registers to debuggers. For
instance, control registers are now available on amd64 when using gdb to
debug a guest.

The stub indicates support by including the string
"qXfer:features:read+" in its feature list. The debugger queries for
target descriptions by sending the query "qXfer:features:read:" followed
by a file path.

The XML definitions are copied from QEMU and installed to
/usr/share/bhyve/gdb.

Note that we currently don't handle the SIMD registers at all, since
that's of somewhat limited utility (for me at least) and since that
requires new ioctls to fetch the register values.

Reviewed by: jhb
MFC after: 2 weeks
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D43666

bhyve: Add a slirp network backend

This enables a subset of the functionality provided by QEMU's user
networking implementation. In particular, it uses net/libslirp, the
same library as QEMU.

libslirp is permissively licensed but has some dependencies which make
it impractical to bring into the base system (glib in particular). I
thus opted to make bhyve dlopen the libslirp.so, which can be installed
via pkg. The library header is imported into bhyve.

The slirp backend takes a "hostfwd" which is identical to QEMU's
hostfwd. When configured, bhyve opens a host socket and listens for
connections, which get forwarded to the guest. For instance,
"hostfwd=tcp::1234-:22" allows one to ssh into the guest by ssh'ing to
port 1234 on the host, e.g., via 127.0.0.1. I didn't try to hook up
guestfwd support since I don't personally have a use-case for it yet,
and I think it won't interact nicely with the capsicum sandbox.

Reviewed by: jhb
Tested by: rew
MFC after: 1 month
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D42510

bhyve: Split backends into separate files

Currently the net_backend structure definition is private to
net_backends.c, so all of the backend definitions are there. While
adding a new backend to use libslirp, it was noted that this file is
somewhat cluttered. Move the netmap and netgraph backends to their own
files and clean up includes a bit. No functional change intended.

Reviewed by: corvink, jhb
MFC after: 3 weeks
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D42689

bhyve: Move vcpu initialization into a MD source file

- Make handling of x86 config options, like x86.x2apic, conditional to
amd64.
- Move fbsdrun_set_capabilities() and spinup_vcpu() to a new file,
bhyverun_machdep.c. The moved code is all highly x86 specific.

I'm not sure how best to handle the namespace. I'm using "bhyve_" for
MD functions called from MI code. We also have "fbsdrun_" for some MI
routines that are typically called from MD code. The file name is
prefixed by "bhyverun_".

Reviewed by: corvink
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40987

bhyve: Make gdb support optional

Add a BHYVE_GDB_SUPPORT make variable that can be set by per-arch
makefiles. When set, BHYVE_GDB is defined and can be used as a
preprocessor predicate. Use it to guard gdb stub calls in MI code.

The arm64 bhyve port currently does not have a functional gdb stub, but
that's not critical to landing the port, so this mechanism slightly
reduces the friction of adding support for a new platform.

Reviewed by: corvink, jhb
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40986

bhyve: Make most I/O port handling specific to amd64

- The qemu_fwcfg interface, as implemented, is I/O port-based, but QEMU
implements an MMIO interface that we'll eventually want to port for
arm64.
- Retain support for I/O space PCI BARs, simply treat them like MMIO
BARs for most purposes, similar to what the arm64 kernel does. Such
BARs are created by virtio devices.

Reviewed by: corvink, jhb
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40741

bhyve: Conditionally compile framebuffer-related code

The arm64 port does not implement VGA, so move the device model sources.
Compile framebuffer code only on amd64 for now, but do not move the
sources, as we ought to be able to add support later.

No functional change intended.

Reviewed by: corvink, jhb
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40740

bhyve: Move legacy PCI interrupt handling under amd64/

Specifically, move IO-APIC, LPC and PIRQ routing code under amd64/.

Use ifdefs to conditionally compile related code in other files. In
particular, legacy PCI interrupt handling is now compiled only on amd64.
This is not too invasive, but suggestions for a more modular approach
would be appreciated.

I am not sure why qemu fwcfg handling is tied to LPC, and I suspect it
should be decoupled. In this commit I just apply an ifdef hammer, but
we will eventually want fwcfg on arm64 as well.

No functional change intended.

Reviewed by: corvink, jhb
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40739

bhyve: Compile some device models only on amd64

These models register legacy PCI interrupts, which won't be supported in
the arm64 port. In principle it should be possible to make these models
work on arm64 with a bit of effort, so don't move the sources to the
amd64 subdirectory.

No functional change intended.

Reviewed by: corvink, jhb
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40738

bhyve: Move AP startup code to amd64/

This code is only invoked via MD vmexit handlers. No functional change
intended.

Reviewed by: corvink, jhb
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40737

bhyve: Move the gvt-d driver to amd64/

It is amd64-only. No functional change intended.

Reviewed by: corvink, jhb
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40736

bhyve: Move power management code to amd64/

This implements various x86-specific interfaces. No functional change
intended.

Reviewed by: corvink, jhb
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40735

bhyve: Move the RTC driver to amd64/

No functional change intended.

Reviewed by: corvink, jhb
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40734

bhyve: Move MSR emulation into amd64/

No functional change intended.

Reviewed by: corvink, jhb
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40733

bhyve: Split vmexit handling into a separate file

Put it in amd64, since most of it is MD and won't be used on arm64. Add
a bit of glue to bhyverun.h to make CPU startup and shutdown work
without having to export more global variables. AP startup will be
reworked further in a future revision.

This makes bhyverun.c much more machine-independent.

No functional change intended.

Reviewed by: corvink, jhb
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40556

bhyve: Move kernemu to amd64/

This code handles instruction emulation for accesses to various
amd64-specific MMIO regions.

No functional change intended.

Reviewed by: corvink, jhb, emaste
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40554

bhyve: Move more amd64-specific code under amd64/

mptable and the e820 are both rather amd64-specific and can be moved
easily.

In the case of e820, move the registration with qemu_fwcfg into e820.c,
as it simplifies bhyverun.c a bit and I can't see any downsides.

No functional change intended.

Reviewed by: corvink, jhb, emaste
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40552

bhyve: Move some more amd64-specific drivers to their own subdir

No functional change intended.

Reviewed by: corvink, jhb, emaste
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40551

bhyve: Start moving machine-dependent code into subdirectories

In preparation for an arm64 port, make an easy change which puts some
machine-dependent code in its own directory.

Going forward, code which is only used on one platform should live in a
MD directory. We should strive to layer modules in such a way as to
avoid polluting shared code with lots of ifdefs. For some existing
files this will take some effort.

task_switch.c and fwctl.c are an easy place to start: the former is very
x86-specific, and the latter provides an I/O port interface which can't
be used on anything other than x86. (fwcfg as implemented has the same
problem, but QEMU also supports a MMIO fwcfg interface.) So I propose
that we start by simply making those files conditional.

Reviewed by: corvink, jhb
MFC after: 1 week
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D40501

Remove $FreeBSD$: one-line sh pattern

Remove /^\s*#[#!]?\s*\$FreeBSD\$.*$\n/

bhyve: add Qemu PPI emulation for TPM devices

Windows requires a physical presence interface to recognize the TPM
device. Qemu's OVMF has an implementation for the PPI which can be
reused. Using the Qemu PPI makes it very easy because we don't have to
implement new PPI functionality into our OVMF. The Qemu implementation
is already there.

Reviewed by: markj
MFC after: 1 week
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D40462

Revert "bhyve: add command line parameter and parsing for migration"

Unfortunately, this feature didn't receive much feedback in the past.
However, after committing this, some people came up and complain that
this feature requires some more discussion before upstreaming it.
Additionally, it wasn't a good idea to start this new feature by adding
a new command line parameter as it fixes the user interface.

This reverts commit c9fdd4f3cc18c03683de85318ba8d318f96b58c4.

bhyve: add command line parameter and parsing for migration

This covers warm and live migration.

Reviewed by: corvink
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D34717

bhyve: add basic CRB interface for TPM devices

Add a basic emulation for the command and response buffer interface of
TPM devices. This commit only implements some CRB register and resets
them.

Reviewed by: markj
MFC after: 1 week
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D40456

bhyve: add basic TPM passthrough emulation

At the moment, the emulation only opens a file descriptor to the TPM
device. Some subsequent commits will read and write from it.

Reviewed by: markj
MFC after: 1 week
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D40455

bhyve: add empty GVT-d emulation

Don't emulate anything yet. Just check if the user would like to pass an
Intel GPU to the guest.

Reviewed by: jhb, markj
MFC after: 1 week
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D40038

bhyve: Sort SRCS

No functional change intended.

Reviewed by: corvink, jhb
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D40553

bhyve: add basic TPM device

Add an empty TPM device struct which will be used for TPM emulation in
subsequent commits.

Reviewed by: markj
MFC after: 1 week
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D40452

bhyve: specify OpenSSL 1.1 API

OPENSSL_API_COMPAT can be used to specify the OpenSSL API version in
use for the purpose of hiding deprecated interfaces and enabling
the appropriate deprecation notices.

This change is a NFC while we're still using OpenSSL 1.1.1 but will
avoid deprecation warnings upon the switch to OpenSSL 3.0. A future
change can then switch bhyve to use OpenSSL 3.0 APIs.

Reviewed by: jhb
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D39998

bhyve: enable capsicum for snapshot code

Reviewed by: corvink
Sponsored by: vStack
Differential Revision: https://reviews.freebsd.org/D38860

bhyve: pass E820 table to guest

E820 table will be used to report valid RAM ranges and reserve special
memory areas like graphics memory for GPU passthrough.

Reviewed by: markj
MFC after: 1 week
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D39550

bhyve: add helper struct for qemus acpi table loader

The hypervisor is aware of all system properties. For the guest bios
it's hard and complex to detect all system properties. For that reason,
it would be better if the hypervisor creates acpi tables instead of the
guest. Therefore, the hypervisor has to send the acpi tables to the
guest. At the moment, bhyve just copies the acpi tables into the guest
memory. This approach has some restrictions. You have to keep sure that
the guest doesn't overwrite them accidentally. Additionally, the size of
acpi tables is limited.

Providing a plain copy of all acpi tables by fwcfg isn't possible. Acpi
tables have to point to each other. So, if the guest copies the acpi
tables into memory by it's own, it has to patch the tables. Due to
different layouts for different acpi tables, there's no generic way to
do that. For that reason, qemu created a table loader interface. It
contains commands for the guest for loading specific blobs into guest
memory and patching those blobs.

This commit adds a qemu_loader class which handles the creation of qemu
loader commands. At the moment, the WRITE_POINTER command isn't
implement. It won't be required by bhyve's acpi table generation yet.

Reviewed by: markj
MFC after: 1 week
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D38438

ext2fs: extract crc16 into sys/crc16.h

deduplicate this as it might be needed for other drivers (e.g. Apple SPI-HID)

Sponsored by: https://www.patreon.com/valpackett
Reviewed by: chuck, imp
MFC after: 1 month
Differential revision: https://reviews.freebsd.org/D32879

bhyve: add cmdline option to enable qemu's fwcfg

Let the user decide if he wants to use bhyve's fwctl or qemu's fwcfg. He
can set the interface by adding a fwcfg option to bootrom:

-l bootrom,<path/to/rom>,fwcfg=bhyve
-l bootrom,<path/to/rom>,fwcfg=qemu

Reviewed by: markj
MFC after: 1 week
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D38337

bhyve: Move libcasper dependecy to lib9p

libcasper(3) is not used in bhyve. So move dependency to the appropriate
place.

Reviewed by: markj
MFC after: 1 week
Sponsored by: vStack
Differential Revision: https://reviews.freebsd.org/D38905

bhyve: add helper struct for acpi device handling

To simplify the handling of different acpi devices like qemu fwcfg or a
tpm, add a helper struct. It will handle the reporting of acpi
resources.

Reviewed by: markj
MFC after: 1 week
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D38327

bhyve: Enable the default compiler warnings

Disable -Wcast-align for now since we have many instances of that
warning (I fixed some but not most of them) and platforms on which bhyve
runs don't particularly care about unaligned accesses.

Reviewed by: corvink
Differential Revision: https://reviews.freebsd.org/D37296

bhyve: Disable thread safety analysis

The warnings that arise are bogus and have to be muted with
__no_lock_analysis in most cases. As a step towards enabling the
default warning level for bhyve, just disable them.

Reviewed by: corvink, jhb
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D37295

bhyve: add basic basl implementation

Basl is the bhyve ASL compiler. At the moment, it's just a small wrapper
to call iasl, the Intel ASL compiler. As bhyve will gain support for
qemu's ACPI table loader in the future, it has to create ACPI tables on
it's own. Therefore, it makes sense to create a new file which keeps the
code for basl.

This first implementation of basl supports creating an ACPI table by
appending raw bytes to it. It's also capable of loading all tables into
guest memory.

Reviewed by: jhb, markj (older version)
Approved by: manu (mentor)
MFC after: 2 weeks
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D36984

Append Keyboard Layout specified option for using VNC.
Part two: Append bhyve -K option for specified keyboard layout
with layout setting files every languages.
Since the cmd option '-k' was used in the meantime
it was changed to '-K'

PR: 246121
Submitted by: koinec@yahoo.co.jp
Reviewed by: grehan@
Differential Revision: https://reviews.freebsd.org/D29473

MFC after: 4 weeks

Add a virtio-input device emulation.

This will be used to inject keyboard/mouse input events into a guest.
The command line syntax is:
-s <slot>,virtio-input,/dev/input/eventX

Reviewed by: jhb (bhyve), grehan
Obtained from: Corvin Köhne <C.Koehne@beckhoff.com>
MFC after: 3 weeks
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D30020

Refactor configuration management in bhyve.

Replace the existing ad-hoc configuration via various global variables
with a small database of key-value pairs. The database supports
heirarchical keys using a MIB-like syntax to name the path to a given
key. Values are always stored as strings. The API used to manage
configuation values does include wrappers to handling boolean values.
Other values use non-string types require parsing by consumers.

The configuration values are stored in a tree using nvlists. Leaf
nodes hold string values. Configuration values are permitted to
reference other configuration values using '%(name)'. This permits
constructing template configurations.

All existing command line arguments now set configuration values. For
devices, the "-s" option parses its option argument to generate a list
of key-value pairs for the given device.

A new '-o' command line option permits setting an individual
configuration variable. The key name is always given as a full path
of dot-separated components.

A new '-k' command line option parses a simple configuration file.
This configuration file holds a flat list of 'key=value' lines where
the 'key' is the full path of a configuration variable. Lines
starting with a '#' are comments.

In general, bhyve starts by parsing command line options in sequence
and applying those settings to configuration values. Once this is
complete, bhyve then begins initializing its state based on the
configuration values. This means that subsequent configuration
options or files may override or supplement previously given settings.

A special 'config.dump' configuration value can be set to true to help
debug configuration issues. When this value is set, bhyve will print
out the configuration variables as a flat list of 'key=value' lines.

Most command line argments map to a single configuration variable,
e.g. '-w' sets the 'x86.strictmsr' value to false. A few command
line arguments have less obvious effects:

- Multiple '-p' options append their values (as a comma-seperated
list) to "vcpu.N.cpuset" values (where N is a decimal vcpu number).

- For '-s' options, a pci.<bus>.<slot>.<function> node is created.
The first argument to '-s' (the device type) is used as the value of
a "device" variable. Additional comma-separated arguments are then
parsed into 'key=value' pairs and used to set additional variables
under the device node. A PCI device emulation driver can provide
its own hook to override the parsing of the additonal '-s' arguments
after the device type.

After the configuration phase as completed, the init_pci hook
then walks the "pci.<bus>.<slot>.<func>" nodes. It uses the
"device" value to find the device model to use. The device
model's init routine is passed a reference to its nvlist node
in the configuration tree which it can query for specific
variables.

The result is that a lot of the string parsing is removed from
the device models and centralized. In addition, adding a new
variable just requires teaching the model to look for the new
variable.

- For '-l' options, a similar model is used where the string is
parsed into values that are later read during initialization.
One key note here is that the serial ports use the commonly
used lowercase names from existing documentation and examples
(e.g. "lpc.com1") instead of the uppercase names previously
used internally in bhyve.

Reviewed by: grehan
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D26035

bhyve: fix build without casper/capsicum support

Fix typo introduced in 966026246e62769f3bcd8247a47fe0f4f0433aba.

Pointed out by: jbeich

bhyve: fix build without casper/capsicum support

PR: 252353

Remove bvmconsole and bvmdebug.

Now that bhyve(8) supports UART, bvmconsole and bvmdebug are no longer needed.

This also removes the '-b' and '-g' flag from bhyve(8). These two flags were
marked deprecated in r368519.

Reviewed by: grehan, kevans
Approved by: kevans (mentor)
Differential Revision: https://reviews.freebsd.org/D27490

Add legacy debug/test interfaces for kvm unit tests.

Implement the legacy debug/test interfaces expected by KVM-unit-tests'
realmode, emulator, and ioapic tests.

Submitted by: adam_fenn.io
Reviewed by: markj, grehan
Approved by: grehan (bhyve)
MFC after: 3 weeks
Relnotes: Yes
Differential Revision: https://reviews.freebsd.org/D27130

Add virtio-9p (aka VirtFS) filesystem sharing to bhyve.

VirtFS allows sharing an arbitrary directory tree between bhyve virtual
machine and the host. Current implementation has a fairly complete support
for 9P2000.L protocol, except for the extended attribute support. It has
been verified to work with the qemu-kvm hypervisor.

Reviewed by: rgrimes, emaste, jhb, trasz
Approved by: trasz (mentor)
MFC after: 1 month
Relnotes: yes
Sponsored by: Conclusive Engineering (development), vStack.com (funding)
Differential Revision: https://reviews.freebsd.org/D10335

vmm(4), bhyve(8): Expose kernel-emulated special devices to userspace

Expose the special kernel LAPIC, IOAPIC, and HPET devices to userspace
for use in, e.g., fallback instruction emulation (when userspace has a
newer instruction decode/emulation layer than the kernel vmm(4)).

Plumb the ioctl through libvmmapi and register the memory ranges in
bhyve(8).

Reviewed by: grehan
Differential Revision: https://reviews.freebsd.org/D24525

Add a new bhyve network backend that allow to connect the VM to the netgraph(4) network.
The backend uses the socket API with the PF_NETGRAPH protocol family, which is provided by the ng_socket(4).

To use the new backend, provide the following bhyve option:
-s X:Y:Z,[virtio-net|e1000],netgraph,socket=[ng_socket name],path=[destination node],hook=[our socket src hook],peerhook=[dst node hook]

Reviewed by: vmaffione, lutz_donnerhacke.de
Approved by: vmaffione (mentor)
Sponsored by: vstack.com
Differential Revision: https://reviews.freebsd.org/D24620

Initial support for bhyve save and restore.

Save and restore (also known as suspend and resume) permits a snapshot
to be taken of a guest's state that can later be resumed. In the
current implementation, bhyve(8) creates a UNIX domain socket that is
used by bhyvectl(8) to send a request to save a snapshot (and
optionally exit after the snapshot has been taken). A snapshot
currently consists of two files: the first holds a copy of guest RAM,
and the second file holds other guest state such as vCPU register
values and device model state.

To resume a guest, bhyve(8) must be started with a matching pair of
command line arguments to instantiate the same set of device models as
well as a pointer to the saved snapshot.

While the current implementation is useful for several uses cases, it
has a few limitations. The file format for saving the guest state is
tied to the ABI of internal bhyve structures and is not
self-describing (in that it does not communicate the set of device
models present in the system). In addition, the state saved for some
device models closely matches the internal data structures which might
prove a challenge for compatibility of snapshot files across a range
of bhyve versions. The file format also does not currently support
versioning of individual chunks of state. As a result, the current
file format is not a fixed binary format and future revisions to save
and restore will break binary compatiblity of snapshot files. The
goal is to move to a more flexible format that adds versioning,
etc. and at that point to commit to providing a reasonable level of
compatibility. As a result, the current implementation is not enabled
by default. It can be enabled via the WITH_BHYVE_SNAPSHOT=yes option
for userland builds, and the kernel option BHYVE_SHAPSHOT.

Submitted by: Mihai Tiganus, Flavius Anton, Darius Mihai
Submitted by: Elena Mihailescu, Mihai Carabas, Sergiu Weisz
Relnotes: yes
Sponsored by: University Politehnica of Bucharest
Sponsored by: Matthew Grooms (student scholarships)
Sponsored by: iXsystems
Differential Revision: https://reviews.freebsd.org/D19495

bhyve(8): Add VM Generation Counter ACPI device

Add an implementatation of the 'Virtual Machine Generation ID' spec to
Bhyve. The spec provides a randomly generated GUID (at bhyve start) in
device memory, along with an ACPI device with _CID VM_Gen_Counter and ADDR
evaluating to a Package pointing at that GUID.

A GPE is defined which Notifies the ACPI Device when the generation changes
(such as when a snapshot is rolled back). At this time, Bhyve does not
support snapshotting, so the GPE is never actually raised.

Suggested by: rpokala
Discussed with: grehan
Differential Revision: https://reviews.freebsd.org/D23165

bhyve: abstraction for network backends

Bhyve can currently emulate two virtual NICs, namely virtio-net and e1000,
and connect to the host network through two backends, namely tap and netmap.
However, there is no interface between virtual NIC functionalities and
backend functionalities. As a result, the backend code is duplicated between
the two virtual NIC implementations and also within the same virtual NIC.
Also, e1000 cannot currently use netmap as a backend.
This patch introduces a network backend API between virtio-net/e1000 and
tap/netmap, to improve code reuse and add missing functionalities.
Virtual NICs and backends can negotiate virtio-net features, such as checksum
offload and TSO. If the backend supports the features, it will propagate this
information to the guest, so that the latter can make use of them. Currently,
only netmap VALE ports support the features, but support should be added to
tap in the future.

Reviewed by: jhb, bryanv
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D20659

Replay r349335 by scottl accidentally reverted by r349352

Add the PCI HDAudio device model from the 2016 GSoC. Detailed information
can be found at

https://wiki.freebsd.org/SummerOfCode2016/HDAudioEmulationForBhyve

This commit has evolved from the original work to include Capsicum
integration. As part of that, it only opens the host audio devices once
and leaves them open, instead of opening and closing them on each guest
access. Thanks to Peter Grehan and Marcelo Araujo for their help in
bringing the work forward and providing some of the final techncial push.

Submitted by: Alex Teaca <iateaca@freebsd.org>
Differential Revision: D7840, D12419

Remove NAND and NANDFS support

NANDFS has been broken for years. Remove it. The NAND drivers that
remain are for ancient parts that are no longer relevant. They are
polled, have terrible performance and just for ancient arm
hardware. NAND parts have evolved significantly from this early work
and little to none of it would be relevant should someone need to
update to support raw nand. This code has been off by default for
years and has violated the vnode protocol leading to panics since it
was committed.

Numerous posts to arch@ and other locations have found no actual users
for this software.

Relnotes: Yes
No Objection From: arch@
Differential Revision: https://reviews.freebsd.org/D20745

Add the PCI HDAudio device model from the 2016 GSoC. Detailed information
can be found at

https://wiki.freebsd.org/SummerOfCode2016/HDAudioEmulationForBhyve

This commit has evolved from the original work to include Capsicum
integration. As part of that, it only opens the host audio devices once
and leaves them open, instead of opening and closing them on each guest
access. Thanks to Peter Grehan and Marcelo Araujo for their help in
bringing the work forward and providing some of the final techncial push.

Submitted by: Alex Teaca <iateaca@freebsd.org>
Differential Revision: D7840, D12419

bhyve: move common code to net_utils.c

Both virtio_net and e82545 network frontends have code to validate and
generate MAC addresses. These functionalities are replicated in the two
files, so we move them in a separate compilation unit.

Reviewed by: rgrimes, bryanv, imp, kevans
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D20626

Stop compiling bhyve(8) with -O0.

DEBUG_FLAGS is always added to CFLAGS. This setting appears to be
accidental and came in with r243327.

Reviewed by: anish, emaste, jhb, rgrimes
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D19787

Allow the bhyve VNC server to listen on IPv6 for incoming connections.

Alternatively to IPv4 address:port this will allow to listen on IPv6
link-local (incl. scope), a specific address, or ::. Addresses have
to be given in RFC2732 format so that [::]:port parsing will work.

This patch also starts to introduce WITH_INET/INET6_SUPPORT to bhyve.

PR: 232018
Submitted by: Dave Rush (northwoodlogic.free gmail.com) (original)
Reviewed by: Dave Rush (updated verison)
MFC after: 3 days

- Add bhyve NVMe device emulation.

The initial work on bhyve NVMe device emulation was done by the GSoC student
Shunsuke Mie and was heavily modified in performan, functionality and
guest support by Leon Dang.

bhyve:
-s <n>,nvme,devpath,maxq=#,qsz=#,ioslots=#,sectsz=#,ser=A-Z

accepted devpath:
/dev/blockdev
/path/to/image
ram=size_in_MiB

Tested with guest OS: FreeBSD Head, Linux Fedora fc27, Ubuntu 18.04,
OpenSuse 15.0, Windows Server 2016 Datacenter.
Tested with all accepted device paths: Real nvme, zdev and also with ram.
Tested on: AMD Ryzen Threadripper 1950X 16-Core Processor and
Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz.

Tests at: https://people.freebsd.org/~araujo/bhyve_nvme/nvme.txt

Submitted by: Shunsuke Mie <sux2mfgj_gmail.com>,
Leon Dang <leon_digitalmsx.com>
Reviewed by: chuck (early version), grehan
Relnotes: Yes
Sponsored by: iXsystems Inc.
Differential Revision: https://reviews.freebsd.org/D14022

- Add bhyve virtio-scsi storage backend support.

Example of configuration:
ctl.conf:
portal-group pg0 {
discovery-auth-group no-authentication
listen 0.0.0.0
listen [::]
}

target iqn.2012-06.com.example:target0 {
auth-group no-authentication
portal-group pg0
port ioctl/5/3

lun 0 {
path /z/test.img
size 8G
}
lun 1 {
path /z/test1.img
size 8G
}
}

bhyve <...> -s 4,virtio-scsi,/dev/cam/ctl5.3,iid=3 <VM_NAME>

From inside guest:
root@:~ # zpool status test
pool: test
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
test ONLINE 0 0 0
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0

dmesg:
da0 at vtscsi0 bus 0 scbus0 target 0 lun 0
da0: <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
da0: Serial Number MYSERIAL0000
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 8192MB (16777216 512 byte sectors)
da1 at vtscsi0 bus 0 scbus0 target 0 lun 1
da1: <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
da1: Serial Number MYSERIAL0001
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 8192MB (16777216 512 byte sectors)

Discussed with: grehan
Reviewed by: mav
Obtained from: TrueOS
Relnotes: Yes
Sponsored by: iXsystems Inc.
Tested with: FreeBSD HEAD, Fedora 28 (Workstation) and
Ubuntu 18.04.
Differential Revision: https://reviews.freebsd.org/D15276

Initial debug server for bhyve.

This commit adds a new debug server to bhyve. Unlike the existing -g
option which provides an efficient connection to a debug server
running in the guest OS, this debug server permits inspection and
control of the guest from within the hypervisor itself without
requiring any cooperation from the guest. It is similar to the debug
server provided by qemu.

To avoid conflicting with the existing -g option, a new -G option has
been added that accepts a TCP port. An IPv4 socket is bound to this
port and listens for connections from debuggers. In addition, if the
port begins with the character 'w', the hypervisor will pause the
guest at the first instruction until a debugger attaches and
explicitly continues the guest. Note that only a single debugger can
attach to a guest at a time.

Virtual CPUs are exposed to the remote debugger as threads. General
purpose register values can be read for each virtual CPU. Other
registers cannot currently be read, and no register values can be
changed by the debugger.

The remote debugger can read guest memory but not write to guest
memory. To facilitate source-level debugging of the guest, memory
addresses from the debugger are treated as virtual addresses (rather
than physical addresses) and are resolved to a physical address using
the active virtual address translation of the current virtual CPU.
Memory reads should honor memory mapped I/O regions, though the debug
server does not attempt to honor any alignment or size constraints
when accessing MMIO.

The debug server provides limited support for controlling the guest.
The guest is suspended when a debugger is attached and resumes when a
debugger detaches. A debugger can suspend a guest by sending a Ctrl-C
request (e.g. via Ctrl-C in GDB). A debugger can also continue a
suspended guest while remaining attached. Breakpoints are not yet
supported. Single stepping is supported on Intel CPUs that support
MTRAP VM exits, but is not available on other systems.

While the current debug server has limited functionality, it should
at least be usable for basic debugging now. It is also a useful
checkpoint to serve as a base for adding additional features.

Reviewed by: grehan
Differential Revision: https://reviews.freebsd.org/D15022

Add VNC Authentication support based on RFC6143 section 7.2.2.

Submitted by: Fabian Freyer <fabian.freyer@physik.tu-berlin.de>
Reworked by: myself
Reviewed by: grehan, rgrimes and jilles
MFC after: 1 week.
Relnotes: Yes.
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D10818

Add virtio-console support to bhyve.

Adds virtio-console device support to bhyve, allowing to create
bidirectional character streams between host and guest.

Syntax:
-s <slotnum>,virtio-console,port1=/path/to/port1.sock,anotherport=...

Maximum of 16 ports per device can be created. Every port is named
and corresponds to an Unix domain socket created by bhyve. bhyve
accepts at most one connection per port at a time.

Limitations:
- due to lack of destructors of in bhyve, sockets on the filesystem
must be cleaned up manually after bhyve exits
- there's no way to use "console port" feature, nor the console port
resize as of now
- emergency write is advertised, but no-op as of now

Approved by: trasz
MFC after: 1 month
Relnotes: yes
Sponsored by: iXsystems, Inc.
Differential Revision: D7185

Revert unwanted change leaked into r302932.

Increase I82545_MAX_TXSEGS from 20 to 64 and add checks for it.

There seems no hard limit on number of segments per packet in the chip,
and 20 appeared insufficient. Hope 64 will be enough, but if not -- add
check to report that and drop the packet instead of corrupting stack.

Add emulation for Intel e1000 (e82545) network adapter.

The code was successfully tested with FreeBSD, Linux, Solaris and Windows
guests. This interface is predictably slower (about 2x) then virtio-net,
but it is very helpful for guests not supporting virtio-net by default.

Thanks to Jeremiah Lott and Peter Grehan for doing original heavy lifting.

Don't use SYSDIR to avoid conflicts with existing usage.
Also, use SRCTOP to locate the top of the source tree
instead of a relative path.

PR: 208856

Allow the location of the kernel source tree to be overridden.
This makes it easier for the bhyve executable to be built
out of the tree.

Simple sysctl-like firmware query interface. Similar in operation
to the qemu one, and uses the same i/o ports but with different
messaging. Requires the 'bootrom' option to be enabled.

This is used by UEFI (and potentially other BIOSs/firmware) to
request information from bhyve. Currently, only the number of
vCPUs is made available, with more to follow.

A very large thankyou to Ben Perrault who helped out testing
an earlier version of this, and bhyve/Windows in general.

Reviewed by: tychon
Discussed with: neel
Sponsored by: Nahanni Systems

Restructure memory allocation in bhyve to support "devmem".

devmem is used to represent MMIO devices like the boot ROM or a VESA framebuffer
where doing a trap-and-emulate for every access is impractical. devmem is a
hybrid of system memory (sysmem) and emulated device models.

devmem is mapped in the guest address space via nested page tables similar
to sysmem. However the address range where devmem is mapped may be changed
by the guest at runtime (e.g. by reprogramming a PCI BAR). Also devmem is
usually mapped RO or RW as compared to RWX mappings for sysmem.

Each devmem segment is named (e.g. "bootrom") and this name is used to
create a device node for the devmem segment (e.g. /dev/vmm/testvm.bootrom).
The device node supports mmap(2) and this decouples the host mapping of
devmem from its mapping in the guest address space (which can change).

Reviewed by: tychon
Discussed with: grehan
Differential Revision: https://reviews.freebsd.org/D2762
MFC after: 4 weeks

Fix overlinking in bhyve:

libvmmapi is actually needed to be linked to libutil, not bhyve nor bhyveload

Make FreeBSD-bhyve an indivual package

Convert usr.sbin to LIBADD
Reduce overlinking

Move the ACPI PM timer emulation into vmm.ko.

This reduces variability during timer calibration by keeping the emulation
"close" to the guest. Additionally having all timer emulations in the kernel
will ease the transition to a per-VM clock source (as opposed to using the
host's uptime keep track of time).

Discussed with: grehan

Add emulation for legacy x86 task switching mechanism.

FreeBSD/i386 uses task switching to handle double fault exceptions and this
change enables that to work.

Reported by: glebius

Implement a PCI interrupt router to route PCI legacy INTx interrupts to
the legacy 8259A PICs.
- Implement an ICH-comptabile PCI interrupt router on the lpc device with
8 steerable pins configured via config space access to byte-wide
registers at 0x60-63 and 0x68-6b.
- For each configured PCI INTx interrupt, route it to both an I/O APIC
pin and a PCI interrupt router pin. When a PCI INTx interrupt is
asserted, ensure that both pins are asserted.
- Provide an initial routing of PCI interrupt router (PIRQ) pins to
8259A pins (ISA IRQs) and initialize the interrupt line config register
for the corresponding PCI function with the ISA IRQ as this matches
existing hardware.
- Add a global _PIC method for OSPM to select the desired interrupt routing
configuration.
- Update the _PRT methods for PCI bridges to provide both APIC and legacy
PRT tables and return the appropriate table based on the configured
routing configuration. Note that if the lpc device is not configured, no
routing information is provided.
- When the lpc device is enabled, provide ACPI PCI link devices corresponding
to each PIRQ pin.
- Add a VMM ioctl to adjust the trigger mode (edge vs level) for 8259A
pins via the ELCR.
- Mark the power management SCI as level triggered.
- Don't hardcode the number of elements in Packages in the source for
the DSDT. iasl(8) will fill in the actual number of elements, and
this makes it simpler to generate a Package with a variable number of
elements.

Reviewed by: tycho

Provide a very basic stub for the 8042 PS/2 keyboard controller.

Reviewed by: jhb
Approved by: neel (co-mentor)

Add support for the virtio RNG entropy-source device.
Call through to /dev/random synchronously to fill
virtio buffers with RNG data.

Tested with FreeBSD-CURRENT and Ubuntu guests.

Submitted by: Leon Dang
Discussed with: markm
MFC after: 3 weeks
Sponsored by: Nahanni Systems

Move the atpit device model from userspace into vmm.ko for better
precision and lower latency.

Approved by: grehan (co-mentor)

Replace the userspace atpic stub with a more functional vmm.ko model.

New ioctls VM_ISA_ASSERT_IRQ, VM_ISA_DEASSERT_IRQ and VM_ISA_PULSE_IRQ
can be used to manipulate the pic, and optionally the ioapic, pin state.

Reviewed by: jhb, neel
Approved by: neel (co-mentor)

Add SMBIOS support.

A new option, -U, can be used to set the UUID in the System
Information (Type 1) structure. Manpage fix to follow.

Approved by: grehan (co-mentor)

Enhance the support for PCI legacy INTx interrupts and enable them in
the virtio backends.
- Add a new ioctl to export the count of pins on the I/O APIC from vmm
to the hypervisor.
- Use pins on the I/O APIC >= 16 for PCI interrupts leaving 0-15 for
ISA interrupts.
- Populate the MP Table with I/O interrupt entries for any PCI INTx
interrupts.
- Create a _PRT table under the PCI root bridge in ACPI to route any
PCI INTx interrupts appropriately.
- Track which INTx interrupts are in use per-slot so that functions
that share a slot attempt to distribute their INTx interrupts across
the four available pins.
- Implicitly mask INTx interrupts if either MSI or MSI-X is enabled
and when the INTx DIS bit is set in a function's PCI command register.
Either assert or deassert the associated I/O APIC pin when the
state of one of those conditions changes.
- Add INTx support to the virtio backends.
- Always advertise the MSI capability in the virtio backends.

Submitted by: neel (7)
Reviewed by: neel
MFC after: 2 weeks

Cosmetic change - switch over to vertical SRCS to make it
easier to keep files in alpha order.

Reviewed by: neel

Support soft power-off via the ACPI S5 state for bhyve guests.
- Implement the PM1_EVT and PM1_CTL registers required by ACPI.
The PM1_EVT register is mostly a dummy as bhyve doesn't support any
of the hardware-initiated events. The only bit of PM1_CNT that is
implemented are the sleep request bits (SPL_EN and SLP_TYP) which
request a graceful power off for S5. In particular, for S5, bhyve
exits with a non-zero value which terminates the loop in vmrun.sh.
- Emulate the Reset Control register at I/O port 0xcf9 and advertise
it as the reset register via ACPI.
- Advertise an _S5 package.
- Extend the in/out interface to allow an in/out handler to request
that the hypervisor trigger a reset or power-off.
- While here, note that all vCPUs in a guest support C1 ("hlt").

Reviewed by: neel (earlier version)

bhyve(8) man page.

mdoc formatting and much input and review from Warren Block (wblock@).

Reviewed by: many
MFC after: 3 days

Move the ioapic device model from userspace into vmm.ko. This is needed for
upcoming in-kernel device emulations like the HPET.

The ioctls VM_IOAPIC_ASSERT_IRQ and VM_IOAPIC_DEASSERT_IRQ are used to
manipulate the ioapic pin state.

Discussed with: grehan@
Submitted by: Tycho Nightingale (tycho.nightingale@pluribusnetworks.com)

Add support for PCI-to-ISA LPC bridge emulation. If the LPC bus is attached
to a virtual machine then we implicitly create COM1 and COM2 ISA devices.

Prior to this change the only way of attaching a COM port to the virtual
machine was by presenting it as a PCI device that is mapped at the legacy
I/O address 0x3F8 or 0x2F8.

There were some issues with the original approach:
- It did not work at all with UEFI because UEFI will reprogram the PCI device
BARs and remap the COM1/COM2 ports at non-legacy addresses.
- OpenBSD GENERIC kernel does not create a /dev/console because it expects
the uart device at the legacy 0x3F8/0x2F8 address to be an ISA device.
- It was functional with a FreeBSD guest but caused the console to appear
on /dev/ttyu2 which was not intuitive.

The uart emulation is now independent of the bus on which it resides. Thus it
is possible to have uart devices on the PCI bus in addition to the legacy
COM1/COM2 devices behind the LPC bus.

The command line option to attach ISA COM1/COM2 ports to a virtual machine is
"-s <bus>,lpc -l com1,stdio".

The command line option to create a PCI-attached uart device is:
"-s <bus>,uart[,stdio]"

The command line option to create PCI-attached COM1/COM2 device is:
"-S <bus>,uart[,stdio]". This style of creating COM ports is deprecated.

Discussed with: grehan
Reviewed by: grehan
Submitted by: Tycho Nightingale (tycho.nightingale@pluribusnetworks.com)

M share/examples/bhyve/vmrun.sh
AM usr.sbin/bhyve/legacy_irq.c
AM usr.sbin/bhyve/legacy_irq.h
M usr.sbin/bhyve/Makefile
AM usr.sbin/bhyve/uart_emul.c
M usr.sbin/bhyve/bhyverun.c
AM usr.sbin/bhyve/uart_emul.h
M usr.sbin/bhyve/pci_uart.c
M usr.sbin/bhyve/pci_emul.c
M usr.sbin/bhyve/inout.c
M usr.sbin/bhyve/pci_emul.h
M usr.sbin/bhyve/inout.h
AM usr.sbin/bhyve/pci_lpc.c
AM usr.sbin/bhyve/pci_lpc.h

Parse the memory size parameter using expand_number() to allow specifying
the memory size more intuitively (e.g. 512M, 4G etc).

Submitted by: rodrigc
Reviewed by: grehan
Approved by: re (blanket)

Hook up the AHCI and blockif code to the build.

Approved by: re@ (blanket)

Major rework of the virtio code. Split out common parts, and modify
the net/block devices accordingly.

Submitted by: Chris Torek torek at torek dot net
Reviewed by: grehan

Remove dangling ISA uart stubs.

Obtained from: NetApp

Install <dev/agp/agpreg.h> and <dev/pci/pcireg.h> as userland headers
in /usr/include.

MFC after: 2 weeks

Rename fbsdrun.* -> bhyverun.*

bhyve is intended to be a generic hypervisor, and not FreeBSD-specific.

(renaming internal routines will come later)

Reviewed by: neel
Obtained from: NetApp

Revamp the x86 instruction emulation in bhyve.

On a nested page table fault the hypervisor will:
- fetch the instruction using the guest %rip and %cr3
- decode the instruction in 'struct vie'
- emulate the instruction in host kernel context for local apic accesses
- any other type of mmio access is punted up to user-space (e.g. ioapic)

The decoded instruction is passed as collateral to the user-space process
that is handling the PAGING exit.

The emulation code is fleshed out to include more addressing modes (e.g. SIB)
and more types of operands (e.g. imm8). The source code is unified into a
single file (vmm_instruction_emul.c) that is compiled into vmm.ko as well
as /usr/sbin/bhyve.

Reviewed by: grehan
Obtained from: NetApp

ACPI support for bhyve.

The -A option will create the minimal set of required ACPI tables in
guest memory. Since ACPI mandates an IOAPIC, the -I option must also
be used.

Template ASL files are created, and then passed to the iasl compiler
to generate AML files. These are then loaded into guest physical mem.

In support of this, the ACPI PM timer is implemented, in 32-bit mode.

Tested on 7.4/8.*/9.*/10-CURRENT.

Reviewed by: neel
Obtained from: NetApp
Discussed with: jhb (a long while back)

Remove mptable generation code from libvmmapi and move it to bhyve.
Firmware tables require too much knowledge of system configuration,
and it's difficult to pass that information in general terms to a library.
The upcoming ACPI work exposed this - it will also livein bhyve.

Also, remove code specific to NetApp from the mptable name, and remove
the -n option from bhyve.

Reviewed by: neel
Obtained from: NetApp

Rework how guest MMIO regions are dealt with.

- New memory region interface. An RB tree holds the regions,
with a last-found per-vCPU cache to deal with the common case
of repeated guest accesses to MMIO registers in the same page.

- Support memory-mapped BARs in PCI emulation.

mem.c/h - memory region interface

instruction_emul.c/h - remove old region interface.
Use gpa from EPT exit to avoid a tablewalk to
determine operand address. Determine operand size
and use when calling through to region handler.

fbsdrun.c - call into region interface on paging
exit. Distinguish between instruction emul error
and region not found

pci_emul.c/h - implement new BAR callback api.
Split BAR alloc routine into routines that
require/don't require the BAR phys address.

ioapic.c
pci_passthru.c
pci_virtio_block.c
pci_virtio_net.c
pci_uart.c - update to new BAR callback i/f

Reviewed by: neel
Obtained from: NetApp

Add an explicit exit code 'SPINUP_AP' to tell the controlling process that an
AP needs to be activated by spinning up an execution context for it.

The local apic emulation is now completely done in the hypervisor and it will
detect writes to the ICR_LO register that try to bring up the AP. In response
to such writes it will return to userspace with an exit code of SPINUP_AP.

Reviewed by: grehan

Device model for ioapic emulation.

With this change the uart emulation is entirely interrupt driven.

Obtained from: NetApp

Add 16550 uart emulation as a PCI device. This allows it to
be activated as part of the slot config options.
The syntax is:

-s <slotnum>,uart[,stdio]

The stdio parameter instructs the code to perform i/o using
stdin/stdout. It can only be used for one instance.
To allow legacy i/o ports/irqs to be used, a new variant of
the slot command, -S, is introduced. When used to specify a
slot, the device will use legacy resources if it supports
them; otherwise it will be treated the same as the '-s' option.
Specifying the -S option with the uart will first use the 0x3f8/irq 4
config, and the second -S will use 0x2F8/irq 3.

Interrupt delivery is awaiting the arrival of the i/o apic code,
but this works fine in uart(4)'s polled mode.

This code was written by Cynthia Lu @ MIT while an intern at NetApp,
with further work from neel@ and grehan@.

Obtained from: NetApp

MSI-x interrupt support for PCI pass-thru devices.

Includes instruction emulation for memory r/w access. This
opens the door for io-apic, local apic, hpet timer, and
legacy device emulation.

Submitted by: ryan dot berryhill at sandvine dot com
Reviewed by: grehan
Obtained from: Sandvine

Import of bhyve hypervisor and utilities, part 1.
vmm.ko - kernel module for VT-x, VT-d and hypervisor control
bhyve - user-space sequencer and i/o emulation
vmmctl - dump of hypervisor register state
libvmm - front-end to vmm.ko chardev interface

bhyve was designed and implemented by Neel Natu.

Thanks to the following folk from NetApp who helped to make this available:
Joe CaraDonna
Peter Snyder
Jeff Heller
Sandeep Mann
Steve Miller
Brian Pawlowski