bhyve.8: Update and rearrange to improve readability

bhyve's man page is a very long block of text that has grown to
proportions that make it hard to read. In particular, the level of
nesting of various content means man can no longer render the text in a
user-friendly way.

To remedy this:
- move the -s argument documentation into a separate section and
reformat the various arguments so they are consistent
- add documentation on how to use the -o config.dump feature
- make the listing of various arguments more consistent
- consolidated duplicate listings of TPM backends
- add an example for the config.dump feature
- fix various formatting inconsistencies.

Reviewed by: emaste, imp, jrm, Pau Amma <pauamma@gundo.com>, rgrimes
Differential Revision: https://reviews.freebsd.org/D43940

bhyve.8: Document arm64 support

- Mention the options that are amd64-only.
- Provide a minimal example for booting an arm64 guest.

Reviewed by: corvink
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D44738

bhyve.8: Remove mention of the -A flag

It is a no-op on amd64 now and is not implemented on arm64, so let's
remove mention of it altogether so as to reduce confusion for arm64
users.

Reviewed by: corvink, jhb
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D44737

bhyve: verify that user provided fbuf values fall within limits

The upper limits for height/width are already checked, this commit
enforces the lower limit as well.

The behavior is also changed such that, the framebuffer fails to
initialize if the provided values don't parse cleanly.

Reviewed by: corvink, jhb
Differential Revision: https://reviews.freebsd.org/D42692

bhyve.8: fix -k description

Apparently 19eaa01beaba6 inserted -K description in the middle of -k.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Differential revision: https://reviews.freebsd.org/D43164

bhyve.8: add missed dot

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Differential revision: https://reviews.freebsd.org/D43164

bhyve: Add a slirp network backend

This enables a subset of the functionality provided by QEMU's user
networking implementation. In particular, it uses net/libslirp, the
same library as QEMU.

libslirp is permissively licensed but has some dependencies which make
it impractical to bring into the base system (glib in particular). I
thus opted to make bhyve dlopen the libslirp.so, which can be installed
via pkg. The library header is imported into bhyve.

The slirp backend takes a "hostfwd" which is identical to QEMU's
hostfwd. When configured, bhyve opens a host socket and listens for
connections, which get forwarded to the guest. For instance,
"hostfwd=tcp::1234-:22" allows one to ssh into the guest by ssh'ing to
port 1234 on the host, e.g., via 127.0.0.1. I didn't try to hook up
guestfwd support since I don't personally have a use-case for it yet,
and I think it won't interact nicely with the capsicum sandbox.

Reviewed by: jhb
Tested by: rew
MFC after: 1 month
Sponsored by: Innovate UK
Differential Revision: https://reviews.freebsd.org/D42510

bhyve(8): fix manpage formatting

In commit 67c26eb2a57 a stray ".El" was introduced, breaking
formatting of options.

Reviewed by: corvink, markj
Fixes: 67c26eb2a57c ("bhyve: add cmdline option for TPM emulation")
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D42434

bhyve: Document the hw.vmm.maxcpu tunable and the current limit on vCPUs

Reviewed by: corvink (original version)
Co-authored-by: John Baldwin <jhb@FreeBSD.org>
Differential Revision: https://reviews.freebsd.org/D40074

bhyve(8): Fix style warnings emitted by mandoc, no content changes

bhyve(8): Expand the -p option text with additional text and an example

PR: 249981

bhyve: always generate ACPI tables

Most systems don't work properly without sane ACPI tables. Therefore,
we're always generating them.

Reviewed by: markj
MFC after: 1 week
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D41778

bhyve: add cmdline option for TPM emulation

At the moment, only a TPM passthru is supported. The cmdline looks like:

-l tpm,passthru,/dev/tpm0

Reviewed by: markj
MFC after: 1 week
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D32961

Remove $FreeBSD$: two-line nroff pattern

Remove /^\.\\"\n\.\\"\s*\$FreeBSD\$$\n/

Revert "bhyve: add command line parameter and parsing for migration"

Unfortunately, this feature didn't receive much feedback in the past.
However, after committing this, some people came up and complain that
this feature requires some more discussion before upstreaming it.
Additionally, it wasn't a good idea to start this new feature by adding
a new command line parameter as it fixes the user interface.

This reverts commit c9fdd4f3cc18c03683de85318ba8d318f96b58c4.

bhyve: add bootindex option for several devices

The bootindex option creates an entry in the "bootorder" fwcfg file.
This file can be picked up by the guest firmware to determine the
bootorder. Nevertheless, it's not guaranteed that the guest firmware
uses the bootorder. At the moment, our OVMF ignores the bootorder. This
will change in the future.

If guest firmware supports the "bootorder" fwcfg file and no device uses
the bootindex option, the boot order is determined by the firmware
itself. If one or more devices specify a bootindex, the first bootable
device with the lowest bootindex will be booted. It's not garanteed that
devices without a bootindex will be recognized as bootable from the
firmware in that case.

Reviewed by: jhb
MFC after: 1 week
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D39285

bhyve: add command line parameter and parsing for migration

This covers warm and live migration.

Reviewed by: corvink
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D34717

bhyve: add cmdline option for user defined fw_cfg items

Some guest allow to configure themself by fw_cfg. E.g. Fedora CoreOs can
be provisioned by adding a JSON file as fw_cfg item.

Reviewed by: jhb
MFC after: 1 week
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D38338

bhyve(8): fix typo in examples

PR: 271259

bhyve: add cmdline option to enable qemu's fwcfg

Let the user decide if he wants to use bhyve's fwctl or qemu's fwcfg. He
can set the interface by adding a fwcfg option to bootrom:

-l bootrom,<path/to/rom>,fwcfg=bhyve
-l bootrom,<path/to/rom>,fwcfg=qemu

Reviewed by: markj
MFC after: 1 week
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D38337

bhyve: Support other schemes for naming pass-through devices.

Permit naming pass through devices using the syntax accepted by
pciconf (pci[<domain>:]<bus>:<slot>:<func>) as well as by device name
(e.g. "ppt0").

While here, fix an error in the manpage that had the bus and slot
arguments for the original /-delimited scheme swapped.

Reviewed by: imp, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36147

bhyve.8: Fix a typo

The Li macro is deprecated. Also, the Cm macro should be used here
instead for consistency with the rest of the manual and style.mdoc(5).

Fixes: e47fe3183e1f bhyve: add ROM emulation
MFC after: 1 month

bhyve: add ROM emulation

Some PCI devices especially GPUs require a ROM to work properly.
The ROM is executed by boot firmware to initialize the device.
To add a ROM to a device use the new ROM option for passthru device
(e.g. -s passthru,0/2/0,rom=<path>/<to>/<rom>).

It's necessary that the ROM is executed by the boot firmware.
It won't be executed by any OS.
Additionally, the boot firmware should be configured to execute the
ROM file.
For that reason, it's only possible to use a ROM when using
OVMF with enabled bus enumeration.

Differential Revision: https://reviews.freebsd.org/D33129
Sponsored by: Beckhoff Automation GmbH & Co. KG
MFC after: 1 month

Append Keyboard Layout specified option for using VNC.
Part two: Append bhyve -K option for specified keyboard layout
with layout setting files every languages.
Since the cmd option '-k' was used in the meantime
it was changed to '-K'

PR: 246121
Submitted by: koinec@yahoo.co.jp
Reviewed by: grehan@
Differential Revision: https://reviews.freebsd.org/D29473

MFC after: 4 weeks

bhyve: Support a _VARS.fd file for bootrom

OVMF creates two separate .fd files, a _CODE.fd file containing
the UEFI code, and a _VARS.fd file containing a template of an
empty UEFI variable store.

OVMF decides to write variables to the memory range just below the
boot rom code if it detects a CFI flash device. So here we add
just the barest facsimile of CFI command handling to bootrom.c
that is needed to placate OVMF.

Submitted by: D Scott Phillips <d.scott.phillips@intel.com>
Sponsored by: Intel Corporation
Differential Revision: https://reviews.freebsd.org/D19976
MFC After: 1 week

bhyve: Update usage and synopsis for the -k flag

Let's make it clear to users that -k is for configuration files.
Also, point to bhyve_config(5) in the paragraph describing the flag.

Reviewed by: jhb
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D32467

bhyve.8: Fix markup of the -G flag

bhyve: Update the -G description in the SYNPOSIS.

It was missing both the 'w' flag and 'bind_address'.

byhve: add option to specify IP address for gdb

Allow user to specify the IP address available for gdb debugger.

Reviewed by: jhb, grehan, rgrimes, bcr (man pages)
Differential Revision: https://reviews.freebsd.org/D29607

Add a virtio-input device emulation.

This will be used to inject keyboard/mouse input events into a guest.
The command line syntax is:
-s <slot>,virtio-input,/dev/input/eventX

Reviewed by: jhb (bhyve), grehan
Obtained from: Corvin Köhne <C.Koehne@beckhoff.com>
MFC after: 3 weeks
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D30020

bhyve.8: Fix mandoc -Tlint issues

While here, keep network backends section consistent with other
sections.

MFC after: 2 weeks

bhyve: Document the format for HD audio backends

- This change is done for consistency with other backend definitions.

MFC after: 2 weeks

bhyve.8: Improve AHCI backends documentation

- Document the backend format.

MFC after: 2 weeks

bhyve.8: Improve documentation of NVME backend

- Document the configuration format.
- Document two additional configuration options: eui64 and dsm.

MFC after: 2 weeks

bhyve.8: Improve framebuffer backends description

- Use appropriate mdoc macros
- Document that tcp= is a synonym to rfb= (tcp is used in the examples,
but never mentioned)
- Clarify the IP address specification

MFC after: 2 weeks

bhyve.8: Clean up virtio console device backends description

MFC after: 2 weeks

bhyve.8: Clean up TTY, boot ROM, and pass-through descriptions

MFC after: 2 weeks

bhyve.8: Clean up 9P device backends section

MFC after: 2 weeks

bhyve.8: Clean up SCSI device backends section

MFC after: 2 weeks

bhyve.8: Clean up block storage device backends description

MFC after: 2 weeks

bhyve.8: Clean up network backends section

- Reformat the format lists, use appropriate mdoc macros for
readability.
- Add a missing Oxford comma.

MFC after: 2 weeks

bhyve.8: Improve emulation description of the -s flag

- Set width of the list to the longest key word for readability.
- Separate descriptions of amd_hostbridge and hostbridge emulations.
Also, wordsmith their descriptions for consistency with other entries.
- Use Cm instead of Li for command modifiers.
- Do not stylize AMD with Li, there's no need to do it.
- Mention COM3 and COM4 in the definition of lpc.
- Fix a typo in the definition of ahci-hd ("hard drive" instead of
"hard-drive").

MFC after: 2 weeks

bhyve.8: Clean up the slot description of -s

Also, remove the macros of the nested list which contained slot,
emulation and conf. This decreases the indention of the -s description.
It was necessary to clean up the slot description.

MFC after: 2 weeks

bhyve.8: Clean-up synopsis of -s

- Document "-s help" separately for readability.
- Use appropriate mdoc macros.

MFC after: 2 weeks

bhyve.8: Fix indention in the signals table

MFC after: 2 weeks

bhyve.8: Clean up description of -r

There is no need to wrap those flags in Op macros.

MFC after: 2 weeks

bhyve.8: Fix the synopsis of -p

Use appropriate mdoc macros.

MFC after: 2 weeks

bhyve.8: Improve the description of the -m flag

- Stylize the synopsis with proper mdoc macros
- Do some wordsmithing on the description for consistency.

MFC after: 2 weeks

bhyve.8: Improve the description and synopsis of -l

- Describe "-l help" separately for readability.
- List all the supported comX devices explicitly
- Use Cm instead of Ar for command modifiers (i.e., literal values a
user can specify as an argument to the command).
- Explain where to get more information about the possible values of the
conf argument.

MFC after: 2 weeks

bhyve.8: Sort the options in the OPTIONS section

No content change intended. Just moving the option descriptions around
to follow the order suggested by style(9).

MFC after: 2 weeks

bhyve.8: Make synopsis more readable

There is no need to squeeze all the possible options into one synopsis
entry. Let "-l help" and "-s help" be listed separately.

While here, keep -s and its arguments on the same line.

MFC after: 2 weeks

Refactor configuration management in bhyve.

Replace the existing ad-hoc configuration via various global variables
with a small database of key-value pairs. The database supports
heirarchical keys using a MIB-like syntax to name the path to a given
key. Values are always stored as strings. The API used to manage
configuation values does include wrappers to handling boolean values.
Other values use non-string types require parsing by consumers.

The configuration values are stored in a tree using nvlists. Leaf
nodes hold string values. Configuration values are permitted to
reference other configuration values using '%(name)'. This permits
constructing template configurations.

All existing command line arguments now set configuration values. For
devices, the "-s" option parses its option argument to generate a list
of key-value pairs for the given device.

A new '-o' command line option permits setting an individual
configuration variable. The key name is always given as a full path
of dot-separated components.

A new '-k' command line option parses a simple configuration file.
This configuration file holds a flat list of 'key=value' lines where
the 'key' is the full path of a configuration variable. Lines
starting with a '#' are comments.

In general, bhyve starts by parsing command line options in sequence
and applying those settings to configuration values. Once this is
complete, bhyve then begins initializing its state based on the
configuration values. This means that subsequent configuration
options or files may override or supplement previously given settings.

A special 'config.dump' configuration value can be set to true to help
debug configuration issues. When this value is set, bhyve will print
out the configuration variables as a flat list of 'key=value' lines.

Most command line argments map to a single configuration variable,
e.g. '-w' sets the 'x86.strictmsr' value to false. A few command
line arguments have less obvious effects:

- Multiple '-p' options append their values (as a comma-seperated
list) to "vcpu.N.cpuset" values (where N is a decimal vcpu number).

- For '-s' options, a pci.<bus>.<slot>.<function> node is created.
The first argument to '-s' (the device type) is used as the value of
a "device" variable. Additional comma-separated arguments are then
parsed into 'key=value' pairs and used to set additional variables
under the device node. A PCI device emulation driver can provide
its own hook to override the parsing of the additonal '-s' arguments
after the device type.

After the configuration phase as completed, the init_pci hook
then walks the "pci.<bus>.<slot>.<func>" nodes. It uses the
"device" value to find the device model to use. The device
model's init routine is passed a reference to its nvlist node
in the configuration tree which it can query for specific
variables.

The result is that a lot of the string parsing is removed from
the device models and centralized. In addition, adding a new
variable just requires teaching the model to look for the new
variable.

- For '-l' options, a similar model is used where the string is
parsed into values that are later read during initialization.
One key note here is that the serial ports use the commonly
used lowercase names from existing documentation and examples
(e.g. "lpc.com1") instead of the uppercase names previously
used internally in bhyve.

Reviewed by: grehan
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D26035

Fix manpage markup in 2c8bb126de56ce49637ad752e78957d9ebafba02

bhyve: Add missing man page section on the nodelete block-device-option

Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D28272

Support COM3 and COM4 serial ports.

Submitted by: Jan Poctavek <janci@binaryparadise.com>, otis
Reviewed by: grehan (bhyve), imp, 0mp (manpages)
Differential Revision: https://reviews.freebsd.org/D28207

Remove bvmconsole and bvmdebug.

Now that bhyve(8) supports UART, bvmconsole and bvmdebug are no longer needed.

This also removes the '-b' and '-g' flag from bhyve(8). These two flags were
marked deprecated in r368519.

Reviewed by: grehan, kevans
Approved by: kevans (mentor)
Differential Revision: https://reviews.freebsd.org/D27490

Add deprecation notice for bvmconsole and bvmdebug

Now that bhyve(8) supports UART, bvmconsole and bvmdebug are no longer needed.

Mark the '-b' and '-g' flag as deprecated for bhyve(8).

These will be removed in 13.

Reviewed by: jhb, grehan
Approved by: kevans (mentor)
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D27519

Add legacy debug/test interfaces for kvm unit tests.

Implement the legacy debug/test interfaces expected by KVM-unit-tests'
realmode, emulator, and ioapic tests.

Submitted by: adam_fenn.io
Reviewed by: markj, grehan
Approved by: grehan (bhyve)
MFC after: 3 weeks
Relnotes: Yes
Differential Revision: https://reviews.freebsd.org/D27130

Add virtio-9p (aka VirtFS) filesystem sharing to bhyve.

VirtFS allows sharing an arbitrary directory tree between bhyve virtual
machine and the host. Current implementation has a fairly complete support
for 9P2000.L protocol, except for the extended attribute support. It has
been verified to work with the qemu-kvm hypervisor.

Reviewed by: rgrimes, emaste, jhb, trasz
Approved by: trasz (mentor)
MFC after: 1 month
Relnotes: yes
Sponsored by: Conclusive Engineering (development), vStack.com (funding)
Differential Revision: https://reviews.freebsd.org/D10335

Support the setting of additional AHCI controller parameters.

Allow the serial number, firmware revision, model number and nominal media
rotation rate (nmrr) parameters to be set from the command line.

Note that setting the nmrr value can be used to indicate the AHCI
device is an SSD.

Submitted by: Wanpeng Qian
Reviewed by: jhb, grehan (#bhyve)
Approved by: jhb, grehan
MFC after: 3 weeks
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D24174

bhyve: allow for automatic destruction on power-off

Introduce -D flag that allows for the VM to be destroyed on guest initiated
power-off by the bhyve(8) process itself.
This is quality of life change that allows for simpler deployments without
the need for bhyvectl --destroy.

Requested by: swills
Reviewed by: 0mp (manpages), grehan, kib, swills
Approved by: kib (mentor)
MFC after: 2 weeks
Sponsored by: Mysterious Code Ltd.
Differential Revision: https://reviews.freebsd.org/D25414

bhyve(8): Add the netgraph network backend decription to the manpage.

Reviewed by: vmaffione, bcr
Approved by: vmaffione (mentor)
Sponsored by: vstack.com
Differential Revision: https://reviews.freebsd.org/D24846

bhyve: update man page to describe the virtio-net mtu option

r359704 introduced an 'mtu' option for the virtio-net device emulation.
Update the man page to describe the new option.

Reviewed by: bcr
Differential Revision: https://reviews.freebsd.org/D24723

Initial support for bhyve save and restore.

Save and restore (also known as suspend and resume) permits a snapshot
to be taken of a guest's state that can later be resumed. In the
current implementation, bhyve(8) creates a UNIX domain socket that is
used by bhyvectl(8) to send a request to save a snapshot (and
optionally exit after the snapshot has been taken). A snapshot
currently consists of two files: the first holds a copy of guest RAM,
and the second file holds other guest state such as vCPU register
values and device model state.

To resume a guest, bhyve(8) must be started with a matching pair of
command line arguments to instantiate the same set of device models as
well as a pointer to the saved snapshot.

While the current implementation is useful for several uses cases, it
has a few limitations. The file format for saving the guest state is
tied to the ABI of internal bhyve structures and is not
self-describing (in that it does not communicate the set of device
models present in the system). In addition, the state saved for some
device models closely matches the internal data structures which might
prove a challenge for compatibility of snapshot files across a range
of bhyve versions. The file format also does not currently support
versioning of individual chunks of state. As a result, the current
file format is not a fixed binary format and future revisions to save
and restore will break binary compatiblity of snapshot files. The
goal is to move to a more flexible format that adds versioning,
etc. and at that point to commit to providing a reasonable level of
compatibility. As a result, the current implementation is not enabled
by default. It can be enabled via the WITH_BHYVE_SNAPSHOT=yes option
for userland builds, and the kernel option BHYVE_SHAPSHOT.

Submitted by: Mihai Tiganus, Flavius Anton, Darius Mihai
Submitted by: Elena Mihailescu, Mihai Carabas, Sergiu Weisz
Relnotes: yes
Sponsored by: University Politehnica of Bucharest
Sponsored by: Matthew Grooms (student scholarships)
Sponsored by: iXsystems
Differential Revision: https://reviews.freebsd.org/D19495

Improve formatting of synopsis section

This patch is about sorting the arguments and using proper mdoc(7) macros
to stylize arguments and command modifiers for much better readability.

Further style fixes in other sections within the bhyve manual page are
going to be worked on in upcoming patches.

Reviewed by: rgrimes
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D24526

Support software breakpoints in the debug server on Intel CPUs.

- Allow the userland hypervisor to intercept breakpoint exceptions
(BP#) in the guest. A new capability (VM_CAP_BPT_EXIT) is used to
enable this feature. These exceptions are reported to userland via
a new VM_EXITCODE_BPT that includes the length of the original
breakpoint instruction. If userland wishes to pass the exception
through to the guest, it must be explicitly re-injected via
vm_inject_exception().

- Export VMCS_ENTRY_INST_LENGTH as a VM_REG_GUEST_ENTRY_INST_LENGTH
pseudo-register. Injecting a BP# on Intel requires setting this to
the length of the breakpoint instruction. AMD SVM currently ignores
writes to this register (but reports success) and fails to read it.

- Rework the per-vCPU state tracked by the debug server. Rather than
a single 'stepping_vcpu' global, add a structure for each vCPU that
tracks state about that vCPU ('stepping', 'stepped', and
'hit_swbreak'). A global 'stopped_vcpu' tracks which vCPU is
currently reporting an event. Event handlers for MTRAP and
breakpoint exits loop until the associated event is reported to the
debugger.

Breakpoint events are discarded if the breakpoint is not present
when a vCPU resumes in the breakpoint handler to retry submitting
the breakpoint event.

- Maintain a linked-list of active breakpoints in response to the GDB
'Z0' and 'z0' packets.

Reviewed by: markj (earlier version)
MFC after: 2 months
Differential Revision: https://reviews.freebsd.org/D20309

Document that the debug server supports writing to guest memory.

This was added in r348212.

Correct name of vmm(4) pptdevs variable.

Reported by: nwolff@ixsystems.com

Replay r349336 by scottl accidentally reverted by r349352

Add a section about the HD Audio module support

Remove NAND and NANDFS support

NANDFS has been broken for years. Remove it. The NAND drivers that
remain are for ancient parts that are no longer relevant. They are
polled, have terrible performance and just for ancient arm
hardware. NAND parts have evolved significantly from this early work
and little to none of it would be relevant should someone need to
update to support raw nand. This code has been off by default for
years and has violated the vnode protocol leading to panics since it
was committed.

Numerous posts to arch@ and other locations have found no actual users
for this software.

Relnotes: Yes
No Objection From: arch@
Differential Revision: https://reviews.freebsd.org/D20745

Add a section about the HD Audio module support

Fix spelling of identifier

Approved by: phk (mentor)
MFC after: 3 days

Allow CTL device specification in bhyve virtio-scsi.

There was a large refactoring done in CTL to allow multiple ioctl frontend
ports (and respective devices) to be created, particularly for bhyve.
Unfortunately, respective part of bhyve functionality got lost somehow from
the original virtio-scsi commit. This change allows wanted device path to
be specified in either of two ways:
-s 6,virtio-scsi,/dev/cam/ctl1.1
-s 6,virtio-scsi,dev=/dev/cam/ctl2.3
If neither is specified, the default /dev/cam/ctl device is used.

While there, remove per-queue CTL device opening, which makes no sense at
this point.

Reported by: wg
Reviewed by: araujo
MFC after: 3 days
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D18504

Provide basic descriptions for VMX exit reason (from "Intel 64 and IA-32
Architectures Software Developer’s Manual Volume 3"). Add the document
to SEE ALSO in bhyve.8 (and pet manlint here a bit).

Reviewed by: jhb, rgrimes, 0mp
Approved by: kib (mentor)
Differential Revision: https://reviews.freebsd.org/D17531

Allow the bhyve VNC server to listen on IPv6 for incoming connections.

Alternatively to IPv4 address:port this will allow to listen on IPv6
link-local (incl. scope), a specific address, or ::. Addresses have
to be given in RFC2732 format so that [::]:port parsing will work.

This patch also starts to introduce WITH_INET/INET6_SUPPORT to bhyve.

PR: 232018
Submitted by: Dave Rush (northwoodlogic.free gmail.com) (original)
Reviewed by: Dave Rush (updated verison)
MFC after: 3 days

Add manpage entry for the new bhyve options -s "help" and -l "help".

Reported by: 0mp
Sponsored by: iXsystems Inc.

Improve bhyve exit(3) error code.

The bhyve(8) exit status indicates how the VM was terminated:

0 rebooted
1 powered off
2 halted
3 triple fault

The problem is when we have wrappers around bhyve that parses the exit
error code and gets an exit(1) for an error but interprets it as "powered off".
So to mitigate this issue and makes it less error prone for third part
applications, I have added a new exit code 4 that is "exited due to an error".

For now the bhyve(8) exit status are:
0 rebooted
1 powered off
2 halted
3 triple fault
4 exited due to an error

Reviewed by: @jhb
MFC after: 2 weeks.
Sponsored by: iXsystems Inc.
Differential Revision: https://reviews.freebsd.org/D16161

- Add bhyve NVMe device emulation.

The initial work on bhyve NVMe device emulation was done by the GSoC student
Shunsuke Mie and was heavily modified in performan, functionality and
guest support by Leon Dang.

bhyve:
-s <n>,nvme,devpath,maxq=#,qsz=#,ioslots=#,sectsz=#,ser=A-Z

accepted devpath:
/dev/blockdev
/path/to/image
ram=size_in_MiB

Tested with guest OS: FreeBSD Head, Linux Fedora fc27, Ubuntu 18.04,
OpenSuse 15.0, Windows Server 2016 Datacenter.
Tested with all accepted device paths: Real nvme, zdev and also with ram.
Tested on: AMD Ryzen Threadripper 1950X 16-Core Processor and
Intel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz.

Tests at: https://people.freebsd.org/~araujo/bhyve_nvme/nvme.txt

Submitted by: Shunsuke Mie <sux2mfgj_gmail.com>,
Leon Dang <leon_digitalmsx.com>
Reviewed by: chuck (early version), grehan
Relnotes: Yes
Sponsored by: iXsystems Inc.
Differential Revision: https://reviews.freebsd.org/D14022

- Add bhyve virtio-scsi storage backend support.

Example of configuration:
ctl.conf:
portal-group pg0 {
discovery-auth-group no-authentication
listen 0.0.0.0
listen [::]
}

target iqn.2012-06.com.example:target0 {
auth-group no-authentication
portal-group pg0
port ioctl/5/3

lun 0 {
path /z/test.img
size 8G
}
lun 1 {
path /z/test1.img
size 8G
}
}

bhyve <...> -s 4,virtio-scsi,/dev/cam/ctl5.3,iid=3 <VM_NAME>

From inside guest:
root@:~ # zpool status test
pool: test
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
test ONLINE 0 0 0
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0

dmesg:
da0 at vtscsi0 bus 0 scbus0 target 0 lun 0
da0: <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
da0: Serial Number MYSERIAL0000
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 8192MB (16777216 512 byte sectors)
da1 at vtscsi0 bus 0 scbus0 target 0 lun 1
da1: <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
da1: Serial Number MYSERIAL0001
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 8192MB (16777216 512 byte sectors)

Discussed with: grehan
Reviewed by: mav
Obtained from: TrueOS
Relnotes: Yes
Sponsored by: iXsystems Inc.
Tested with: FreeBSD HEAD, Fedora 28 (Workstation) and
Ubuntu 18.04.
Differential Revision: https://reviews.freebsd.org/D15276

Initial debug server for bhyve.

This commit adds a new debug server to bhyve. Unlike the existing -g
option which provides an efficient connection to a debug server
running in the guest OS, this debug server permits inspection and
control of the guest from within the hypervisor itself without
requiring any cooperation from the guest. It is similar to the debug
server provided by qemu.

To avoid conflicting with the existing -g option, a new -G option has
been added that accepts a TCP port. An IPv4 socket is bound to this
port and listens for connections from debuggers. In addition, if the
port begins with the character 'w', the hypervisor will pause the
guest at the first instruction until a debugger attaches and
explicitly continues the guest. Note that only a single debugger can
attach to a guest at a time.

Virtual CPUs are exposed to the remote debugger as threads. General
purpose register values can be read for each virtual CPU. Other
registers cannot currently be read, and no register values can be
changed by the debugger.

The remote debugger can read guest memory but not write to guest
memory. To facilitate source-level debugging of the guest, memory
addresses from the debugger are treated as virtual addresses (rather
than physical addresses) and are resolved to a physical address using
the active virtual address translation of the current virtual CPU.
Memory reads should honor memory mapped I/O regions, though the debug
server does not attempt to honor any alignment or size constraints
when accessing MMIO.

The debug server provides limited support for controlling the guest.
The guest is suspended when a debugger is attached and resumes when a
debugger detaches. A debugger can suspend a guest by sending a Ctrl-C
request (e.g. via Ctrl-C in GDB). A debugger can also continue a
suspended guest while remaining attached. Breakpoints are not yet
supported. Single stepping is supported on Intel CPUs that support
MTRAP VM exits, but is not available on other systems.

While the current debug server has limited functionality, it should
at least be usable for basic debugging now. It is also a useful
checkpoint to serve as a base for adding additional features.

Reviewed by: grehan
Differential Revision: https://reviews.freebsd.org/D15022

Add the ability to control the CPU topology of created VMs
from userland without the need to use sysctls, it allows the old
sysctls to continue to function, but deprecates them at
FreeBSD_version 1200060 (Relnotes for deprecate).

The command line of bhyve is maintained in a backwards compatible way.
The API of libvmmapi is maintained in a backwards compatible way.
The sysctl's are maintained in a backwards compatible way.

Added command option looks like:
bhyve -c [[cpus=]n][,sockets=n][,cores=n][,threads=n][,maxcpus=n]
The optional parts can be specified in any order, but only a single
integer invokes the backwards compatible parse. [,maxcpus=n] is
hidden by #ifdef until kernel support is added, though the api
is put in place.

bhyvectl --get-cpu-topology option added.

Reviewed by: grehan (maintainer, earlier version),
Reviewed by: bcr (manpages)
Approved by: bde (mentor), phk (mentor)
Tested by: Oleg Ginzburg <olevole@olevole.ru> (cbsd)
MFC after: 1 week
Relnotes: Y
Differential Revision: https://reviews.freebsd.org/D9930

Bump manpage date.

Add VNC Authentication support based on RFC6143 section 7.2.2.

Submitted by: Fabian Freyer <fabian.freyer@physik.tu-berlin.de>
Reworked by: myself
Reviewed by: grehan, rgrimes and jilles
MFC after: 1 week.
Relnotes: Yes.
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D10818

Improve documentation of fbuf device.

Submitted by: novel
Reviewed by: grehan, bcr
Differential Revision: https://reviews.freebsd.org/D10014

Document raw framebuffer device and XHCI device configurations.

Minor style improvements in bhyve.8

Replace "as of now" with "at present". As the change is a really minor one,
don't bump .Dd.

Suggested by: wblock
Approved by: wblock (implicit)

bhyve: document virtio-console in the manpage

Reviewed by: bcr, wblock, jceel
Approved by: grehan
Differential Revision: https://reviews.freebsd.org/D9564

Typo fixed: arbitraty -> arbitrary.

PR: 213559
Submitted by: olgeni

Add emulation for Intel e1000 (e82545) network adapter.

The code was successfully tested with FreeBSD, Linux, Solaris and Windows
guests. This interface is predictably slower (about 2x) then virtio-net,
but it is very helpful for guests not supporting virtio-net by default.

Thanks to Jeremiah Lott and Peter Grehan for doing original heavy lifting.

Allow AHCI controller to support up to 32 arbitrary devices.

While old syntax is still supported, new syntax looks like this:

-s 3,ahci,hd:/dev/zvol/XXX,hd:/dev/zvol/YYY,cd:/storage/ZZZ.iso

Sponsored by: iXsystems, Inc.

bhyve: improve memory size documentation

A couple of minor memory size option related nits:

- use common name 'memsize' (instead of 'max-size' or just 'size')
- bhyve: update usage with memsize unit suffix, drop legacy "MB"
unit
- bhyveload: update usage with memsize unit suffix
- bhyve(8): document default size
- bhyveload(8): use memsize formatting like it's done
in bhyve(8)

Reviewed by: wblock, grehan
Approved by: re (kib), wblock, grehan
Differential Revision: https://reviews.freebsd.org/D6952

Update the bhyve(8) man page:

- Document powering off by sending SIGTERM signal
- Document exit codes

Reviewed by: wblock, neel
Approved by: wblock
Differential Revision: D5982

Another small typo.

MFC after: 3 days

Fix minor typo.

MFC after: 3 days

Manpage cleanup.

- new sentence -> new line
- fix manpage references
- fix macro usage
- fix a typo

MFC after: 1 week

Restructure memory allocation in bhyve to support "devmem".

devmem is used to represent MMIO devices like the boot ROM or a VESA framebuffer
where doing a trap-and-emulate for every access is impractical. devmem is a
hybrid of system memory (sysmem) and emulated device models.

devmem is mapped in the guest address space via nested page tables similar
to sysmem. However the address range where devmem is mapped may be changed
by the guest at runtime (e.g. by reprogramming a PCI BAR). Also devmem is
usually mapped RO or RW as compared to RWX mappings for sysmem.

Each devmem segment is named (e.g. "bootrom") and this name is used to
create a device node for the devmem segment (e.g. /dev/vmm/testvm.bootrom).
The device node supports mmap(2) and this decouples the host mapping of
devmem from its mapping in the guest address space (which can change).

Reviewed by: tychon
Discussed with: grehan
Differential Revision: https://reviews.freebsd.org/D2762
MFC after: 4 weeks

Allow configuration of the sector size advertised to the guest.

The default behavior is to infer the logical and physical sector sizes from
the block device backend. However older versions of Windows only work with
specific logical/physical combinations:
- Vista and Windows 7: 512/512
- Windows 7 SP1: 512/512 or 512/4096

For this reason allow the sector size to be specified using the following
block device option: sectorsize=logical[/physical]

Reported by: Leon Dang (ldang@nahannisys.com)
Reviewed by: grehan
MFC after: 2 weeks

Add "-u" option to bhyve(8) to indicate that the RTC should maintain UTC time.

The default remains localtime for compatibility with the original device model
in bhyve(8). This is required for OpenBSD guests which assume that the RTC
keeps UTC time.

Reviewed by: grehan
Pointed out by: Jason Tubnor (jason@tubnor.net)
MFC after: 2 weeks

Fix improper .Fx macro usage.

Differential Revision: https://reviews.freebsd.org/D1158
Reviewed by: wblock@
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Update the bhyve(8) manual to reflect that it is no
longer considered 'experimental.'

Reviewed by: grehan
MFC after: 3 days
Sponsored by: The FreeBSD Foundation

Sort command flags in usage output and the manpages.

- Document -b to enable the bvmcons console (but mark it as deprecated
similar to -g.)
- Document -U to set the SMBIOS UUID.
- Add missing options to the usage output and to the manpage Synopsis.
- Don't claim that bvmdebug is amd64-only (it is also a device, not an
option).

use .Mt to mark up email addresses consistently (part2)

PR: 191174
Submitted by: Franco Fichtner <franco@lastsummer.de>

Remove blank lines.

Don't include the guest memory segments in the bhyve(8) process core dump.
This has not added a lot of value when debugging bhyve issues while greatly
increasing the time and space required to store the core file.

Passing the "-C" option to bhyve(8) will change the default and dump guest
memory in the core dump.

Requested by: grehan
Reviewed by: grehan

Modify the "-p" option to be more flexible when associating a 'vcpu' with
a 'hostcpu'. The new format of the argument string is "vcpu:hostcpu".

This allows pinning a subset of the vcpus if desired.

It also allows pinning a vcpu to more than a single 'hostcpu'.

Submitted by: novel (initial version)

Don't allow MPtable generation if there are multiple PCI hierarchies. This is
because there isn't a standard way to relay this information to the guest OS.

Add a command line option "-Y" to bhyve(8) to inhibit MPtable generation.

If the virtual machine is using PCI devices on buses other than 0 then it can
still use ACPI tables to convey this information to the guest.

Discussed with: grehan@

Add support for the virtio RNG entropy-source device.
Call through to /dev/random synchronously to fill
virtio buffers with RNG data.

Tested with FreeBSD-CURRENT and Ubuntu guests.

Submitted by: Leon Dang
Discussed with: markm
MFC after: 3 weeks
Sponsored by: Nahanni Systems

Document the "-a" and "-x" options to match the changes in r262236.

Reviewed by: grehan

Update bhyve(8) man page to describe the usage of the "-s" option to assign
bus numbers to emulated devices. Also add the restriction that the LPC bridge
emulation can only be configured on bus 0.

Reviewed by: grehan@

Remove support for legacy PCI devices. These haven't been needed since
support for LPC uart devices was added and it conflicts with upcoming
patches to add PCI INTx support.

Reviewed by: neel

virtio-block does not exist, the correct name is virtio-blk.

PR: 185573
Submitted by: Allan Jude
Facilitated by: Snow B.V.
MFC after: 3 days

mdoc: nuke whitespace.

Add an option to ignore accesses by the guest to unimplemented MSRs.

Also, ignore a couple of SandyBridge uncore PMC MSRs that Centos 6.4 writes
to during boot.

Reviewed by: grehan

mdoc: sort SEE ALSO.

bhyve(8) man page.

mdoc formatting and much input and review from Warren Block (wblock@).

Reviewed by: many
MFC after: 3 days