usr.sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

bhyve: add bootindex option for several devices

The bootindex option creates an entry in the "bootorder" fwcfg file.
This file can be picked up by the guest firmware to determine the
bootorder. Nevertheless, it's not guaranteed that the guest firmware
uses the bootorder. At the moment, our OVMF ignores the bootorder. This
will change in the future.

If guest firmware supports the "bootorder" fwcfg file and no device uses
the bootindex option, the boot order is determined by the firmware
itself. If one or more devices specify a bootindex, the first bootable
device with the lowest bootindex will be booted. It's not garanteed that
devices without a bootindex will be recognized as bootable from the
firmware in that case.

Reviewed by: jhb
MFC after: 1 week
Sponsored by: Beckhoff Automation GmbH & Co. KG
Differential Revision: https://reviews.freebsd.org/D39285

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

bhyve: Remove vmctx argument from PCI device model methods.

Most of these arguments were unused. Device models which do need
access to the vmctx in one of these methods can obtain it from the
pi_vmctx member of the pci_devinst argument instead.

Reviewed by: corvink, markj
Differential Revision: https://reviews.freebsd.org/D38096

CTL: Allow userland supply tags via ioctl frontend.

Before this ioctl frontend always replaced tags with sequential ones.
It was done for ctladm, that can not keep track of global tag list.
But in case of virtio-scsi in bhyve we can pass provided tags as-is.
It should be on virtio-scsi initiator to provide us valid tags. It
should allow proper task management, error reporting, etc. In case
of several virtio-scsi devices, they should use different CTL ports
or initiator IDs to avoid conflicts, but this is expected by design.

PR: 267539

CTL: Increase maximum SCSI tag size from 32 to 64 bits.

SAM-5 specification states maximum size of command identifier (tag),
defined by specific transports, should not be larger than 64 bits.
While most of supported transports use 32 bits or less, it was
reported that virtio-scsi uses 64 bits. Truncation to 32 bits in
bhyve code caused false tag conflict errors reported and possibly
other issues.

This changes CTL ABI and HA protocol, so CTL_HA_VERSION is bumped.

While we make HA protocol incompatible, increase default maximum
number of ports in CTL from 256 to 1024, matching number of LUNs.
There are many reports from people who need many iSCSI targets with
only one LUN each. Increased memory consumption should be less of
a problem these days.

PR: 267539

bhyve virtio-scsi: Fix residual reporting.

CTL does not really use residual field and it always returned zero.
Use ext_data_filled instead.

MFC after: 2 weeks

bhyve: Avoid arithmetic on void pointers

No functional change intended.

MFC after: 1 week

bhyve: Annotate unused function parameters

MFC after: 1 week

bhyve: Use designated initializers for virtio_consts tables

This is easier to read and addresses some compiler warnings.

One might expect these tables to be read-only but it seems that the
snapshot/restore code may modify them.

MFC after: 2 weeks

bhyve virtio-scsi: Avoid out of bounds accesses to guest requests.

- Ignore I/O requests with insufficiently sized input or output
buffers (those not containing compete request headers).

- Ignore control requests with improperly sized buffers.

- While here, explicitly zero the output header of an I/O request to
avoid leaking malloc garbage from the host if the header is not
fully populated.

PR: 264521
Reported by: Robert Morris <rtm@lcs.mit.edu>
Reviewed by: mav, emaste
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36271

bhyve virtio-scsi: Tidy warning and debug prints.

Use a consistent prefix ("virtio-scsi: ") similar to the e1000 device
model.

Reviewed by: mav, emaste
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D36270

bhyve: Mark variables and functions as static where appropriate

Mark them const as well when it makes sense to do so. No functional
change intended.

MFC after: 1 week
Sponsored by: The FreeBSD Foundation

bhyve: missing mutex initializations

Explicitly initialize the mutex that a PCI virtio module passes back to
virtio.

It so happens that these mutexes were being initialized regardless, no
functional change intended.

Reviewed by: chuck, jhb
Differential Revision: https://reviews.freebsd.org/D34372

bhyve: clean up trailing whitespaces

Clean up trailing whitespaces. No functional changes.

Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D33681

bhyve: Nuke double-semicolons

A distinct number of double-semicolons ended up in bhyve. Take a pass at
getting rid of many of these harmless typos.

MFC after: 3 days

bhyve: Fix vq_getchain() error handling bugs in various device models

Reviewed by: grehan, khng
Approved by: so
Security: CVE-2021-29631
Security: FreeBSD-SA-21:13.bhyve

bhyve: Gracefully handle virtio-scsi with no conf

Fixes segfault with the command `bhyve -s 0,virtio-scsi`, which is used
by some third party software to probe bhyve for virtio-scsi support.

Reviewed by: jhb
MFC after: 1 day
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D29926

bhyve: change vq_getchain to return iovecs in both directions

The old prototype requires callers to inspect flags of each descriptors
to get the starting position of host-writable iovecs.

vq_getchain() is changed to return a virtio request with the number of
host-readable iovecs and host-writable iovecs instead. Callers can avoid
boilerplate code of getting the start offset of host-writable iovecs.

Sponsored by: The FreeBSD Foundation
MFC after: 3 weeks
Reviewed by: afedorov
Approved by: philip (mentor)
Differential Revision: https://reviews.freebsd.org/D29433

bhyve: Enable virtio-scsi legacy config parsing.

The previous commit added the handler to parse the command line
options for virtio-scsi devices but forgot to set the correct function
pointer to point to the handler.

Reported by: vangyzen
Reviewed by: vangyzen
Fixes: 621b5090487de9fed1b503769702a9a2a27cc7bb
Differential Revision: https://reviews.freebsd.org/D29438

Refactor configuration management in bhyve.

Replace the existing ad-hoc configuration via various global variables
with a small database of key-value pairs. The database supports
heirarchical keys using a MIB-like syntax to name the path to a given
key. Values are always stored as strings. The API used to manage
configuation values does include wrappers to handling boolean values.
Other values use non-string types require parsing by consumers.

The configuration values are stored in a tree using nvlists. Leaf
nodes hold string values. Configuration values are permitted to
reference other configuration values using '%(name)'. This permits
constructing template configurations.

All existing command line arguments now set configuration values. For
devices, the "-s" option parses its option argument to generate a list
of key-value pairs for the given device.

A new '-o' command line option permits setting an individual
configuration variable. The key name is always given as a full path
of dot-separated components.

A new '-k' command line option parses a simple configuration file.
This configuration file holds a flat list of 'key=value' lines where
the 'key' is the full path of a configuration variable. Lines
starting with a '#' are comments.

In general, bhyve starts by parsing command line options in sequence
and applying those settings to configuration values. Once this is
complete, bhyve then begins initializing its state based on the
configuration values. This means that subsequent configuration
options or files may override or supplement previously given settings.

A special 'config.dump' configuration value can be set to true to help
debug configuration issues. When this value is set, bhyve will print
out the configuration variables as a flat list of 'key=value' lines.

Most command line argments map to a single configuration variable,
e.g. '-w' sets the 'x86.strictmsr' value to false. A few command
line arguments have less obvious effects:

- Multiple '-p' options append their values (as a comma-seperated
list) to "vcpu.N.cpuset" values (where N is a decimal vcpu number).

- For '-s' options, a pci.<bus>.<slot>.<function> node is created.
The first argument to '-s' (the device type) is used as the value of
a "device" variable. Additional comma-separated arguments are then
parsed into 'key=value' pairs and used to set additional variables
under the device node. A PCI device emulation driver can provide
its own hook to override the parsing of the additonal '-s' arguments
after the device type.

After the configuration phase as completed, the init_pci hook
then walks the "pci.<bus>.<slot>.<func>" nodes. It uses the
"device" value to find the device model to use. The device
model's init routine is passed a reference to its nvlist node
in the configuration tree which it can query for specific
variables.

The result is that a lot of the string parsing is removed from
the device models and centralized. In addition, adding a new
variable just requires teaching the model to look for the new
variable.

- For '-l' options, a similar model is used where the string is
parsed into values that are later read during initialization.
One key note here is that the serial ports use the commonly
used lowercase names from existing documentation and examples
(e.g. "lpc.com1") instead of the uppercase names previously
used internally in bhyve.

Reviewed by: grehan
MFC after: 3 months
Differential Revision: https://reviews.freebsd.org/D26035

bhyve: virtio shares definitions between sys/dev/virtio

Definitions inside usr.sbin/bhyve/virtio.h are thrown away.
Definitions in sys/dev/virtio are used instead.

This reduces code duplication.

Sponsored by: The FreeBSD Foundation
Reviewed by: grehan
Approved by: philip (mentor)
Differential Revision: https://reviews.freebsd.org/D29084

bhyve: add wrapper for debug printf statements

Add printf() wrapper to use CR/CRLF terminators depending on whether
stdio is mapped to a tty open in raw mode.
Try to use the wrapper everywhere.
For now we leave the custom DPRINTF/WPRINTF defined by device
models, but we may remove them in the future.

Reviewed by: grehan, jhb
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D22657

bhyve: uniform printf format string newlines

Some of the printf statements only use LF to get a newline. However, a CR character is also required for the serial console to print debug logs in a nice way.
Fix those code locations that only use LF, by adding a CR character.

Reviewed by: markj, aleksandr.fedorov@itglobal.com
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D22552

usr.sbin/bhyve: prevent use-after-free in virtio scsi request handling

Coverity CID: 1393377
Approved by: araujo, jhb
Differential Revision: https://reviews.freebsd.org/D20915

bhyve: Fix vtscsi maximum segment config

The seg_max value reported to the guest should be two less than the
host's maximum, in order to leave room for the request and the
response. This is analogous to r347033 for virtio_block.

We hit the "too many segments to enqueue" assertion on OneFS because
we increase MAXPHYS to 256 KB.

Reviewed by: bryanv
Discussed with: cem jhb rgrimes
MFC after: 1 week
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D20529

bhyve: virtio: introduce vq_kick_enable() and vq_kick_disable()

The VirtIO standard supports two schemes for notification suppression:
a notification enable bit and a more sophisticated one (event_idx) that
also supports delayed notifications. Currently bhyve fully supports
only the first scheme. This patch hides the notification suppression
internals by means of two inline routines, vq_kick_enable() and
vq_kick_disable(), and makes the code more readable.
Moreover, further improve readability by replacing the call to mb()
with a call to atomic_thread_fence_seq_cst(), which is already used
in virtio.c

Reviewed by: pmooney_pfmooney.com, bryanv
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D20581

Fix thread name buffer overflow.

MFC after: 1 week

Allow CTL device specification in bhyve virtio-scsi.

There was a large refactoring done in CTL to allow multiple ioctl frontend
ports (and respective devices) to be created, particularly for bhyve.
Unfortunately, respective part of bhyve functionality got lost somehow from
the original virtio-scsi commit. This change allows wanted device path to
be specified in either of two ways:
-s 6,virtio-scsi,/dev/cam/ctl1.1
-s 6,virtio-scsi,dev=/dev/cam/ctl2.3
If neither is specified, the default /dev/cam/ctl device is used.

While there, remove per-queue CTL device opening, which makes no sense at
this point.

Reported by: wg
Reviewed by: araujo
MFC after: 3 days
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D18504

Make virtio-scsi pass SCSI Task Attributes to CTL.

MFC after: 1 week
Sponsored by: iXsystems, Inc.

Fix several iov handling bugs in bhyve virtio-scsi backend.

- buf_to_iov() does not use buflen parameter, allowing out of bound read.
- buf_to_iov() leaks memory if seek argument > 0.
- iov_to_buf() doesn't need to reallocate buffer for every segment.
- there is no point to use size_t for iov counts, int is more then enough.
- some iov function arguments can be constified.
- pci_vtscsi_request_handle() used truncate_iov() incorrectly, allowing
getting out of buffer and possibly corrupting data.
- pci_vtscsi_controlq_notify() written returned status at wrong offset.
- pci_vtscsi_controlq_notify() leaked one buffer per event.

Reported by: wg
Reviewed by: araujo
MFC after: 1 week
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D18465

Fill initid explicitly on requests.

Unfortunately ctl_scsi_zero_io() wipes that field, so it was always zero.
While there, targ_port is set by kernel, so user-space should not fill it.

MFC after: 1 week

style(9): Fix tabs after #define.

No functional change intended.

- Add bhyve virtio-scsi storage backend support.

Example of configuration:
ctl.conf:
portal-group pg0 {
discovery-auth-group no-authentication
listen 0.0.0.0
listen [::]
}

target iqn.2012-06.com.example:target0 {
auth-group no-authentication
portal-group pg0
port ioctl/5/3

lun 0 {
path /z/test.img
size 8G
}
lun 1 {
path /z/test1.img
size 8G
}
}

bhyve <...> -s 4,virtio-scsi,/dev/cam/ctl5.3,iid=3 <VM_NAME>

From inside guest:
root@:~ # zpool status test
pool: test
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
test ONLINE 0 0 0
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0

dmesg:
da0 at vtscsi0 bus 0 scbus0 target 0 lun 0
da0: <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
da0: Serial Number MYSERIAL0000
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 8192MB (16777216 512 byte sectors)
da1 at vtscsi0 bus 0 scbus0 target 0 lun 1
da1: <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-5 SCSI device
da1: Serial Number MYSERIAL0001
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 8192MB (16777216 512 byte sectors)

Discussed with: grehan
Reviewed by: mav
Obtained from: TrueOS
Relnotes: Yes
Sponsored by: iXsystems Inc.
Tested with: FreeBSD HEAD, Fedora 28 (Workstation) and
Ubuntu 18.04.
Differential Revision: https://reviews.freebsd.org/D15276