nvmecontrol: Allow optional /dev/ for device names

nvmecontrol operates on devices. Allow a user to specify the /dev/ if
they want. Any device that starts with / will be treated as if it was a
full path for maximum flexbility.

Sponsored by: Netflix

sbin: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

Make use of the getlocalbase() function for run-time adjustment of the
local software base directory, as committed in SVN rev. 367813.

The pkg and mailwrapper programs used the LOCALBASE environment variable
for this purpose and this functionality is preserved by getlocalbase().

After this change, the value of the user.localbase sysctl variable is used
if present (and not overridden in the environment).

The nvmecontrol program gains support of a dynamic path to its plugin
directory with this update.

Differential Revision: https://reviews.freebsd.org/D27237

Revert the whole getlocalbase() set of changes while a different design is
hashed out.

Fix the previous revision, it suffered from an incomplete change to the
getlocalbase API. Also don't erroneously subtract the lenth from the
buffer a second time.

Replace hardcoded references to _PATH_LOCALBASE with calls to getlocalbase.3

Reviewed by: imp, se

Improve nvmecontrol error reporting.

MFC after: 1 week
Sponsored by: iXsystems, Inc.

Replace literal uses of /usr/local in C sources with _PATH_LOCALBASE

Literal references to /usr/local exist in a large number of files in
the FreeBSD base system. Many are in contributed software, in configuration
files, or in the documentation, but 19 uses have been identified in C
source files or headers outside the contrib and sys/contrib directories.

This commit makes it possible to set _PATH_LOCALBASE in paths.h to use
a different prefix for locally installed software.

In order to avoid changes to openssh source files, LOCALBASE is passed to
the build via Makefiles under src/secure. While _PATH_LOCALBASE could have
been used here, there is precedent in the construction of the path used to
a xauth program which depends on the LOCALBASE value passed on the compiler
command line to select a non-default directory.

This could be changed in a later commit to make the openssh build
consistently use _PATH_LOCALBASE. It is considered out-of-scope for this
commit.

Reviewed by: imp
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D26942

Open device with O_RDONLY when command is non-invasive.

This allows to use some of the subcommands against mounted nvd devices.

MFC after: 1 week
Sponsored by: iXystems, Inc.

Add IOCTL to translate nvdX into nvmeY and NSID.

While very useful by itself, it also makes `nvmecontrol` not depend on
hardcoded device names parsing, that in its turn makes simple to take
nvdX (and potentially any other) device names as arguments.

Also added IOCTL bypass from nvdX to respective nvmeYnsZ makes them
interchangeable for management purposes.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Feature-complete NVMe Namespace Management.

This adds several previously missed but important subcommands to list
namespaces and controllers. It also fixes few previously added but
just found with real testing to be broken subcommands.

Also while there, add possibility to explicitly specify nsid for
`nvmecontrol identify` subcommand. It may be useful to specify nsids
not having own devices, for example 0xffffffff, or just newly created
ones.

MFC after: 2 weeks
Relnotes: yes
Sponsored by: iXsystems, Inc.

Create generic command / arg parsing routines

Create a set of routines and structures to hold the data for the args
for a command. Use them to generate help and to parse args. Convert
all the current commands over to the new format. "comnd" is a hat-tip
to the TOPS-20 %COMND JSYS that (very) loosely inspired much of the
subsequent command line notions in the industry, but this is far
simpler (the %COMND man page is longer than this code) and not in the
kernel... Also, it implements today's de-facto
command [verb]+ [opts]* [args]*
format rather than the old, archaic TOPS-20 command format :)

This is a snapshot of a work in progress to get the nvme passthru
stuff committed. In time it will become a private library and used
by some other programs in the tree that conform to the above pattern.

Differential Revision: https://reviews.freebsd.org/D19296

Rework logpage extensibility.

Move from using a linker set to a constructor function that's
called. This simplifies the code and is slightly more obvious. We now
keep a list of page decoders rather than having an array we managed
before. Commands will move to something similar in the future.

Reviewed by: jhb@
Differential Revision: https://reviews.freebsd.org/D19275

Const poison the command interface

Make the pointers we pass into the commands const, also make the
linker set mirrors const.

Suggested by: cem@
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D18459

Dynamically load .so modules to expand functionality

o Dynamically load all the .so files found in /libexec/nvmecontrol and
/usr/local/libexec/nvmecontrol.
o Link nvmecontrol -rdynamic so that its symbols are visible to the
libraries we load.
o Create concatinated linker sets that we dynamically expand.
o Add the linked-in top and logpage linker sets to the mirrors for them
and add those sets to the mirrors when we load a new .so.
o Add some macros to help hide the names of the linker sets.
o Update the man page.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D18455

fold

Usage cleanup pt 2

Eliminage redundant spaces and nvmecontrol at start of all the usage
strings. Update the usage printing code to add them back when
presenting to the user. Allow multi-line usage messages and print
proper leading spaces for lines starting with a space.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D18403

Usage cleanup pt 1

Provide a usage() function that takes a struct nvme_function pointer
and produces a usage mssage. Eliminate all now-redundant usage
functions. Propigate the new argument through the program as needed.
Use common routine to print usage.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D18403

Return after we find the dispatched function.

If the dispatched function doesn't exit, then we get can get a
spurious function not found message. They all do exit, but this is a
little cleaner.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D18403

Move nvmecontrol to using linker sets for commands

More commands will be added to nvmecontrol. Also, there will be a few
more vendor commands (some of which may need to remain private to
companies writing them). The first step on that journey is to move to
using linker sets to dispatch commands. The next step will be using
dlopen to bring in the .so's that have the command that might need
to remain private for seamless integration.

Similar changes to this will be needed for vendor specific log pages.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D18403

Make NVMe compatible with the original API

The original NVMe API used bit-fields to represent fields in data
structures defined by the specification (e.g. the op-code in the command
data structure). The implementation targeted x86_64 processors and
defined the bit fields for little endian dwords (i.e. 32 bits).

This approach does not work as-is for big endian architectures and was
changed to use a combination of bit shifts and masks to support PowerPC.
Unfortunately, this changed the NVMe API and forces #ifdef's based on
the OS revision level in user space code.

This change reverts to something that looks like the original API, but
it uses bytes instead of bit-fields inside the packed command structure.
As a bonus, this works as-is for both big and little endian CPU
architectures.

Bump __FreeBSD_version to 1200081 due to API change

Reviewed by: imp, kbowling, smh, mav
Approved by: imp (mentor)
Differential Revision: https://reviews.freebsd.org/D16404

Add NVMe Namespace Management support to nvmecontrol(8).

This allows create/delete/attach/detach namespaces on new NVMe controllers.
This is only a first user-level part of the bigger change set. Kernel part
required to detect and handle the configuration changes without reboot is
completely independent and will be added separately.

Submitted by: Matt Williams <mffbsdw@gmail.com> (original version)
Differential Revision: https://reviews.freebsd.org/D11399

Add `nvmecontrol format` subcommand.

It allows to change namespace parameters, such as block size, metadata,
protection information, etc. and/or erase the data.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

NVMe: Add big-endian support

Remove bitfields from defined structures as they are not portable.
Instead use shift and mask macros in the driver and nvmecontrol application.

NVMe is now working on powerpc64 host.

Submitted by: Michal Stanek <mst@semihalf.com>
Obtained from: Semihalf
Reviewed by: imp, wma
Sponsored by: IBM, QCM Technologies
Differential revision: https://reviews.freebsd.org/D13916

various: general adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

No functional change intended.

Exit with usage if argv[1] is NULL in dispatch. This fixes core dumps
when a command has subcommands, but the user doesn't give the
parameters on the command line.

Sponsored by: Netflix

Implement 5 wdc-specific nvme control options for their HGST drives:
wdc cap-diag Capture diagnostic data from drive
wdc drive-log Capture drive history data from drive
wdc get-crash-dump Retrieve firmware crash dump from drive

Move the usage and command name lookup into functions.

Implement power command to list all power modes, find out the power
mode we're in and to set the power mode.

Revert r295062 to unbreak buildworld

Some of the structures referenced in power.c (in particular
struct nvme_power_state) are missing from sbin/nvmecontrol/

Pointyhat to: imp
Reported by: Jenkins [*], O. Hartmann <ohartman@zedat.fu-berlin.de>, Outback Dingo <outbackdingo@gmail.com>

Implement power command to list all power modes, find out the power
mode we're in and to set the power mode.

Use _PATH_DEV (from paths.h) for the "/dev/" string, rather than
hard-coding it.

Sponsored by: Intel
Suggested by: kib
Reviewed by: kib, carl
MFC after: 3 days

Simplify open_dev() by returning errno values rather than just 0 or 1.

Also remove stat() call and just rely on errno from open() call to discern
whether dev node exists or not.

Sponsored by: Intel
Reviewed by: kib, carl
MFC after: 3 days

Always initialize fd to 0 in open_dev().

Sponsored by: Intel
Reviewed by: carl
MFC after: 3 days

Send per-namespace logpage commands to the controller devnode, so they
are processed as admin commands, not I/O commands.

As part of this change, pull out the code for parsing a namespace node
string into a separate function, since it is used for both identify and
logpage commands.

Sponsored by: Intel
MFC after: 3 days

Incorporate feedback from bde@ based on r252672 changes:

* Use 0/1 instead of sysexits. Man pages are confusing on this topic,
but 0/1 is sufficient for nvmecontrol.
* Use err function family where possible instead of fprintf/exit.
* Fix some typing errors.
* Clean up some error message inconsistencies.

Sponsored by: Intel
Submitted by: bde (parts of firmware.c changes)
MFC after: 3 days

Fix build.

Add firmware replacement and activation support to nvmecontrol(8) through
a new firmware command.

NVMe controllers may support up to 7 firmware slots for storing of
different firmware revisions. This new firmware command supports
firmware replacement (i.e. firmware download) with or without immediate
activation, or activation of a previously stored firmware image. It
also supports selection of the firmware slot during replacement
operations, using IDENTIFY information from the controller to
check that the specified slot is valid.

Newly activated firmware does not take effect until the new controller
reset, either via a reboot or separate 'nvmecontrol reset' command to the
same controller.

Submitted by: Joe Golio <joseph.golio@emc.com>
Obtained from: EMC / Isilon Storage Division
MFC after: 3 days

Add log page support to nvmecontrol(8) through a new logpage command.

This includes pretty printers for all of the standard NVMe log pages
(Error, SMART/Health, Firmware), as well as hex output for non-standard
or vendor-specific log pages.

Submitted by: Joe Golio <joseph.golio@emc.com>
Obtained from: EMC / Isilon Storage Division
MFC after: 3 days

Add generalized helper functions for printing hex data as part of
nvmecontrol commands.

Submitted by: Joe Golio <joseph.golio@emc.com>
Obtained from: EMC / Isilon Storage Division
MFC after: 3 days

Ensure that a device node is specified when invoking the identify or reset
commands.

Also improve the checking of device node names, so that better error
messages are displayed when incorrect names are specified.

Sponsored by: Intel
MFC after: 3 days

Add an nvme_function structure array, defining the name, C function and
usage message for each nvmecontrol command. This helps reduce some code
clutter both now and for future commits which will add logpage and
firmware support to nvmecontrol(8).

Also move helper function prototypes to the end of the header file, after
the per-command functions.

Sponsored by: Intel
MFC after: 3 days

Break out code related to the reset command into a separate source file.

Sponsored by: Intel
MFC after: 3 days

Break out code related to the perftest command into a separate file.

Sponsored by: Intel
MFC after: 3 days

Break out code related to the identify command into a separate source file.

Sponsored by: Intel
MFC after: 3 days

Break out code related to the devlist command into a separate source file.

Sponsored by: Intel
MFC after: 3 days

Add separate *_usage() functions for each nvmecontrol command.

Sponsored by: Intel
MFC after: 3 days

Add an open_dev() helper function to be used in the various places
where nvmecontrol needs to open a handle to a controller or namespace
device node.

Sponsored by: Intel
MFC after: 3 days

Remove the NVME_IDENTIFY_CONTROLLER and NVME_IDENTIFY_NAMESPACE IOCTLs and replace
them with the NVMe passthrough equivalent.

Sponsored by: Intel

Fix typo (devicde -> device).

Use errno and strerror to print more descriptive messages when operations
fail in nvmecontrol(8).

While here, use consistent checks of return values from stat, open and
ioctl.

Sponsored by: Intel
Suggested by: carl
Reviewed by: carl

Add controller reset capability to nvme(4) and ability to explicitly
invoke it from nvmecontrol(8).

Controller reset will be performed in cases where I/O are repeatedly
timing out, the controller reports an unrecoverable condition, or
when explicitly requested via IOCTL or an nvme consumer. Since the
controller may be in such a state where it cannot even process queue
deletion requests, we will perform a controller reset without trying
to clean up anything on the controller first.

Sponsored by: Intel
Reviewed by: carl

Add nvmecontrol(8) source code and beginnings of a man page to the tree.

Sponsored by: Intel
Contributions from: Joe Golio/EMC <joseph dot golio at emc dot com>