Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r267342:

Use bitwise OR instead of logical OR when constructing value for
SET_FEATURES/NUMBER_OF_QUEUES command.

Sponsored by: Intel
Approved by: re (gjb)

MFC r263311:

nvme: Allocate all MSI resources up front so that we can fall back to
INTx if necessary.

MFC r263310:

nvme: Close hole where nvd(4) would not be notified of all nvme(4)
instances if modules loaded during boot.

MFC r263278:

nvme: NVMe specification dictates 4-byte alignment for PRPs (not 8).

MFC r263277:

nvme: Remove the software progress marker SET_FEATURE command during
controller initialization.

The spec says OS drivers should send this command after controller
initialization completes successfully, but other NVMe OS drivers are
not sending this command. This change will therefore reduce differences
between the FreeBSD and other OS drivers.

MFC r260382:

For IDENTIFY passthrough commands to Chatham prototype controllers, copy
the spoofed identify data into the user buffer rather than issuing the
command to the controller, since Chatham IDENTIFY data is always spoofed.

While here, fix a bug in the spoofed data for Chatham submission and
completion queue entry sizes.

MFC r254303:

If a controller fails to initialize, do not notify consumers (nvd) of its
namespaces.

Sponsored by: Intel

MFC r254302:

Send a shutdown notification in the driver unload path, to ensure
notification gets sent in cases where system shuts down with driver
unloaded.

Sponsored by: Intel

MFC r256169 (by jimharris):
Fix the LINT build.

MFC r257534:

Create a unique unit number for each controller and namespace cdev.

Sponsored by: Intel

MFC r256155:

Do not leak resources during attach if nvme_ctrlr_construct() or the initial
controller resets fail.

Sponsored by: Intel

MFC r256154:

Log and then disable asynchronous notification of persistent events after
they occur.

This prevents repeated notifications of the same event.

Status of these events may be viewed at any time by viewing the
SMART/Health Info Page using nvmecontrol, whether or not asynchronous
events notifications for those events are enabled. This log page can
be viewed using:

nvmecontrol logpage -p 2 <ctrlr id>

Future enhancements may re-enable these notifications on a periodic basis
so that if the notified condition persists, it will continue to be logged.

Sponsored by: Intel

MFC r256153:

Do not enable temperature threshold as an asynchronous event notification
on NVMe controllers that do not support it.

Sponsored by: Intel

MFC r256152:

Extend some 32-bit fields and variables to 64-bit to prevent overflow
when calculating data in nvmecontrol perftest.

Sponsored by: Intel

MFC r256151:

Add driver-assisted striping for upcoming Intel NVMe controllers that can
benefit from it.

Sponsored by: Intel

MFC change 254389:

There are some slight differences from the version in FreeBSD/head.
__FreeBSD_version has been bumped to 902503 for the availability
of the SI_UNMAPPED cdev flag, and the D_UNMAPPED_IO cdevsw flag
remains in place. D_UNMAPPED_IO no longer does anything. Drivers that
use that flag will just wind up having mapped I/O by default. The
impact will only be on performance, not functionality.

Change the way that unmapped I/O capability is advertised.

The previous method was to set the D_UNMAPPED_IO flag in the cdevsw
for the driver. The problem with this is that in many cases (e.g.
sa(4)) there may be some instances of the driver that can handle
unmapped I/O and some that can't. The isp(4) driver can handle
unmapped I/O, but the esp(4) driver currently cannot. The cdevsw
is shared among all driver instances.

So instead of setting a flag on the cdevsw, set a flag on the cdev.
This allows drivers to indicate support for unmapped I/O on a
per-instance basis.

sys/conf.h: Remove the D_UNMAPPED_IO cdevsw flag and replace it
with an SI_UNMAPPED cdev flag.

kern_physio.c: Look at the cdev SI_UNMAPPED flag to determine
whether or not a particular driver can handle
unmapped I/O.

geom_dev.c: Set the SI_UNMAPPED flag for all GEOM cdevs.
Since GEOM will create a temporary mapping when
needed, setting SI_UNMAPPED unconditionally will
work.

Remove the D_UNMAPPED_IO flag.

nvme_ns.c: Set the SI_UNMAPPED flag on cdevs created here
if NVME_UNMAPPED_BIO_SUPPORT is enabled.

vfs_aio.c: In aio_qphysio(), check the SI_UNMAPPED flag on a
cdev instead of the D_UNMAPPED_IO flag on the cdevsw.

sys/param.h: Bump __FreeBSD_version to 1000045 for the switch from
setting the D_UNMAPPED_IO flag in the cdevsw to setting
SI_UNMAPPED in the cdev.

Reviewed by: kib, jimharris
Sponsored by: Spectra Logic

MFC r253476:

Add message when nvd disks are attached and detached.

As part of this commit, add an nvme_strvis() function which borrows
heavily from cam_strvis(). This will allow stripping of
leading/trailing whitespace and also handle unprintable characters
in model/serial numbers. This function goes into a new nvme_util.c
file which is used by both the driver and nvmecontrol.

Approved by: re (kib)
Sponsored by: Intel

MFC r253474:

Fix nvme(4) and nvd(4) to support non 512-byte sector sizes.

Recent testing with QEMU that has variable sector size support for
NVMe uncovered some of these issues. Chatham prototype boards supported
only 512 byte sectors.

Approved by: re (kib)
Sponsored by: Intel

MFC r253438:

Use pause() instead of DELAY() when polling for completion of admin
commands during controller initialization.

DELAY() does not work here during config_intrhook context - we need to
explicitly relinquish the CPU for the admin command completion to
get processed.

Approved by: re (kib)
Sponsored by: Intel

MFC r253437:

Define constants for the lengths of the serial number, model number
and firmware revision in the controller's identify structure.

Also modify consumers of these fields to ensure they only use the
specified number of bytes for their respective fields.

Approved by: re (kib)
Sponsored by: Intel

MFC r253209:

Fix a poorly worded comment in nvme(4).

Approved by: re (kib)

MFC r253113:

Add comment explaining why CACHE_LINE_SIZE is defined in nvme_private.h
if not already defined elsewhere.

Approved by: re (kib)

MFC r253112:

Update copyright dates.

Approved by: re (kib)

MFC r253108:

Do not retry failed async event requests.

Approved by: re (kib)

MFC r253107:

Add pci_enable_busmaster() and pci_disable_busmaster() calls in
nvme_attach() and nvme_detach() respectively.

Approved by: re (kib)

MFC r252278:

Add firmware replacement and activation support to nvmecontrol(8) through
a new firmware command.

NVMe controllers may support up to 7 firmware slots for storing of
different firmware revisions. This new firmware command supports
firmware replacement (i.e. firmware download) with or without immediate
activation, or activation of a previously stored firmware image. It
also supports selection of the firmware slot during replacement
operations, using IDENTIFY information from the controller to
check that the specified slot is valid.

Newly activated firmware does not take effect until the new controller
reset, either via a reboot or separate 'nvmecontrol reset' command to the
same controller.

Submitted by: Joe Golio <joseph.golio@emc.com>
Obtained from: EMC / Isilon Storage Division

MFC r252273:

Remove remaining uio-related code.

The nvme_physio() function was removed quite a while ago, which was the
only user of this uio-related code.

Sponsored by: Intel

MFC r252272:

Fail any passthrough command whose transfer size exceeds the controller's
max transfer size. This guards against rogue commands coming in from
userspace.

Also add KASSERTS for the virtual address and unmapped bio cases, if the
transfer size exceeds the controller's max transfer size.

Sponsored by: Intel

MFC r252271:

Use MAXPHYS to specify the maximum I/O size for nvme(4).

Also allow admin commands to transfer up to this maximum I/O size, rather
than the artificial limit previously imposed. The larger I/O size is very
beneficial for upcoming firmware download support. This has the added
benefit of simplifying the code since both admin and I/O commands now use
the same maximum I/O size.

Sponsored by: Intel

240618, 240621, 240633, 240671, 240672, 240697, 240700, 241433,
241434, 241657, 241658, 241659, 241660, 241661, 241662, 241663,
241664, 241665, 241689, 242420, 243951, 244410, 244411, 244413,
244549, 245136, 247963, 248729, 248730, 248731, 248732, 248733,
248734, 248735, 248736, 248737, 248738, 248739, 248740, 248741,
248746, 248747, 248748, 248749, 248754, 248755, 248756, 248757,
248758, 248759, 248760, 248761, 248762, 248763, 248764, 248765,
248766, 248767, 248768, 248769, 248770, 248771, 248772, 248773,
248780, 248834, 248835, 248913, 248977, 249067, 249416, 249417,
249418, 249419, 249420, 249421, 249422, 249432

MFC r240616, r240619, r240620:

Merge initial NVM Express (NVMe) commits to stable/9.

This only pulls the nvme(4), nvd(4) and nvmecontrol(8) code into stable/9.
A follow-up commit will link it into the build and merge all of the
additional commits made for NVMe over the last few months.

Sponsored by: Intel

This is the first of several commits which will add NVM Express (NVMe)
support to FreeBSD. A full description of the overall functionality
being added is below. nvmexpress.org defines NVM Express as "an optimized
register interface, command set and feature set fo PCI Express (PCIe)-based
Solid-State Drives (SSDs)."

This commit adds nvme(4) and nvd(4) driver source code and Makefiles
to the tree.

Full NVMe functionality description:
Add nvme(4) and nvd(4) drivers and nvmecontrol(8) for NVM Express (NVMe)
device support.

There will continue to be ongoing work on NVM Express support, but there
is more than enough to allow for evaluation of pre-production NVM Express
devices as well as soliciting feedback. Questions and feedback are welcome.

nvme(4) implements NVMe hardware abstraction and is a provider of NVMe
namespaces. The closest equivalent of an NVMe namespace is a SCSI LUN.
nvd(4) is an NVMe consumer, surfacing NVMe namespaces as GEOM disks.
nvmecontrol(8) is used for NVMe configuration and management.

The following are currently supported:
nvme(4)
- full mandatory NVM command set support
- per-CPU IO queues (enabled by default but configurable)
- per-queue sysctls for statistics and full command/completion queue
dumps for debugging
- registration API for NVMe namespace consumers
- I/O error handling (except for timeoutsee below)
- compilation switches for support back to stable-7

nvd(4)
- BIO_DELETE and BIO_FLUSH (if supported by controller)
- proper BIO_ORDERED handling

nvmecontrol(8)
- devlist: list NVMe controllers and their namespaces
- identify: display controller or namespace identify data in
human-readable or hex format
- perftest: quick and dirty performance test to measure raw
performance of NVMe device without userspace/physio/GEOM
overhead

The following are still work in progress and will be completed over the
next 3-6 months in rough priority order:
- complete man pages
- firmware download and activation
- asynchronous error requests
- command timeout error handling
- controller resets
- nvmecontrol(8) log page retrieval

This has been primarily tested on amd64, with light testing on i386. I
would be happy to provide assistance to anyone interested in porting
this to other architectures, but am not currently planning to do this
work myself. Big-endian and dmamap sync for command/completion queues
are the main areas that would need to be addressed.

The nvme(4) driver currently has references to Chatham, which is an
Intel-developed prototype board which is not fully spec compliant.
These references will all be removed over time.

Sponsored by: Intel
Contributions from: Joe Golio/EMC <joseph dot golio at emc dot com>