cam: Check if cam_simq_alloc fails for the xpt bus during module init

This is very unlikely to fail (and if it does, CAM isn't going to work
regardless), but fail with an error rather than a gauranteed panic via
NULL pointer dereference.

PR: 276770
Reported by: Qiushi <w290680224@gmail.com>

cam: Let cam_periph_unmapmem() return an error

As of commit b059686a71c8, cam_periph_unmapmem() can legitimately fail
if the copyout() operation fails. However, this failure was never
signaled to upper layers. In practice it is unlikely to occur
since cap_periph_mapmem() would most likely fail in such
circumstances anyway, but an error is nonetheless possible.

However, some code reading revealed a few paths where the return value
of cam_periph_mapmem() is not checked, and this is definitely a bug.
Add error checking there and let cam_periph_unmapmem() return errors
from copyout().

Reviewed by: dab, mav
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D43201

CAM: Replace random sbuf_printf() with cheaper cat/putc.

CAM: Remove return value from xpt_path_sbuf()

It is wrong to call sbuf_len() on third-party sbuf. If that sbuf
has a drain function, it ends up in assertion. But even would it
work, it would return not newly written length, but the full one.
Searching through the sources I don't see this value used.

cam: Remove left-over sys/cdefs.h in sys/cam

These weren't removed when $FreeBSD$ was removed. They aren't needed and
now are a style(9) nonconformity.

Sponsored by: Netflix

cam: Minor opt_cam.h cleanup

sys/cam/cam.h includes opt_cam.h, so none of the clients need to do
this. cam.h does all the right dancing to conditionally include
opt_cam.h only when it makes sense. It generally only matters when
cam_debug.h is included (it must be included before that). Many of the
stray opt_cam.h includes were after cam_debug.h which would be a problem
were it not included in cam/cam.h. The other users of CAM options that
aren't debug all already include cam/cam.h.

Also trim unneeded sys/cdefs.h files from the files touched.

Sponsored by: Netflix

cam: Make cam_debug macros atomic

The CAM_DEBUG* macros use multiple printfs to dump the data. This is
suboptimal when tracing things that produce even a moderate amount since
it gets intertwingled. I can't even turn on tracing with a 24-disk HBA
on boot without it getting messed up. Add helper routines to work around
clang's over-use of the stack: that way we only pay the stack penalty
when a trace hits.

Sponsored by: Netflix
Reviewed by: ken, mav
Differential Revision: https://reviews.freebsd.org/D42411

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

cam_xpt: Reimplement xpt_*nounce_periph in terms of the _sbuf versions

Use an sbuf that drains to printf to avoid duplicating code in the two
versions of each function.

Reviewed by: mav
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D41263

cam_xpt: Remove fallbacks for non-sbuf protocol methods

This includes removing the kern.cam.announce_nosbuf sysctl.

Reviewed by: mav, imp
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D41262

cam xpt_*nounce_periph*: Various fixes for periphs without a protocol

If the periph doesn't have a valid protocol, these routines emit
fallback messages. However, the fallback messages duplicated the
periph name and unit number, and in the case of *denounce* included a
spurious newline.

Reviewed by: mav, imp
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D41177

cam: Migrate to modern uintXX_t from u_intXX_t

As per https://lists.freebsd.org/archives/freebsd-scsi/2023-July/000257.html
move to the modern uintXX_t.

MFC After: 3 days
Sponsored by: Netflix

cam_xpt: Properly fail if a sim uses an unsupported transport.

The default xport ops for a new bus is xport_default, not NULL, so
check for that when determining if a bus failed to find a suitable
transport. In addition, the path needs to be freed with xpt_free_path
instead of a plain free so that the path's reference on the sim is
dropped; otherwise, cam_sim_free in the caller after xpt_bus_register
returns failure will hang forever.

Note that we have to exempt the xpt bus from this check as it uses
xport_default on purpose.

Reviewed by: mav, imp
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D40617

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

cam: Properly mask out the status bits to get completion code

ccb_h.status has two parts: the actual status and some addition bits to
indicate additional information. It must be masked before comparing
against completion codes. Add new inline function cam_ccb_success to
simplify this to test whether or not the request succeeded. Most of the
code already does this, but a few places don't (the rest likely should
be converted to use cam_ccb_status and/or cam_ccb_success, but that's
for another day). This caused at least one bug in recognizing devices
behind a SATA port multiplexer, though some of these checks were
fine with the special knowledge of the code paths involved.

PR: 270459
Sponsored by: Netflix
MFC After: 1 week (and maybe a EN requst)
Reviewed by: ken, mav
Differential Revision: https://reviews.freebsd.org/D39572

cam_xpt: Prefer bool to int where it's a boolean

In the places where we set an integer to 0 or 1 and then use it like a
boolean, replace int with bool and 0/1 with false/true. Left alone
places where this is a function argument or return value. No functional
changes intended.

Sponsored by: Netflix

cam: Relax callouts precisions.

On large systems even relatively rare callouts may fire many times
per second. This should allow them to aggregate better, since we do
not require any precision when polling for media change, etc.

MFC after: 2 weeks

cam(4): Improve XPT_DEV_MATCH

Remove *_MATCH_NONE enums, making no sense and so never used. Make
*_MATCH_ANY enums 0 (no any match flags set), previously used by
*_MATCH_NONE. Bump CAM_VERSION to 0x1a reflecting those changes and
add compat shims.

When traversing through buses and devices do not descend if we can
already see that requested pattern does not match the bus or device.
It allows to save significant amount of time on system with thousands
of disks when doing limited searches.

Reviewed by: imp
MFC after: 2 weeks
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D32304

cam: Avoiding waking up doneq threads if we're dumping

Depending on the state of the target doneq thread at the time of the
panic, the wakeup can hang indefinitely in thread_lock_block_wait().
That function should likely be modified to return immediately if the
scheduler is stopped, but it is also preferable to avoid wakeups in
general after a panic.

Reported by: pho
Reviewed by: mav, imp
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32126

cam: Mark the qos data is valid in xpd_done_direct() too.

Sponsored by: Netflix

cam: change xpt_clone_path to return int

xpt_clone_path originally returned a cam_status, but it doesn't do I/O
and should return an errno instead. I added it last year and it's only
used in one place. It's not yet documented, so no doc changes are
nneeded.

Reviewed by: scottl@, mav@
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D30884

cam: fix xpt_bus_register and xpt_bus_deregister return errno

xpt_bus_register and xpt_bus_deregister returns a hybrid error that's
neither a cam_status, nor an errno, but a mix of both. Update
xpt_bus_register and xpt_bus_deregister to return an errno. The vast
majority of current users compare against zero, which can also be
spelled CAM_SUCCESS. Nobody uses CAM_FAILURE, so remove that symbol
to prevent comfusion (nothing returns it either).

Where the return value is saved, ensure that the variable 'error' is
used to store an errno and 'status' is used to store a cam_status where
it makes the code clearer (usually just in functions that already mix
and match). Where the return value isn't used at all, avoid storing it
at all.

Reviewed by: scottl@, mav@ (earlier version)
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D30860

cam: add xpt_path_sim_device to return device_t associated with a path

Return the device associated with the sim's bus when it called
xpt_bus_register, if any. Most real SIMs in the tree set this device,
but some virtual ones do not have a device_t assocaited with them.

Reviewed by: scottl@, mav@ (earlier version)
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D30853

cam: save parent_dev in xpt_bus_register

Reviewed by: scottl@
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D30846

cam: kill trailing white space in two spots

cam: Set the CAM_QOS_VALID when valid

When the elapsed time of the operation is complete and stored in the QOS
field, set the CAM_QOS_VALID bit. In iosched, test to make sure it's
set before using it.

Sponsored by: Netflix

cam: remove xpt_polled_action

Since periph_runccb now handles all the polling stuff, and
xpt_polled_action is now unused and can be removed.

Sponsored by: Netflix
Reviewed by: mav@
Differential Revision: https://reviews.freebsd.org/D30394

cam: remove sim callout

Nothing is using the sim callout to unfreeze the queue. Remove it to
simplify the SIM. This was introduced in the original CAM commit in 1998
but setting the CAM_SIM_REL_TIMEOUT_PENDING flag was removed in 1999 in
commit 87cfaf0e1fbd which reworked how bus reset worked. That work was
merged just after 3.2R was released. Remove the unused residuals.

Sponsored by: Netflix
Reviewed by: scottl@, mav@
Differential Revision: https://reviews.freebsd.org/D30383

mmccam: Add two new XPT for MMC and use them in mmc_sim and sdhci

For the discovery phase of SD/eMMC we need to do some transaction in a async
way.
The classic CAM XPT_{GET,SET}_TRAN_SETTING cannot be used in a async way.
This also allow us to split the discovery phase into a more complete state
machine and we don't mtx_sleep with a random number to wait for completion
of the tasks.
For mmc_sim we now do the SET_TRAN_SETTING in a taskqueue so we can call
the needed function for regulators/clocks without the cam lock(s). This part is
still needed to be done for sdhci.
We also now save the host OCR in the discovery phase as it wasn't done before and
only worked because the same ccb was reused.

Reviewed by: imp, kibab, bz
Differential Revision: https://reviews.freebsd.org/D30038

cam: allocate CCBs from UMA for SCSI and ATA IO

This patch makes it possible for CAM to use small CCBs allocated
from an periph-specific UMA zone instead of the usual, huge ones.
The end result is that CCBs issued via da(4) take 544B (size of
ccb_scsiio) instead of the usual 2kB (size of 'union ccb', ~1.5kB,
rounded up by malloc(9)). For ATA it's 272B. We waste less
memory, we avoid zeroing the unused 1kB, and it should be easier
to allocate those CCBs in low memory conditions. It should also
be possible to use uma_zone_reserve(9) to improve behaviour
in low memory conditions even further.

Note that this does not change the size, or the layout, of CCBs
as such. CCBs get allocated in various different ways, in particular
on the stack, and I don't want to redo all that. Instead, this
provides an opt-in mechanism for the periph to declare "my start()
callback is fine with receiving a CCB allocated from this UMA zone".
In other words, most of the code works exactly as it used to; the
change only happens to IOs issued by xpt_run_allockq(), which
is - conveniently - pretty much all that matters for performance.

The reason for doing it this way is that it's pretty small, localized
change, and can be implemented gradually and iteratively: take a
periph, make sure its start() callback only casts the CCBs it takes
to a particular type of CCB, for example ccb_scsiio, and that it only
casts CCBs returned by cam_periph_getccb() to that type, then add UMA
zone for that size, and declare it safe to XPT.

This is disabled by default. Set 'kern.cam.ada.enable_uma_ccbs=1'
and 'kern.cam.da.enable_uma_ccbs=1' tunables to enable it. Testing
is welcome; I will flip the default to enable in two weeks from now.

Reviewed By: imp
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D28674

cam: make sure to clear CCBs allocated on the stack

This is required for small CCBs support, where we need to track
whether the CCB was allocated from an UMA zone or not. There are
no (intended) functional changes with the current source.

Reviewed By: imp
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D29484

cam: Run all XPT_ASYNC ccbs in a dedicated thread

Queue all XPT_ASYNC ccb's and run those in a new cam async thread. This thread
is allowed to sleep for things like memory. This should allow us to make all the
registration routines for cam periph drivers simpler since they can assume they
can always allocate memory. This is a separate thread so that any I/O that's
completed in xpt_done_td isn't held up.

This should fix the panics for WAITOK alloations that are elsewhere in the
storage stack that aren't so easy to convert to NOWAIT. Additional future work
will convert other allocations in the registration path to WAITOK should
detailed analysis show it to be safe.

Reviewed by: chs@, rpokala@
Differential Revision: https://reviews.freebsd.org/D29210

cam: Properly find the sim in the assertion in xpt_pollwait().

I had missed merging this fixup into
447b3557a9cc5f00a301be8404339f21a9a0faa8 before pushing it.

Pointy hat to: jhb
MFC after: 2 weeks

cam: Permit non-pollable sims.

Some CAM sim drivers do not support polling (notably iscsi(4)).
Rather than using a no-op poll routine that always times out requests,
permit a SIM to set a NULL poll callback. cam_periph_runccb() will
fail polled requests non-pollable sims immediately as if they had
timed out.

Reviewed by: scottl, mav (earlier version)
Reviewed by: imp
MFC after: 2 weeks
Sponsored by: Chelsio
Differential Revision: https://reviews.freebsd.org/D28453

Make MAXPHYS tunable. Bump MAXPHYS to 1M.

Replace MAXPHYS by runtime variable maxphys. It is initialized from
MAXPHYS by default, but can be also adjusted with the tunable kern.maxphys.

Make b_pages[] array in struct buf flexible. Size b_pages[] for buffer
cache buffers exactly to atop(maxbcachebuf) (currently it is sized to
atop(MAXPHYS)), and b_pages[] for pbufs is sized to atop(maxphys) + 1.
The +1 for pbufs allow several pbuf consumers, among them vmapbuf(),
to use unaligned buffers still sized to maxphys, esp. when such
buffers come from userspace (*). Overall, we save significant amount
of otherwise wasted memory in b_pages[] for buffer cache buffers,
while bumping MAXPHYS to desired high value.

Eliminate all direct uses of the MAXPHYS constant in kernel and driver
sources, except a place which initialize maxphys. Some random (and
arguably weird) uses of MAXPHYS, e.g. in linuxolator, are converted
straight. Some drivers, which use MAXPHYS to size embeded structures,
get private MAXPHYS-like constant; their convertion is out of scope
for this work.

Changes to cam/, dev/ahci, dev/ata, dev/mpr, dev/mpt, dev/mvs,
dev/siis, where either submitted by, or based on changes by mav.

Suggested by: mav (*)
Reviewed by: imp, mav, imp, mckusick, scottl (intermediate versions)
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D27225

Microoptimize cam_num_doneqs math in xpt_done().

MFC after: 1 week

cam: clean up empty lines in .c and .h files

Hold the mutex when releasing a callout.

In xpt_release_device(), callout_stop() was being called without
holding the mutex (send_mtx) that is used to protect the callout.

So, move the mtx_unlock() call so that it is protected.

MFC after: 1 week
Sponsored by: Spectra Logic

Use the more descriptive src_ccb and dst_ccb for the two ccbs being merged.

MFC after: 1 week

cam: ANSIfy 0-argument function definitions

No functional change.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D24854

Now that we don't have special-case geom hacking defined in md_var.h, stop
including it. sparc64 was the last straggler here, but these weren't removed at
the time.

Give some indication of the CCB that's in flight when we panic.

Print the pointer to ccb so we can find it (for what good it does)
as well as the type of operation in flight when the cam_path has
been freed out from under us. This helps both core analysis as well
as automated systems that collect panic strings but little else.

Remove redundantly repetitive static __inline forward function
declarations.

We typically don't use them elsewhere in the kernel, and they aren't
needed here: the actual functions are a few lines away and aren't
mutually recursive.

Eliminate xpt_copy_path.

It's used in exactly one place. In that place it's used so we can hold the lock
on the device associated with the path (since we do a xpt_path_lock and unlock
pair around the callback). Instead, inline taking and dropping the reference to
the device so we can ensure we can unlock the mutex after the callback finishes
if the path in the ccb that's queued to be processed by xpt_scanner_thread is
destroyed while being processed. We don't actually need the path itself for
anything other than dereferencing it to get the device to do the lock and
unlock.

This also makes the locking / use model for cam_path a little cleaner by
eliminating a case where we needlessly copy the object.

Reviewed by: chuck, chs, ken
Differential Revision: https://reviews.freebsd.org/D24008

All paths lead to xpt_done, so move it after the switch.

xpt_async is submitting a CCB, not finishing it up, so use xpt_action() instead
of xpt_done(). Add the missing XPT_ASYNC case to xpt_action_default. xpt_async
wants to use the side-effect of the xpt_done() routine to queue this to the
camisr thread so it can be done in that context. However, this breaks the
symmetry that you create a ccb and call xpt_action() for it to be
dispatched. Restore that symmetry by having it go through that path. As far as I
can tell, this is the only CCB that we create and call xpt_done() on directly.

Get rid of silly /* FALLTHROUGH */ lines

Consistently omit /* FALLTHROUGH */ when we have a case statement that does
nothing. Since compilers don't warn about stacked case statements, and we were
inconsistent, resolve by removing extras.

This is a FALLTHROUGH for sure. no need for xxx comment.

xpt_release_simq_timeout is unused. Remove it.

Remove sparc64 kernel support

Remove all sparc64 specific files
Remove all sparc64 ifdefs
Removee indireeect sparc64 ifdefs

Use callout_func_t instead of the deprecated timeout_t.

Reviewed by: kib, imp
Differential Revision: https://reviews.freebsd.org/D22752

Mark some more hot global variables with __read_mostly.

MFC after: 1 week

Remove xpt_lock mutex.

CAM does not require SIM locks for years, and obviously does not require
it for completely virtual XPT SIM.

MFC after: 2 weeks

Make CAM use root_mount_hold_token() to delay boot.

Before this change CAM used config_intrhook_establish() for this purpose,
but that approach does not allow to delay it again after releasing once.

USB stack uses root_mount_hold() to delay boot until bus scan is complete.
But once it is, CAM had no time to scan SCSI bus, registered by umass(4),
if it already done other scans and called config_intrhook_disestablish().
The new approach makes it work smooth, assuming the USB device is found
during the initial bus scan. Devices appearing on USB bus later may still
require setting kern.cam.boot_delay, but hopefully those are minority.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Refactor xpt_getattr() to make it more readable. No outwardly
visible functional changes, though code flow was modified a bit
internally to lessen the need for goto jumps and chained if
conditionals.

Optimize xpt_getattr().

Do not allocate temporary buffer for attributes we are going to return
as-is, just make sure to NUL-terminate them. Do not zero temporary 64KB
buffer for CDAI_TYPE_SCSI_DEVID, XPT tells us how much data it filled
and there are also length fields inside the returned data also.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Fix panic message.

The panic message lead people to believe some userland CAM request had
caused a problem when in reallity it was for a kernel request (eg the
USER bit was cleared). Reword message. Also, improve a couple of
comments to reflect that the periph shouldn't be completely torn down
before we get here (so the path and sim pointers should be valid, but
aren't and the code is designed to be robust enough in the face of
that to give a specific panic message).

Don't allocate the config_intrhook separately from the softc, it's small
enough that it costs more code to handle the malloc/free than it saves.

Create xpt_sim_poll and refactor a bit using it.

xpt_sim_poll takes the sim to poll as an argument. It will do the
proper locking protocol, call the SIM polling routine, and then call
camisr_runqueue to process completions on any CCBs the SIM's poll
routine completed. It will be used during late shutdown when a SIM is
waiting for CCBs it sent during shutdown to finish and the scheduler
isn't running because we've panic'd.

This sequence was used twice in cam_xpt, so refactor those to use this
new function.

Sponsored by: Netflix
Differential Review: https://reviews.freebsd.org/D16663

remove unneeded inclusion of sys/interrupt.h from several files

It's likely that the header was needed in the past for swi(9).
But now that code does not use swi(9) or any other interfaces defined
in sys/interrupt.h.

MFC after: 1 week

Refactor NVMe CAM integration.

- Remove layering violation, when NVMe SIM code accessed CAM internal
device structures to set pointers on controller and namespace data.
Instead make NVMe XPT probe fetch the data directly from hardware.
- Cleanup NVMe SIM code, fixing support for multiple namespaces per
controller (reporting them as LUNs) and adding controller detach support
and run-time namespace change notifications.
- Add initial support for namespace change async events. So far only
in CAM mode, but it allows run-time namespace arrival and departure.
- Add missing nvme_notify_fail_consumers() call on controller detach.
Together with previous changes this allows NVMe device detach/unplug.

Non-CAM mode still requires a lot of love to stay on par, but at least
CAM mode code should not stay in the way so much, becoming much more
self-sufficient.

Reviewed by: imp
MFC after: 1 month
Sponsored by: iXsystems, Inc.

When devices are invalidated, there's some cases where ccbs for that
device still wind up in xpt_done after the path has been
invalidated. Since we don't always need sim or devq, add some guard
rails to only fail if we have to use them.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D14040

Revert ABI breakage to CAM that came in with MMC/SD support in r320844.
Make it possible to retrieve mmc parameters via the XPT_GET_ADVINFO
call instead. Convert camcontrol to the new scheme.

Reviewed by: imp. kibab
Sponsored by: Netflix
Differential Revision: D13868

Hold a refcount on the periph while running the allocation
queue. This will allow sub-transports to release their
probe pseudo-device with fewer convoluted restrictions.

Obtained from: Netflix

Define xpt_path_inq.

This provides a nice wrarpper around the XPT_PATH_INQ ccb creation and
calling.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D13387

Make cam_periph_runccb be safe to call when we can only do polling.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D13388

sys/cam: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Always null-terminate CAM periph_name and dev_name

Reported by: Coverity
CID: 1010039, 1010040, 1010041, 1010043
Reviewed by: ken, imp
MFC after: 3 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D13194

Remove a double free(9) in xpt_bus_register

In xpt_bus_register(), remove superfluous call to free(). This was mostly
benign since free(9) checks for NULL before doing anything, and
xpt_create_path() is nice enough to NULL out the pointer on failure.
However, it could've segfaulted if malloc(9) failed during
xpt_create_path().

Submitted by: gibbs
MFC after: 3 weeks
Sponsored by: Spectra Logic Corp

Fix NVMe's use of XPT_GDEV_TYPE

This patch changes the way XPT_GDEV_TYPE works for NVMe. The current
ccb_getdev structure includes pointers to the NVMe Identify Controller
and Namespace structures, but these are kernel virtual addresses which
are not accessible from user space.

As an alternative, the patch changes the pointers into padding in
ccb_getdev and adds two new types to ccb_dev_advinfo to retrieve the
Identify Controller (CDAI_TYPE_NVME_CNTRL) and Namespace
(CDAI_TYPE_NVME_NS) data structures.

Reviewed By: rpokala, imp
Differential Revision: https://reviews.freebsd.org/D10466
Submitted by: Chuck Tuffli

Fix 32-bit overflow on latency measurements

o Allow I/O scheduler to gather times on 32-bit systems. We do this by shifting
the sbintime_t over by 8 bits and truncating to 32-bits. This gives us 8.24
time. This is sufficient both in range (256 seconds is about 128x what current
users need) and precision (60ns easily meets the 1ms smallest bucket size
measurements). 64-bit systems are unchanged. Centralize all the time math so
it's easy to tweak tha range / precision tradeoffs in the future.
o While I'm here, the I/O scheduler should be using periph_data rather than
sim_data since it is operating on behalf of the periph.

Differential Review: https://reviews.freebsd.org/D12119

This adds CAM pass(4) support for NVMe IO's. Applications indicate
the IO type (Admin or NVM) using XPT op-codes XPT_NVME_ADMIN or
XPT_NVME_IO.

Submitted by: Chuck Tuffli <chuck@tuffli.net>
Differential Revision: https://reviews.freebsd.org/D10247

Kill some unnecessary noise.

An MMC/SD/SDIO stack using CAM

Implement the MMC/SD/SDIO protocol within a CAM framework. CAM's
flexible queueing will make it easier to write non-storage drivers
than the legacy stack. SDIO drivers from both the kernel and as
userland daemons are possible, though much of that functionality will
come later.

Some of the CAM integration isn't complete (there are sleeps in the
device probe state machine, for example), but those minor issues can
be improved in-tree more easily than out of tree and shouldn't gate
progress on other fronts. Appologies to reviews if specific items
have been overlooked.

Submitted by: Ilya Bakulin
Reviewed by: emaste, imp, mav, adrian, ian
Differential Review: https://reviews.freebsd.org/D4761

merge with first commit, various compile hacks.

Add infrastructure to the ATA and SCSI transports that supports
using a driver-supplied sbuf for printing device discovery
announcements. This helps ensure that messages to the console
will be properly serialized (through sbuf_putbuf) and not be
truncated and interleaved with other messages. The
infrastructure mirrors the existing xpt_announce_periph()
entry point and is opt-in for now. No content or formatting
changes are visible to the operator other than the new coherency.

While here, eliminate the stack usage of the temporary
announcement buffer in some of the drivers. It's moved to the
softc for now, but future work will eliminate it entirely by
making the code flow more linear. Future work will also address
locking so that the sbufs can be dynamically sized.

The scsi_da, scs_cd, scsi_ses, and ata_da drivers are converted
at this point, other drivers can be converted at a later date.
A tunable+sysctl, kern.cam.announce_nosbuf, exists for testing
purposes but will be removed later.

TODO:
Eliminate all of the code duplication and temporary buffers. The
old printf-based methods will be retired, and xpt_announce_periph()
will just be a wrapper that uses a dynamically sized sbuf. This
requires that the register and deregister paths be made malloc-safe,
which they aren't currently.

Sponsored by: Netflix

Minor cosmetic addition to r315673.

Now CAM_SIM_LOCK() macros are not used and may be removed later.

MFC after: 2 weeks

Make CAM SIM lock optional.

For three years now CAM does not use SIM lock, but still enforces SIM to
use it. Remove this requirement, allowing SIMs to have any locking they
prefer, if they pass no mutex to cam_sim_alloc().

MFC after: 2 weeks

Allow XPT_GDEV_STATS for UNCONFIGURED devices.

Queue statistics has nothing to do with presence or absence of INQUIRY
data, etc. Target mode devices are never configured, but have queues.

MFC after: 2 weeks

Reject userland CCBs that have CAM_UNLOCKED set.

CAM_UNLOCKED is internal flag and cannot correctly be set by userland.
Return EINVAL from CAMIOCOMMAND and CAMIOQUEUE if it is set.

Also fix leaks in some of the error paths for CAMIOQUEUE.

PR: 215356
Reviewed by: ken, mav
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D9869

Move inclusion of opt_printf.h around so that we can compile all the
SCSI modules outside of a sub-build from the kernel.

Differential Revision: https://reviews.freebsd.org/D9653
Sponsored by: Netflix

Remove pc98 support completely.
I thank all developers and contributors for pc98.

Relnotes: yes

Squash a couple of uses of xpt_print_path()

Sponsored by: Netflix

Fix a development mis-merge from r312827

Sponsored by: Netflix

Refactor xpt_print_path, xpt_print, and xpt_path_string. Implement
all of them in terms of an sbuf-based back-end, xpt_path_sbuf. This
unifies the implementation, but more importantly it stops the output
fropm being split between 4 or more invocations of printf. The
multiple invocations cause interleaving of the messages on the
console during boot, making the output of disk discovery often
unintelligible. This change helps a lot, but more work is needed.

Reviewed by: ken, mav
Sponsored by: Netflix

"Buses" is the preferred plural of "bus"

Replace archaic "busses" with modern form "buses."

Intentionally excluded:
* Old/random drivers I didn't recognize
* Old hardware in general
* Use of "busses" in code as identifiers

No functional change.

http://grammarist.com/spelling/buses-busses/

PR: 216099
Reported by: bltsrc at mail.ru
Sponsored by: Dell EMC Isilon

Always null-terminate ccb_pathinq.(sim_vid|hba_vid|dev_name)

The sim_vid, hba_vid, and dev_name fields of struct ccb_pathinq are
fixed-length strings. AFAICT the only place they're read is in
sbin/camcontrol/camcontrol.c, which assumes they'll be null-terminated.
However, the kernel doesn't null-terminate them. A bunch of copy-pasted code
uses strncpy to write them, and doesn't guarantee null-termination. For at
least 4 drivers (mpr, mps, ciss, and hyperv), the hba_vid field actually
overflows. You can see the result by doing "camcontrol negotiate da0 -v".

This change null-terminates those fields everywhere they're set in the
kernel. It also shortens a few strings to ensure they'll fit within the
16-character field.

PR: 215474
Reported by: Coverity
CID: 1009997 1010000 1010001 1010002 1010003 1010004 1010005
CID: 1331519 1010006 1215097 1010007 1288967 1010008 1306000
CID: 1211924 1010009 1010010 1010011 1010012 1010013 1010014
CID: 1147190 1010017 1010016 1010018 1216435 1010020 1010021
CID: 1010022 1009666 1018185 1010023 1010025 1010026 1010027
CID: 1010028 1010029 1010030 1010031 1010033 1018186 1018187
CID: 1010035 1010036 1010042 1010041 1010040 1010039
Reviewed by: imp, sephe, slm
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D9037
Differential Revision: https://reviews.freebsd.org/D9038

Fix UUID formatting added in r310360. Oops!

MFC after: 2 weeks

Report UUID and MD5 LUN IDs.

MFC after: 2 weeks

Explicitly initialize cdai.flags.

In SES driver uninitialized value caused unreliable physpath reporting.

MFC after: 1 week
Sponsored by: iXsystems, Inc.

cam: Zero bio pointer in user-supplied SCSI CCBs

The BUF_TRACKING bio pointer only makes sense for kernel consumers of
CCBs.

PR: 214250
Reported by: trasz@
Reviewed by: imp@, markj@
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8477

Add BUF_TRACKING and FULL_BUF_TRACKING buffer debugging

Upstream the BUF_TRACKING and FULL_BUF_TRACKING buffer debugging code.
This can be handy in tracking down what code touched hung bios and bufs
last. The full history is especially useful, but adds enough bloat that
it shouldn't be enabled in release builds.

Function names (or arbitrary string constants) are tracked in a
fixed-size ring in bufs. Bios gain a pointer to the upper buf for
tracking. SCSI CCBs gain a pointer to the upper bio for tracking.

Reviewed by: markj
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8366

Protect ccbq access with devq->send_mtx in the XPT_ABORT handler.

Submitted by: Ryan Libby <rlibby@gmail.com>
Reviewed by: mav
MFC after: 2 weeks
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D7985

Move protocol specific stuff into a linker set object that's
per-protocol. This reduces the number scsi symbols references by
cam_xpt significantly, and eliminates all ata / nvme symbols. There's
still some NVME / ATA specific code for dealing with XPT_NVME_IO and
XPT_ATA_IO respectively, and a bunch of scsi-specific code, but this
is progress.

Differential Revision: https://reviews.freebsd.org/D7289

Switch to linker sets to find the xport callback object. This
eliminates the need to special case everything in cam_xpt for new
transports. It is now a failure to not have a transport object when
registering the bus as well. You can still, however, create a
transport that's unspecified (XPT_)

Differential Revision: https://reviews.freebsd.org/D7289

Fix mismerge and include the nvme support.
Also, print out the name of any CCB's functions that's not supported.

MFC after: 1 week

Add place holder for SDIO CAM stuff for CCB XPT type.

Sponsored by: Netflix

Add NVME IO type.

Improve debugging of xpt.

Sponsored by: Netflix

Add "camcontrol reprobe" subcommand, and implement it for da(4).
This makes it possible to manually force updating capacity data
after the disk got resized. Without it it might be neccessary to
reboot before FreeBSD notices updated disk size under eg VMWare.

Discussed with: imp@
MFC after: 1 month
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D6108

sys/cam: spelling fixes in comments.

No functional change.

cam: unsign some types to match their definitions and avoid overflows.

numpatterns is u_int.

ctl:
CTL_NUM_MODE_PAGES comes from sizeof().
In struct:ctl_scsiio, kern_sg_entries is uint32_t.

MFC after: 2 weeks

New CAM I/O scheduler for FreeBSD. The default I/O scheduler is the same
as before. The common scheduling bits have moved from inline code in
each of the CAM periph drivers into a library that implements the
default scheduling.

In addition, a number of rate-limiting and I/O preference options can
be enabled by adding CAM_IOSCHED_NETFLIX to your config file. A number
of extra stats are also maintained. CAM_IOSCHED_NETFLIX isn't on by
default because it uses a separate BIO_READ and BIO_WRITE queue, so
doesn't honor BIO_ORDERED between these two types of operations. We
already didn't honor it for BIO_DELETE, and we don't depend on
BIO_ORDERED between reads and writes anywhere in the system (it is
currently used with BIO_FLUSH in ZFS to make sure some writes are
complete before others start and as a poor-man's soft dependency in
one place in UFS where we won't be issuing READs until after the
operation completes). However, out of an abundance of caution, it
isn't enabled by default.

Plus, this also brings in NCQ TRIM support for those SSDs that support
it. A black list is also provided for known rogues that use NCQ trim
as an excuse to corrupt the drive. It was difficult to separate out
into a separate commit.

This code has run in production at Netflix for over a year now.

Sponsored by: Netflix, Inc
Differential Revision: https://reviews.freebsd.org/D4609

Move to new value for XPT_GET_SIM_KNOB to avoid clash with XPT_ATA_IO.

Improve cam tracing a little by including the function code in the
traces for xpt_action. Note up-calls (down-calls?) to the SIM as well.

Differential Review: https://reviews.freebsd.org/D4382

Add asynchronous command support to the pass(4) driver, and the new
camdd(8) utility.

CCBs may be queued to the driver via the new CAMIOQUEUE ioctl, and
completed CCBs may be retrieved via the CAMIOGET ioctl. User
processes can use poll(2) or kevent(2) to get notification when
I/O has completed.

While the existing CAMIOCOMMAND blocking ioctl interface only
supports user virtual data pointers in a CCB (generally only
one per CCB), the new CAMIOQUEUE ioctl supports user virtual and
physical address pointers, as well as user virtual and physical
scatter/gather lists. This allows user applications to have more
flexibility in their data handling operations.

Kernel memory for data transferred via the queued interface is
allocated from the zone allocator in MAXPHYS sized chunks, and user
data is copied in and out. This is likely faster than the
vmapbuf()/vunmapbuf() method used by the CAMIOCOMMAND ioctl in
configurations with many processors (there are more TLB shootdowns
caused by the mapping/unmapping operation) but may not be as fast
as running with unmapped I/O.

The new memory handling model for user requests also allows
applications to send CCBs with request sizes that are larger than
MAXPHYS. The pass(4) driver now limits queued requests to the I/O
size listed by the SIM driver in the maxio field in the Path
Inquiry (XPT_PATH_INQ) CCB.

There are some things things would be good to add:

1. Come up with a way to do unmapped I/O on multiple buffers.
Currently the unmapped I/O interface operates on a struct bio,
which includes only one address and length. It would be nice
to be able to send an unmapped scatter/gather list down to
busdma. This would allow eliminating the copy we currently do
for data.

2. Add an ioctl to list currently outstanding CCBs in the various
queues.

3. Add an ioctl to cancel a request, or use the XPT_ABORT CCB to do
that.

4. Test physical address support. Virtual pointers and scatter
gather lists have been tested, but I have not yet tested
physical addresses or scatter/gather lists.

5. Investigate multiple queue support. At the moment there is one
queue of commands per pass(4) device. If multiple processes
open the device, they will submit I/O into the same queue and
get events for the same completions. This is probably the right
model for most applications, but it is something that could be
changed later on.

Also, add a new utility, camdd(8) that uses the asynchronous pass(4)
driver interface.

This utility is intended to be a basic data transfer/copy utility,
a simple benchmark utility, and an example of how to use the
asynchronous pass(4) interface.

It can copy data to and from pass(4) devices using any target queue
depth, starting offset and blocksize for the input and ouptut devices.
It currently only supports SCSI devices, but could be easily extended
to support ATA devices.

It can also copy data to and from regular files, block devices, tape
devices, pipes, stdin, and stdout. It does not support queueing
multiple commands to any of those targets, since it uses the standard
read(2)/write(2)/writev(2)/readv(2) system calls.

The I/O is done by two threads, one for the reader and one for the
writer. The reader thread sends completed read requests to the
writer thread in strictly sequential order, even if they complete
out of order. That could be modified later on for random I/O patterns
or slightly out of order I/O.

camdd(8) uses kqueue(2)/kevent(2) to get I/O completion events from
the pass(4) driver and also to send request notifications internally.

For pass(4) devcies, camdd(8) uses a single buffer (CAM_DATA_VADDR)
per CAM CCB on the reading side, and a scatter/gather list
(CAM_DATA_SG) on the writing side. In addition to testing both
interfaces, this makes any potential reblocking of I/O easier. No
data is copied between the reader and the writer, but rather the
reader's buffers are split into multiple I/O requests or combined
into a single I/O request depending on the input and output blocksize.

For the file I/O path, camdd(8) also uses a single buffer (read(2),
write(2), pread(2) or pwrite(2)) on reads, and a scatter/gather list
(readv(2), writev(2), preadv(2), pwritev(2)) on writes.

Things that would be nice to do for camdd(8) eventually:

1. Add support for I/O pattern generation. Patterns like all
zeros, all ones, LBA-based patterns, random patterns, etc. Right
Now you can always use /dev/zero, /dev/random, etc.

2. Add support for a "sink" mode, so we do only reads with no
writes. Right now, you can use /dev/null.

3. Add support for automatic queue depth probing, so that we can
figure out the right queue depth on the input and output side
for maximum throughput. At the moment it defaults to 6.

4. Add support for SATA device passthrough I/O.

5. Add support for random LBAs and/or lengths on the input and
output sides.

6. Track average per-I/O latency and busy time. The busy time
and latency could also feed in to the automatic queue depth
determination.

sys/cam/scsi/scsi_pass.h:
Define two new ioctls, CAMIOQUEUE and CAMIOGET, that queue
and fetch asynchronous CAM CCBs respectively.

Although these ioctls do not have a declared argument, they
both take a union ccb pointer. If we declare a size here,
the ioctl code in sys/kern/sys_generic.c will malloc and free
a buffer for either the CCB or the CCB pointer (depending on
how it is declared). Since we have to keep a copy of the
CCB (which is fairly large) anyway, having the ioctl malloc
and free a CCB for each call is wasteful.

sys/cam/scsi/scsi_pass.c:
Add asynchronous CCB support.

Add two new ioctls, CAMIOQUEUE and CAMIOGET.

CAMIOQUEUE adds a CCB to the incoming queue. The CCB is
executed immediately (and moved to the active queue) if it
is an immediate CCB, but otherwise it will be executed
in passstart() when a CCB is available from the transport layer.

When CCBs are completed (because they are immediate or
passdone() if they are queued), they are put on the done
queue.

If we get the final close on the device before all pending
I/O is complete, all active I/O is moved to the abandoned
queue and we increment the peripheral reference count so
that the peripheral driver instance doesn't go away before
all pending I/O is done.

The new passcreatezone() function is called on the first
call to the CAMIOQUEUE ioctl on a given device to allocate
the UMA zones for I/O requests and S/G list buffers. This
may be good to move off to a taskqueue at some point.
The new passmemsetup() function allocates memory and
scatter/gather lists to hold the user's data, and copies
in any data that needs to be written. For virtual pointers
(CAM_DATA_VADDR), the kernel buffer is malloced from the
new pass(4) driver malloc bucket. For virtual
scatter/gather lists (CAM_DATA_SG), buffers are allocated
from a new per-pass(9) UMA zone in MAXPHYS-sized chunks.
Physical pointers are passed in unchanged. We have support
for up to 16 scatter/gather segments (for the user and
kernel S/G lists) in the default struct pass_io_req, so
requests with longer S/G lists require an extra kernel malloc.

The new passcopysglist() function copies a user scatter/gather
list to a kernel scatter/gather list. The number of elements
in each list may be different, but (obviously) the amount of data
stored has to be identical.

The new passmemdone() function copies data out for the
CAM_DATA_VADDR and CAM_DATA_SG cases.

The new passiocleanup() function restores data pointers in
user CCBs and frees memory.

Add new functions to support kqueue(2)/kevent(2):

passreadfilt() tells kevent whether or not the done
queue is empty.

passkqfilter() adds a knote to our list.

passreadfiltdetach() removes a knote from our list.

Add a new function, passpoll(), for poll(2)/select(2)
to use.

Add devstat(9) support for the queued CCB path.

sys/cam/ata/ata_da.c:
Add support for the BIO_VLIST bio type.

sys/cam/cam_ccb.h:
Add a new enumeration for the xflags field in the CCB header.
(This doesn't change the CCB header, just adds an enumeration to
use.)

sys/cam/cam_xpt.c:
Add a new function, xpt_setup_ccb_flags(), that allows specifying
CCB flags.

sys/cam/cam_xpt.h:
Add a prototype for xpt_setup_ccb_flags().

sys/cam/scsi/scsi_da.c:
Add support for BIO_VLIST.

sys/dev/md/md.c:
Add BIO_VLIST support to md(4).

sys/geom/geom_disk.c:
Add BIO_VLIST support to the GEOM disk class. Re-factor the I/O size
limiting code in g_disk_start() a bit.

sys/kern/subr_bus_dma.c:
Change _bus_dmamap_load_vlist() to take a starting offset and
length.

Add a new function, _bus_dmamap_load_pages(), that will load a list
of physical pages starting at an offset.

Update _bus_dmamap_load_bio() to allow loading BIO_VLIST bios.
Allow unmapped I/O to start at an offset.

sys/kern/subr_uio.c:
Add two new functions, physcopyin_vlist() and physcopyout_vlist().

sys/pc98/include/bus.h:
Guard kernel-only parts of the pc98 machine/bus.h header with
#ifdef _KERNEL.

This allows userland programs to include <machine/bus.h> to get the
definition of bus_addr_t and bus_size_t.

sys/sys/bio.h:
Add a new bio flag, BIO_VLIST.

sys/sys/uio.h:
Add prototypes for physcopyin_vlist() and physcopyout_vlist().

share/man/man4/pass.4:
Document the CAMIOQUEUE and CAMIOGET ioctls.

usr.sbin/Makefile:
Add camdd.

usr.sbin/camdd/Makefile:
Add a makefile for camdd(8).

usr.sbin/camdd/camdd.8:
Man page for camdd(8).

usr.sbin/camdd/camdd.c:
The new camdd(8) utility.

Sponsored by: Spectra Logic
MFC after: 1 week

Remove compatibility shims for legacy ATA device names.

We got new ATA stack in FreeBSD 8.x, switched to it at 9.x, completely
removed old stack at 10.x, so at 11.x it is time to remove compat shims.

Make pass, sg and targ drivers respect HBA's maxio.

Previous limitation of 64K (DFLTPHYS) is quite annoying.

Initialize async_arg_ptr in xpt_async when called with async_code
AC_ADVINFO_CHANGED.

Without this change, newly inserted hard disks won't always have their
physical path device nodes created. The problem reproduces most readily
when attaching a large number of disks at once.

Differential Revision: https://reviews.freebsd.org/D2290
Reviewed by: mav, imp
MFC after: 2 weeks
Sponsored by: Spectra Logic

Be consistent with M_ZERO when allocating ccbs.

There are four places, all in cam_xpt.c, where ccbs are malloc'ed. Two of
these use M_ZERO, two don't. The two that don't meant that allocated ccbs
had trash in them making it hard to debug errors where they showed up. Due
to this, use M_ZERO all the time when allocating ccbs.

Submitted by: Scott Ferris <scott.ferris@isilon.com>
Sponsored by: EMC/Isilon Storage Division
Reviewed by: scottl, imp
Differential: https://reviews.freebsd.org/D2120

Make periphdriver_register() take XPT lock when modifying the periph_drivers
array.

This fixes a panic that sometimes occured when kldloading ctl.ko.

Reviewed by: mav@
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Add support for probing the SCSI VPD Extended Inquiry page (0x86).

This VPD page is effectively an extension of the standard Inquiry
data page, and includes lots of additional bits.

This commit includes support for probing the page in the SCSI probe code,
and an additional request type for the XPT_DEV_ADVINFO CCB. CTL already
supports the Extended Inquiry page.

Support for querying this page in the sa(4) driver will come later.

sys/cam/scsi/scsi_xpt.c:
Probe the Extended Inquiry page, if the device supports it, and
return it in response to a XPT_DEV_ADVINFO CCB if it is requested.

sys/cam/scsi/cam_ccb.h:
Define a new advanced information CCB data type, CDAI_TYPE_EXT_INQ.

sys/cam/cam_xpt.c:
Free the extended inquiry data in a device when the device goes
away.

sys/cam/cam_xpt_internal.h:
Add an extended inquiry data pointer and length to struct cam_ed.

sys/sys/param.h
Bump __FreeBSD_version for the addition of the new
CDAI_TYPE_EXT_INQ advanced information type.

Sponsored by: Spectra Logic
MFC after: 1 week

Restore the CAM XPT peripheral generation counter, and export it via sysctl.

Define it as an atomic uint32_t. These increments happen infrequently
enough for the atomic overhead to be a problem, and since they're now
independent atomics, they won't contend with xpt_lock_buses().

This counter is useful as a means of cheaply identifying whether any changes
have been made to the CAM peripheral list. Userland programs have no guarantee
that the counter won't change on them while being returned or while processing
the information, so they must be written accordingly.

Discussed with: ken, mav (in general)
MFC after: 1 week
Sponsored by: Spectra Logic

Remove extra mtx_unlock().

Submitted by: Dmitry Luhtionov <dmitryluhtionov@gmail.com>
MFC after: 1 week

When passing LUN IDs through treat ASCII values as fixed-length, not
interpreating NULLs as EOLs, but converting them to spaces.

SPC-4 does not tell that T10-based IDs should be NULL-terminated/padded.
And while it tells that it should include only ASCII chars (0x20-0x7F),
there are some USB sticks (SanDisk Ultra Fit), that have NULLs inside
the value. Treating NULLs as EOLs there made those LUN IDs non-unique.

MFC after: 1 week

Prevent overflow issues in timeout processing

Previously, any timeout value for which (timeout * hz) will overflow the
signed integer, will give weird results, since callout(9) routines will
convert negative values of ticks to '1'. For unsigned integer overflow we
will get sufficiently smaller timeout values than expected.

Switch from callout_reset, which requires conversion to int based ticks
to callout_reset_sbt to avoid this.

Also correct isci to correctly resolve ccb timeout.

This was based on the original work done by Eygene Ryabinkin
<rea@freebsd.org> back in 5 Aug 2011 which used a macro to help avoid
the overlow.

Differential Revision: https://reviews.freebsd.org/D1157
Reviewed by: mav, davide
MFC after: 1 month
Sponsored by: Multiplay

Use proper variable when looping through periphs with CAM_PERIPH_FREE.

PR: 194256
Submitted by: Scott M. Ferris <smferris@gmail.com>
MFC after: 3 days
Sponsored by: EMC/Isilon Storage Division

Correct a comment

Update CAM CCB accounting for the new status quo.

devq_openings counter lost its meaning after allocation queues has gone.
held counter is still meaningful, but problematic to update due to separate
locking of CCB allocation and queuing.

To fix that replace devq_openings counter with allocated counter. held is
now calculated on request as difference between number of allocated, queued
and active CCBs.

MFC after: 1 month

Pull in r267961 and r267973 again. Fix for issues reported will follow.

Revert r267961, r267973:

These changes prevent sysctl(8) from returning proper output,
such as:

1) no output from sysctl(8)
2) erroneously returning ENOMEM with tools like truss(1)
or uname(1)
truss: can not get etype: Cannot allocate memory

Extend the meaning of the CTLFLAG_TUN flag to automatically check if
there is an environment variable which shall initialize the SYSCTL
during early boot. This works for all SYSCTL types both statically and
dynamically created ones, except for the SYSCTL NODE type and SYSCTLs
which belong to VNETs. A new flag, CTLFLAG_NOFETCH, has been added to
be used in the case a tunable sysctl has a custom initialisation
function allowing the sysctl to still be marked as a tunable. The
kernel SYSCTL API is mostly the same, with a few exceptions for some
special operations like iterating childrens of a static/extern SYSCTL
node. This operation should probably be made into a factored out
common macro, hence some device drivers use this. The reason for
changing the SYSCTL API was the need for a SYSCTL parent OID pointer
and not only the SYSCTL parent OID list pointer in order to quickly
generate the sysctl path. The motivation behind this patch is to avoid
parameter loading cludges inside the OFED driver subsystem. Instead of
adding special code to the OFED driver subsystem to post-load tunables
into dynamically created sysctls, we generalize this in the kernel.

Other changes:
- Corrected a possibly incorrect sysctl name from "hw.cbb.intr_mask"
to "hw.pcic.intr_mask".
- Removed redundant TUNABLE statements throughout the kernel.
- Some minor code rewrites in connection to removing not needed
TUNABLE statements.
- Added a missing SYSCTL_DECL().
- Wrapped two very long lines.
- Avoid malloc()/free() inside sysctl string handling, in case it is
called to initialize a sysctl from a tunable, hence malloc()/free() is
not ready when sysctls from the sysctl dataset are registered.
- Bumped FreeBSD version to indicate SYSCTL API change.

MFC after: 2 weeks
Sponsored by: Mellanox Technologies

Report more readable state "-" for idle CAM scan thread.

Fix memory and references leak due to unfreed path in case we can't
allocate bus scan CCB.

MFC after: 2 weeks

Move xpt_run_devq() call before request completion callback where it was
originally.

I am not sure why exactly have I moved it during one of many refactorings
during camlock project, but obviously it opens race window that may cause
use after free panics during SIM (in reported cases umass(4)) detach.

MFC after: 2 weeks

Take additional reference on SCSI probe periph to cover its freeze count.

Otherwise periph may be invalidated and freed before single-stepping freeze
is dropped, causing use after free panic.

Replace several instances of -1 with appropriate CAM_*_WILDCARD and types.

It was equal before r259397, but for good or bad, not any more for LUNs.

This change fixes at least CAM debugging.

Some CAM locks polishing:
- Fix LOR and possible lock recursion when handling high-power commands.
Introduce new lock to protect left power quota and list of frozen devices.
- Correct locking around xpt periph creation.
- Remove seems never used XPT_FLAG_OPEN xpt periph flag.

Fix lock recursion, triggered by `smartctl -a /dev/adaX`.

printf() specifier updates to CAM to handle either 32-bit or 64-bit lun_id_t.

MFC after: 2 weeks

Minor (mostly cosmetical) addition to r256960.

Move CAM_UNQUEUED_INDEX setting to the last moment and under the periph lock.
This fixes race condition with cam_periph_ccbwait(), causing use-after-free.

Unconditionally acquire periph reference on CCB allocation failure.

cam_periph_acquire() can return error if periph already invalidated, but
that may be unacceptable and cause deadlock if the invalidated periph can't
be destroyed without "executing" the scheduled request.

Coverity CID: 1109822
MFC after: 2 months

Merge CAM locking changes from the projects/camlock branch to radically
reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.

Replace big per-SIM locks with bunch of smaller ones:
- per-LUN locks to protect device and peripheral drivers state;
- per-target locks to protect list of LUNs on target;
- per-bus locks to protect reference counting;
- per-send queue locks to protect queue of CCBs to be sent;
- per-done queue locks to protect queue of completed CCBs;
- remaining per-SIM locks now protect only HBA driver internals.

While holding LUN lock it is allowed (while not recommended for performance
reasons) to take SIM lock. The opposite acquisition order is forbidden.
All the other locks are leaf locks, that can be taken anywhere, but should
not be cascaded. Many functions, such as: xpt_action(), xpt_done(),
xpt_async(), xpt_create_path(), etc. are no longer require (but allow) SIM
lock to be held.

To keep compatibility and solve cases where SIM lock can't be dropped, all
xpt_async() calls in addition to xpt_done() calls are queued to completion
threads for async processing in clean environment without SIM lock held.

Instead of single CAM SWI thread, used for commands completion processing
before, use multiple (depending on number of CPUs) threads. Load balanced
between them using "hash" of the device B:T:L address.

HBA drivers that can drop SIM lock during completion processing and have
sufficient number of completion threads to efficiently scale to multiple
CPUs can use new function xpt_done_direct() to avoid extra context switch.
Make ahci(4) driver to use this mechanism depending on hardware setup.

Sponsored by: iXsystems, Inc.
MFC after: 2 months

Unify periph invalidation and destruction reporting.
Print message containing device model and serial number on invalidation.

Requested by: glebius
MFC after: 1 week

Unhide "Serial Number" lines from bootverbose. That information may be
useful for system administration to have in hard copy (in logs) if one of
several devices suddenly dies.

Requested by: glebius
MFC after: 1 week

Close the race on path ID allocation in xpt_bus_register() if two buses are
registered simultaneously. Due to topology unlock between the ID allocation
and the bus registration there is a chance that two buses may get the
same IDs. That is supposed reason of lock assertion panic in CAM during
initial bus scanning after new iscsid initiates two sessions same time.

Reported by: trasz
Approved by: re (glebus, marius)
MFC after: 2 weeks

Make sure the CCB xflags field is initialized to zero so that
CAM_EXTLUN_VALID is not erroneously set. Also add an XPORT_SRP
identifier to the known SCSI transports for the SCSI RDMA protocol, as
used, for example with Infiniband storage.

Reviewed by: scottl
Approved by: re (marius)

Re-do r255853. Along with adding back the API/ABI changes from the
original, this hides the contents of cam_compat.h from ktrace/kdump/truss,
avoiding problems there. There are no user-servicable parts in there, so
no need for those tools to be groping around in there.

Approved by: re

Revert r255853 pending fixes to build errors in usr.bin/kdump

Approved by: re (implicit)

Update the CAM API for FreeBSD 10:

- Remove the timeout_ch field. It's been deprecated since FreeBSD 7.0;
MPSAFE drivers should be managing their own timeout storage. The
remaining non-MPSAFE drivers have been modified to also manage their own
storage, and should be considered for updating to MPSAFE (or removal)
during the FreeBSD 10.x lifecycle.

- Add fields related to soft timeouts and quality of service, to be used
in upcoming work.

- Add room for more flags in the CCB header and path_inq structures.

- Begin support for extended 64-bit LUNs.

- Bump the CAM version number to 0x18, but add compat shims. Tested with
camcontrol and smartctl.

Reviewed by: nathanw, ken, kib
Approved by: re
Obtained from: Netflix

Add debug trace points for freeze/release device queue.

Add new attribute lunname to report only textual LUN-specific device IDs.
While lunid attribute prefers to report numeric ones, having both may be
useful in some situations.

Remove droping topology mutex after iterating 100 periphs in CAMGETPASSTHRU.
That is not so slow and so often operation to handle unneeded otherwise
xsoftc.xpt_generation and respective locking complications.

MFprojects/camlock r249006:
Pass SIM pointer as an argument to camisr_runqueue() instead of doneq
pointer.

MFprojects/camlock r249505:
Change CCB queue resize logic to be able safely handle overallocations:
- (re)allocate queue space in power of 2 chunks with 64 elements minimum
and never shrink it; with only 4/8 bytes per element size is insignificant.
- automatically reallocate the queue to double size if it is overflowed.
- if queue reallocation failed, store extra CCBs in unsorted TAILQ,
fetching them back as soon as some queue element is freed.

To free space in CCB for TAILQ linking, change highpowerq from keeping
high-power CCBs to keeping devices frozen due to high-power CCBs.

This encloses all pieces of queue resize logic inside of cam_queue.[ch],
removing some not obvious duties from xpt_release_ccb().

CAM and mps(4) driver scanning changes.

Add a PIM_NOSCAN flag to the CAM path inquiry CCB. This tells CAM
not to perform a rescan on a bus when it is registered.

We now use this flag in the mps(4) driver. Since it knows what
devices it has attached, it is more efficient for it to just issue
a target rescan on the targets that are attached.

Also, remove the private rescan thread from the mps(4) driver in
favor of the rescan thread already built into CAM. Without this
change, but with the change above, the MPS scanner could run before
or during CAM's initial setup, which would cause duplicate device
reprobes and announcements.

sys/param.h:
Bump __FreeBSD_version to 1000039 for the inclusion of the
PIM_RESCAN CAM path inquiry flag.

sys/cam/cam_ccb.h:
sys/cam/cam_xpt.c:
Added a PIM_NOSCAN flag. If a SIM sets this in the path
inquiry ccb, then CAM won't rescan the bus in
xpt_bus_regsister.

sys/dev/mps/mps_sas.c
For versions of FreeBSD that have the PIM_NOSCAN path
inquiry flag, don't freeze the sim queue during scanning,
because CAM won't be scanning this bus. Instead, hold
up the boot. Don't call mpssas_rescan_target in
mpssas_startup_decrement; it's redundant and I don't
know why it was in there.

Set PIM_NOSCAN in path inquiry CCBs.

Remove methods related to the internal rescan daemon.

Always use async events to trigger a probe for EEDP support.
In older versions of FreeBSD where AC_ADVINFO_CHANGED is
not available, use AC_FOUND_DEVICE and issue the
necessary READ CAPACITY manually.

Provide a path to xpt_register_async() so that we only
receive events for our own SCSI domain.

Improve error reporting in cases where setup for EEDP
detection fails.

sys/dev/mps/mps_sas.h:
Remove softc flags and data related to the scanner thread.

sys/dev/mps/mps_sas_lsi.c:
Unconditionally rescan the target whenever a device is added.

Sponsored by: Spectra Logic
MFC after: 1 week

Make some improvements to r253322 to really rescan target, not a bus.
Add there and in two more places checks for NULL on xpt_alloc_ccb_nowait().

Add infrastructure for doing compatibility shims, as has been sorely
needed for the last 10 years. Far too much of the internal API is
exposed, and every small adjustment causes applications to stop working.
To kick this off, bump the API version to 0x17 as should have been done
with r246713, but add shims to compensate. Thanks to the shims, there
should be no visible change in application behavior.

I have plans to do a significant overhaul of the API to harnen it for
the future, but until then, I welcome others to add shims for older
versions of the API.

Obtained from: Netflix

Make CAM return and GEOM DISK pass through new GEOM::lunid attribute.

SPC-4 specification states that serial number may be property of device,
but not a specific logical unit. People reported about FC storages using
serial number in that way, making it unusable for purposes of LUN multipath
detection. SPC-4 states that designators associated with logical unit from
the VPD page 83h "Device Identification" should be used for that purpose.
Report first of them in the new attribute in such preference order: NAA,
EUI-64, T10 and SCSI name string.

While there, make GEOM DISK properly report GEOM::ident in XML output also
using d_getattr() method, if available. This fixes serial numbers reporting
for SCSI disks in `geom disk list` output and confxml.

Discussed with: gibbs, ken
Sponsored by: iXsystems, Inc.
MFC after: 2 weeks

Added output of device QUIRKS for CAM and AHCI devices during boot.

Reviewed by: mav
Approved by: pjd (mentor)
MFC after: 2 weeks

Expose CAM_BOOT_DELAY as a kernel conf item now.

This allows users who boot without loader to adjust their environments
around slightly buggy or slow hardware.

PR: kern/161809
Submitted by: rozhuk.im@gmail.com
MFC after: 2 weeks

Remove owner field from struct cam_ed, unused at least since FreeBSD 7.

MFprojects/camlock r248982:
Stop abusing xpt_periph in random plases that really have no periph related
to CCB, for example, bus scanning. NULL value is fine in such cases and it
is correctly logged in debug messages as "noperiph". If at some point we
need some real XPT periphs (alike to pmpX now), quite likely they will be
per-bus, and not a single global instance as xpt_periph now.

MFprojects/camlock r248890, r248897, r248898, r248900, r248903, r248905,
r248917, r248918, r248978, r249001, r249014, r249030:

Remove multilevel freezing mechanism, implemented to handle specifics of
the ATA/SATA error recovery, when post-reset recovery commands should be
allocated when queues are already full of payload requests. Instead of
removing frozen CCBs with specified range of priorities from the queue
to provide free openings, use simple hack, allowing explicit CCBs over-
allocation for requests with priority higher (numerically lower) then
CAM_PRIORITY_OOB threshold.

Simplify CCB allocation logic by removing SIM-level allocation queue.
After that SIM-level queue manages only CCBs execution, while allocation
logic is localized within each single device.

Suggested by: gibbs

MFprojects/camlock:
r249017:
Some cosmetic things:
- Unify device to target insertion inside xpt_alloc_device() instead of
duplicating it three times.
- Remove extra checks for empty lists of devices and targets on release
since zero refcount check also implies it.
- Reformat code to reduce indentation.

r249103:
- Add lock assertions to every point where reference counters are modified.
- When reference counters are reaching zero, add assertions that there are
no children items left.
- Add a bit more locking to the xptpdperiphtraverse().

MFprojects/camlock r248928:
Move CAM_DEBUG_CDB messages from the point of queuing to the point of
sending to SIM. That allows to inspect real requests execution order,
respecting priorities, freezing, etc.

MFC after: 2 weeks

Remove xpt_sim_opened(), the only consumer of which was atapicam, which is
now gone.

Use xpt_lock_buses() instead of equivalent mtx_lock(&xsoftc.xpt_topo_lock)
to unify the code.

Reform the busdma API so that new types may be added without modifying
every architecture's busdma_machdep.c. It is done by unifying the
bus_dmamap_load_buffer() routines so that they may be called from MI
code. The MD busdma is then given a chance to do any final processing
in the complete() callback.

The cam changes unify the bus_dmamap_load* handling in cam drivers.

The arm and mips implementations are updated to track virtual
addresses for sync(). Previously this was done in a type specific
way. Now it is done in a generic way by recording the list of
virtuals in the map.

Submitted by: jeff (sponsored by EMC/Isilon)
Reviewed by: kan (previous version), scottl,
mjacob (isp(4), no objections for target mode changes)
Discussed with: ian (arm changes)
Tested by: marius (sparc64), mips (jmallet), isci(4) on x86 (jharris),
amd64 (Fabian Keil <freebsd-listen@fabiankeil.de>)

Fix a panic during CAM EDT traversal.

The problem was a race condition between the EDT traversal used by
things like 'camcontrol devlist', and CAM peripheral driver
removal.

The EDT traversal code holds the CAM topology lock, and wants
to show devices that have been invalidated. It acquires a
reference to the peripheral to make sure the peripheral it is
examining doesn't go away.

However, because the peripheral removal code in camperiphfree()
drops the CAM topology lock to call the peripheral's destructor
routine, we can run into a situation where the EDT traversal
increments the peripheral reference count after free process is
already in progress. At that point, the reference count is
ignored, because it was 0 when we started the process.

Fix this race by setting a flag, CAM_PERIPH_FREE, that I previously
added and checked in xptperiphtraverse() and xptpdperiphtravsere(),
but failed to use. If the EDT traversal code sees that flag,
it will know that the peripheral free process has already started,
and that it should not access that peripheral.

Also, fix an inconsistency in the locking between
xptpdperiphtraverse() and xptperiphtraverse(). They now both
hold the CAM topology lock while calling the peripheral traversal
function.

cam_xpt.c: Change xptperiphtraverse() to hold the CAM topology
lock across calls to the traversal function.

Take out the comment in xptpdperiphtraverse() that
referenced the locking inconsistency.

cam_periph.c: Set the CAM_PERIPH_FREE flag when we are in the
process of freeing a peripheral driver.

Sponsored by: Spectra Logic Corporation
MFC after: 1 week

Remove priority enforcement from xpt_ation(). It is not good and even not
safe in some cases to reduce CCB priority after it was scheduled with high
priority. This fixes reproducible deadlock when command sent through the
pass interface while ATA XPT recovers from command timeout.

Instead of that enforce priority at passioctl(). libcam provides no obvious
interface to specify CCB priority and so much (all?) code specifies zero
(highest) priority. This change limits pass CCBs priority to NORMAL run
level, allowing XPT to complete bus and device recovery after reset before
running any payload.

Don't exclude XPT SIM from locking in xpt_create_path_unlocked().
We don't want xpt periph, device, target or bus disappeared because of
incorrect reference counting.

Use xpt_create_path_unlocked() for initial debug path compilation because
we are not holding respective SIM lock.

Fix XPT_DEBUG paths operations locking:
- Extend the lock to cover xpt_path_release() for the new path.
- While xpt_action() is called while holding right SIM lock for the new
bus, the old path release may require different SIM lock. So we have
to temporary drop the new lock and get the old one.

XPT_DEV_MATCH is probably the only xpt_action() method that is called
without holding SIM lock. It really doesn't need that lock, but adding it
removes that specific exception, allowing to assert locking there later.

Submitted by: ken@ (earlier version)

Protect xpt_getattr() calls with the SIM lock and assert that.

Submitted by: ken@ (earlier version)

Use separate malloc buckets for CAM devices, CCBs and paths. This will
make it easier to track down the source of any use after free problems.

Submitted by: ken@

Don't duplicate path/ccb allocation code, use existing functions.

Really handle xpt_compile_path() error in xpt_bus_register() instead of
print error message and probably crash just after it on NULL dereference.

Found by: Clang Static Analyzer

Implement media change notification for DA and CD removable media devices.
It includes three parts:
1) Modifications to CAM to detect media media changes and report them to
disk(9) layer. For modern SATA (and potentially UAS) devices it utilizes
Asynchronous Notification mechanism to receive events from hardware.
Active polling with TEST UNIT READY commands with 3 seconds period is used
for incapable hardware. After that both CD and DA drivers work the same way,
detecting two conditions: "NOT READY: Medium not present" after medium was
detected previously, and "UNIT ATTENTION: Not ready to ready change, medium
may have changed". First one reported to disk(9) as media removal, second
as media insert/change. To reliably receive second event new
AC_UNIT_ATTENTION async added to make UAs broadcasted to all periphs by
generic error handling code in cam_periph_error().
2) Modifications to GEOM core to handle media remove and change events.
Media removal handled by spoiling all consumers attached to the provider.
Media change event also schedules provider retaste after spoiling to probe
new media. New flag G_CF_ORPHAN was added to consumers to reflect that
consumer is in process of destruction. It allows retaste to create new
geom instance of the same class, while previous one is still dying.
3) Modifications to some GEOM classes: DEV -- to report media change
events to devd; VFS -- to handle spoiling same as orphan to prevent
accessing replaced media. PART class already handles spoiling alike to
orphan.

Reviewed by: silence on geom@ and scsi@
Tested by: avg
Sponsored by: iXsystems, Inc. / PC-BSD
MFC after: 2 months

In camisr() clear CAM_SIM_ON_DONEQ flag after camisr_runqueue() purged SIM
done queue. Clearing it before caused extra SIM queueing in some cases.
It was invisible during normal operation, but during USB device unplug and
respective SIM destruction it could keep pointer on SIM without having
counted reference and as result crash the system by use afer free.

Reported by: hselasky
MFC after: 1 week

Fix several reference counting and object lifetime issues between
the pass(4) and enc(4) drivers and devfs.

The pass(4) driver uses the destroy_dev_sched() routine to
schedule its device node for destruction in a separate thread
context. It does this because the passcleanup() routine can get
called indirectly from the passclose() routine, and that would
cause a deadlock if the close routine tried to destroy its own
device node.

In any case, once a particular passthrough driver number, e.g.
pass3, is destroyed, CAM considers that unit number (3 in this
case) available for reuse.

The problem is that devfs may not be done cleaning up the previous
instance of pass3, and will panic if isn't done cleaning up the
previous instance.

The solution is to get a callback from devfs when the device node
is removed, and make sure we hold a reference to the peripheral
until that happens.

Testing exposed some other cases where we have reference counting
issues, and those were also fixed in the pass(4) driver.

cam_periph.c: In camperiphfree(), reorder some of the operations.

The peripheral destructor needs to be called before
the peripheral is removed from the peripheral is
removed from the list. This is because once we
remove the peripheral from the list, and drop the
topology lock, the peripheral number may be reused.
But if the destructor hasn't been called yet, there
may still be resources hanging around (like devfs
nodes) that haven't been fully cleaned up.

cam_xpt.c: Add an argument to xpt_remove_periph() to indicate
whether the topology lock is already held.

scsi_enc.c: Acquire an extra reference to the peripheral during
registration, and release it once we get a callback
from devfs indicating that the device node is gone.

Call destroy_dev_sched_cb() in enc_oninvalidate()
instead of calling destroy_dev() in the cleanup
routine.

scsi_pass.c: Add reference counting to handle peripheral and
devfs object lifetime issues.

Add a reference to the peripheral and the devfs
node in the peripheral registration.

Don't attempt to add a physical path alias if the
peripheral has been marked invalid.

Release the devfs reference once the initial
physical path alias taskqueue run has completed.

Schedule devfs node destruction in the
passoninvalidate(), and release our peripheral
reference in a new routine, passdevgonecb() once
the devfs node is gone. This allows the peripheral
to fully go away, and the peripheral destructor,
passcleanup(), will get called.

MFC after: 3 days
Sponsored by: Spectra Logic

Add CAM_DEBUG_INFO debug messages for periph created/invalidated/destroyed
and for asyncs sent.

To make CAM debugging easier, compile in some debug flags (CAM_DEBUG_INFO,
CAM_DEBUG_CDB, CAM_DEBUG_PERIPH and CAM_DEBUG_PROBE) by default.
List of these flags can be modified with CAM_DEBUG_COMPILE kernel option.
CAMDEBUG kernel option still enables all possible debug, if not overriden.

Additional 50KB of kernel size is a good price for the ability to debug
problems without rebuilding the kernel. In case where size is important,
debugging can be compiled out by setting CAM_DEBUG_COMPILE option to 0.

Replace #ifdef CAMDEBUG + if + panic() with single KASSERT().

Rewrite enabling NCQ for SATA devices in a way more alike to SCSI TCQ.
This allows to control it with `camcontrol negotiate adaX -T (en|dis)able`
on the fly, same as for SCSI devices.

Sponsored by: iXsystems, Inc.

Use AC_GETDEV_CHANGED async to notify ada driver about DMA and NCQ status
change. Now that allows switching between PIO and DMA modes on the fly.

Allow to change number of openings (used tags) for ATA/SATA devices
via `camcontrol tags ... -N ...`. There is no need to tune it in
usual cases, but some users want to have it for debugging purposes.

MFC after: 2 weeks

MFprojects/zfsd:
- Add low-level support for SATA Enclosure Management Bridge (SEMB)
devices -- SATA equivalents of the SCSI SES/SAF-TE devices.
- Add some utility functions for SCSI SAF-TE devices access.

Sponsored by: iXsystems, Inc.

Add two functions xpt_batch_start() and xpt_batch_done() to the CAM SIM KPI
to allow drivers to handle request completion directly without passing
them to the CAM SWI thread removing extra context switch.
Modify all ATA/SATA drivers to use them.

Reviewed by: gibbs, ken
MFC after: 2 weeks

Add CAM infrastructure to allow reporting when a drive's long read capacity
data changes.

cam_ccb.h: Add a new advanced information type, CDAI_TYPE_RCAPLONG,
for long read capacity data.

cam_xpt_internal.h:
Add a read capacity data pointer and length to struct cam_ed.

cam_xpt.c: Free the read capacity buffer when a device goes away.
While we're here, make sure we don't leak memory for other
malloced fields in struct cam_ed.

scsi_all.c: Update the scsi_read_capacity_16() to take a uint8_t * and
a length instead of just a pointer to the parameter data
structure. This will hopefully make this function somewhat
immune to future changes in the parameter data.

scsi_all.h: Add some extra bit definitions to struct
scsi_read_capacity_data_long, and bump up the structure
size to the full size specified by SBC-3.

Change the prototype for scsi_read_capacity_16().

scsi_da.c: Register changes in read capacity data with the transport
layer. This allows the transport layer to send out an
async notification to interested parties. Update the
dasetgeom() API.

Use scsi_extract_sense_len() instead of
scsi_extract_sense().

scsi_xpt.c: Add support for the new CDAI_TYPE_RCAPLONG advanced
information type.

Make sure we set the physpath pointer to NULL after freeing
it. This allows blindly freeing it in the struct cam_ed
destructor.

sys/param.h: Bump __FreeBSD_version from 1000005 to 1000006 to make it
easier for third party drivers to determine that the read
capacity data async notification is available.

camcontrol.c,
mptutil/mpt_cam.c:
Update these for the new scsi_read_capacity_16() argument
structure.

Sponsored by: Spectra Logic

Fix a race condition in CAM peripheral free handling, locking
in the CAM XPT bus traversal code, and a number of other periph level
issues.

cam_periph.h,
cam_periph.c: Modify cam_periph_acquire() to test the CAM_PERIPH_INVALID
flag prior to allowing a reference count to be gained
on a peripheral. Callers of this function will receive
CAM_REQ_CMP_ERR status in the situation of attempting to
reference an invalidated periph. This guarantees that
a peripheral scheduled for a deferred free will not
be accessed during its wait for destruction.

Panic during attempts to drop a reference count on
a peripheral that already has a zero reference count.

In cam_periph_list(), use a local sbuf with SBUF_FIXEDLEN
set so that mallocs do not occur while the xpt topology
lock is held, regardless of the allocation policy of the
passed in sbuf.

Add a new routine, cam_periph_release_locked_buses(),
that can be called when the caller already holds
the CAM topology lock.

Add some extra debugging for duplicate peripheral
allocations in cam_periph_alloc().

Treat CAM_DEV_NOT_THERE much the same as a selection
timeout (AC_LOST_DEVICE is emitted), but forgo retries.

cam_xpt.c: Revamp the way the EDT traversal code does locking
and reference counting. This was broken, since it
assumed that the EDT would not change during
traversal, but that assumption is no longer valid.

So, to prevent devices from going away while we
traverse the EDT, make sure we properly lock
everything and hold references on devices that
we are using.

The two peripheral driver traversal routines should
be examined. xptpdperiphtraverse() holds the
topology lock for the entire time it runs.
xptperiphtraverse() is now locked properly, but
only holds the topology lock while it is traversing
the list, and not while the traversal function is
running.

The bus locking code in xptbustraverse() should
also be revisited at a later time, since it is
complex and should probably be simplified.

scsi_da.c: Pay attention to the return value from cam_periph_acquire().

Return 0 always from daclose() even if the disk is now gone.

Add some rudimentary error injection support.

scsi_sg.c: Fix reference counting in the sg(4) driver.

The sg driver was calling cam_periph_release() on close,
but never called cam_periph_acquire() (which increments
the reference count) on open.

The periph code correctly complained that the sg(4)
driver was trying to decrement the refcount when it
was already 0.

Sponsored by: Spectra Logic
MFC after: 2 weeks

Move the scsi_da_bios_params() prototype from pc98_machdep.h to md_var.h
where the prototype for pc98_ata_disk_firmware_geom_adjust() also lives
in order to avoid an #ifdef'ed include in cam(4).

For sparc64 also adjust the geometry of da(4) driven disks to not overflow
the 16-bit cylinders field of the VTOC8 disk label (at around 502GB). The
geometry chosen for disks above that limit allows to use disks up to 2TB,
which is the limit of the extended VTOC8 format. The geometry used for
disks smaller than the 16-bit cylinders limit stays the same as used by
cam_calc_geometry(9) for extended translation.
Thanks to Hans-Joerg Sirtl for providing hardware for testing this change.

MFC after: 3 days

Introduce CAM_SIM_POLLED SIM flag, indicating that it works in polling mode.
It blocks CAM SWI usage on requests completion, unneeded because of polling
and denied during kernel dumping because of blocked scheduler.

Before r198899 there was periph flag CAM_PERIPH_POLLED, but that was wrong,
because there is whole SIM is polled or handled by SWI, not a single periph.

Tested by: kib
MFC after: 1 month

Add support for SATA Enclosure Management Bridge (SEMB).

Fixes for sure bus reference miscounting and potential device and
target reference miscounts. It also adds a helper function to get
the current reference counts for components of cam_path for debug
aid. One minor style(9) change.

Partially Obtained from: Chuck Tuffli (Emulex)
Reviewed by: scsi@ (ken)
Approved by: re (kib)
MFC after: 1 month

cam/cam_xpt.c:
In camisr_runqueue(), we need to run the sims queue regardless of
whether or not the current peripheral has more work to do. This
reverts a change mistakenly made in revision 223081.

Reported by: ache

Lay groundwork in CAM for recording and reporting physical path and
other device attributes stored in the CAM Existing Device Table (EDT).
This includes some infrastructure requried by the enclosure services
driver to export physical path information.

Make the CAM device advanced info interface accept store requests.

sys/cam/scsi/scsi_all.c:
sys/cam/scsi/scsi_all.h:
- Replace scsi_get_sas_addr() with a scsi_get_devid() which takes
a callback that decides whether to accept a particular descriptor.
Provide callbacks for NAA IEEE Registered addresses and for SAS
addresses, replacing the old function. This is needed because
the old function doesn't work for an enclosure address for a SAS
device, which is not flagged as a SAS address, but is NAA IEEE
Registered. It may be worthwhile merging this interface with the
devid match interface.
- Add a few more defines for some device ID fields.

sbin/camcontrol/camcontrol.c:
- Update for the CCB_DEV_ADVINFO interface change.

cam/cam_xpt_internal.h:
- Add the new fields for the physical path string to the CAM EDT.
cam/cam_ccb.h:
- Rename CCB_GDEV_ADVINFO to simply CCB_DEV_ADVINFO, and the ccb
structure to ccb_dev_advinfo.
- Add a flag that changes this CCB's action to store, rather than
the default, retrieve.
- Add a new buffer type, CDAI_TYPE_PHYS_PATH, for the new CAM EDT
physpath field.
- Remove the never-implemented transport & proto flags.
cam/cam_xpt.c:
cam/cam_xpt.h:
- Add xpt_getattr(), which provides a wrapper for fetching a device's
attribute using the GEOM strings as key. This method currently
supports "GEOM::ident" and "GEOM::physpath".

Submitted by: will
Reviewed by : gibbs

Extend the XPT_DEV_MATCH api to allow a device search by device ID.
As far as the API is concerned, device ID is a binary blob to be
interpreted by the transport layer. The SCSI implementation assumes
it is an array of VPD device ID descriptors.

sys/cam/cam_ccb.h:
Create a new structure, device_id_match_pattern, and
update the XPT_DEV_MATCH datastructures and flags so
that this pattern type can be used.

sys/cam/cam_xpt.c:
- A single pattern matching on both inquiry data and device
ID is invalid. Report any violators.
- Pass device ID match requests through to the new routine
scsi_devid_match(). The direct call of a SCSI routine is
a layering violation, but no worse than the one a few
lines up that checks inquiry data. Defer cleaning this
up until our future, larger, rototilling of CAM.
- Zero out cam_ed and cam_et nodes on allocation. Prior to
this change, device_id_len and device_id were not inialized,
preventing proper detection of the presence of this
information.

sys/cam/scsi/scsi_all.c:
sys/cam/scsi/scsi_all.h:
Add the scsi_match_devid() routine.

Add a helper function for extracting peripherial driver names

sys/cam/cam_periph.c:
sys/cam/cam_periph.h:
Add the cam_periph_list() method which fills an sbuf
with a comma delimited list of the peripheral instances
associated with a given CAM path.

Add a helper functions for SCSI commands used by the SES driver.

sys/cam/scsi/scsi_all.c:
sys/cam/scsi/scsi_all.h:
Add structure definitions and csio filling functions for
the receive diagnostic results and send diagnostic commands.

Misc CAM XPT cleanups.

sys/cam/cam_xpt.c:
Broadcast AC_FOUND_DEVICE and AC_PATH_REGISTERED
events at the time async event handlers are attached
even when registering just for events on a partitular
SIM. Previously, you had to register for these
events on all SIMs in the system in order to get
the initial broadcast even though subsequent device
and path arrivals would be delivered.

sys/cam/cam_xpt.c:
Remove SIM mutex held asserts from path accessors.
CAM paths are reference counted and it is this
reference count, not the sim mutex, that garantees
they are stable.

Sponsored by: Spectra Logic Corporation

Do not report legacy unit numbers (do not create legacy aliases) for disks
on port multiplier ports above first two. They don't fit into ATA_STATIC_ID
scheme and so can't be mapped properly. No need to pollute dev.

Make CAM_DEBUG_CDB also dump ATA commands in addition to SCSI.

- Add shim to simplify migration to the CAM-based ATA. For each new adaX
device in /dev/ create symbolic link with adY name, trying to mimic old ATA
numbering. Imitation is not complete, but should be enough in most cases to
mount file systems without touching /etc/fstab.
- To know what behavior to mimic, restore ATA_STATIC_ID option in cases
where it was present before.
- Add some more details to UPDATING.

Remove some used variables.

Found with: Clang Static Analyzer

Fix typos - remove duplicate "the".

PR: bin/154928
Submitted by: Eitan Adler <lists at eitanadler.com>
MFC after: 3 days

Fix up a few more sysctl(9) mis-typing found in various LINT builds.

Add Serial Management Protocol (SMP) passthrough support to CAM.

This includes support in the kernel, camcontrol(8), libcam and the mps(4)
driver for SMP passthrough.

The CAM SCSI probe code has been modified to fetch Inquiry VPD page 0x00
to determine supported pages, and will now fetch page 0x83 in addition to
page 0x80 if supported.

Add two new CAM CCBs, XPT_SMP_IO, and XPT_GDEV_ADVINFO. The SMP CCB is
intended for SMP requests and responses. The ADVINFO is currently used to
fetch cached VPD page 0x83 data from the transport layer, but is intended
to be extensible to fetch other types of device-specific data.

SMP-only devices are not currently represented in the CAM topology, and so
the current semantics are that the SIM will route SMP CCBs to either the
addressed device, if it contains an SMP target, or its parent, if it
contains an SMP target. (This is noted in cam_ccb.h, since it will change
later once we have the ability to have SMP-only devices in CAM's topology.)

smp_all.c,
smp_all.h: New helper routines for SMP. This includes
SMP request building routines, response parsing
routines, error decoding routines, and structure
definitions for a number of SMP commands.

libcam/Makefile: Add smp_all.c to libcam, so that SMP functionality
is available to userland applications.

camcontrol.8,
camcontrol.c: Add smp passthrough support to camcontrol. Several
new subcommands are now available:

'smpcmd' functions much like 'cmd', except that it
allows the user to send generic SMP commands.

'smprg' sends the SMP report general command, and
displays the decoded output. It will automatically
fetch extended output if it is available.

'smppc' sends the SMP phy control command, with any
number of potential options. Among other things,
this allows the user to reset a phy on a SAS
expander, or disable a phy on an expander.

'smpmaninfo' sends the SMP report manufacturer
information and displays the decoded output.

'smpphylist' displays a list of phys on an
expander, and the CAM devices attached to those
phys, if any.

cam.h,
cam.c: Add a status value for SMP errors
(CAM_SMP_STATUS_ERROR).

Add a missing description for CAM_SCSI_IT_NEXUS_LOST.

Add support for SMP commands to cam_error_string().

cam_ccb.h: Rename the CAM_DIR_RESV flag to CAM_DIR_BOTH. SMP
commands are by nature bi-directional, and we may
need to support bi-directional SCSI commands later.

Add the XPT_SMP_IO CCB. Since SMP commands are
bi-directional, there are pointers for both the
request and response.

Add a fill routine for SMP CCBs.

Add the XPT_GDEV_ADVINFO CCB. This is currently
used to fetch cached page 0x83 data from the
transport later, but is extensible to fetch many
other types of data.

cam_periph.c: Add support in cam_periph_mapmem() for XPT_SMP_IO
and XPT_GDEV_ADVINFO CCBs.

cam_xpt.c: Add support for executing XPT_SMP_IO CCBs.

cam_xpt_internal.h: Add fields for VPD pages 0x00 and 0x83 in struct
cam_ed.

scsi_all.c: Add scsi_get_sas_addr(), a function that parses
VPD page 0x83 data and pulls out a SAS address.

scsi_all.h: Add VPD page 0x00 and 0x83 structures, and a
prototype for scsi_get_sas_addr().

scsi_pass.c: Add support for mapping buffers in XPT_SMP_IO and
XPT_GDEV_ADVINFO CCBs.

scsi_xpt.c: In the SCSI probe code, first ask the device for
VPD page 0x00. If any VPD pages are supported,
that page is required to be implemented. Based on
the response, we may probe for the serial number
(page 0x80) or device id (page 0x83).

Add support for the XPT_GDEV_ADVINFO CCB.

sys/conf/files: Add smp_all.c.

mps.c: Add support for passing in a uio in mps_map_command(),
so we can map a S/G list at once.

Add support for SMP passthrough commands in
mps_data_cb(). SMP is a special case, because the
first buffer in the S/G list is outbound and the
second buffer is inbound.

Add support for warning the user if the busdma code
comes back with more buffers than will work for the
command. This will, for example, help the user
determine why an SMP command failed if busdma comes
back with three buffers.

mps_pci.c: Add sys/uio.h.

mps_sas.c: Add the SAS address and the parent handle to the
list of fields we pull from device page 0 and cache
in struct mpssas_target. These are needed for SMP
passthrough.

Add support for the XPT_SMP_IO CCB. For now, this
CCB is routed to the addressed device if it supports
SMP, or to its parent if it does not and the parent
does. This is necessary because CAM does not
currently support SMP-only nodes in the topology.

Make SMP passthrough support conditional on
__FreeBSD_version >= 900026. This will make it
easier to MFC this change to the driver without
MFCing the CAM changes as well.

mps_user.c: Un-staticize mpi_init_sge() so we can use it for
the SMP passthrough code.

mpsvar.h: Add a uio and iovecs into struct mps_command for
SMP passthrough commands.

Add a cm_max_segs field to struct mps_command so
that we can warn the user if busdma comes back with
too many segments.

Clear the cm_reply when a command gets freed. If
it is not cleared, reply frames will eventually get
freed into the pool multiple times and corrupt the
pool. (This fix is from scottl.)

Add a prototype for mpi_init_sge().

sys/param.h: Bump __FreeBSD_version to 900026 for the for the
inclusion of the XPT_GDEV_ADVINFO and XPT_SMP_IO
CAM CCBs.

Mostly revert r203420, and add similar functionality into ada(4) since the
existing code caused problems with some SCSI controllers.

A new sysctl kern.cam.ada.spindown_shutdown has been added that controls
whether or not to spin-down disks when shutting down.
Spinning down the disks unloads/parks the heads - this is
much better than removing power when the disk is still
spinning because otherwise an Emergency Unload occurs which may cause damage
to the actuator.

PR: kern/140752
Submitted by: olli
Reviewed by: arundel
Discussed with: mav
MFC after: 2 weeks

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Decrease poll interval from 1000 to 100us. This significantly reduces
kernel dump time, especially with minidump.

Don't lock buses around a call to xptperiphlistmatch- the buses will be
locked at appropriate places.

MFC after: 1 week
X-MFC: 208752

Implement the usage of Report Luns as part of SCSI probing for SCP3 or
better devices. This can be disabled on a per-device basis using quirks as
well.

This also handles the case where there is actually no connected LUN 0
(which can definitely be the case for storage arrays).

Reviewed by: scsi@
MFC after: 1 month

Protect periph drivers list and rearrange things to minimize the chance of
stepping oneself during probing.

Don't blindly decrement a periph probe count.

Reviewed by: scsi@
Obtained from: Alexander Motin, Atillio Rao, Others
MFC after: 1 month

Add a new primitive, XPT_SCAN_TGT, to cover the range between scanning a
whole bus (XPT_SCAN_BUS) and a single lun on that bus (XPT_SCAN_LUN).

It's less resource comsumptive than scanning a whole bus when the
caller knows only one target has changes.

Reviewed by: scsi@
Sponsored by: Panasas
MFC after: 1 month

MFC r207433:
Revert r198705.
As scottl@ noticed, max_target/max_lun was intended to be only a hint for
existing bus scanner. Some FC/SAS SIMs report fake values there, that are
smaller then maximum supported IDs. In that case this check makes impossible
manual scan outside hinted range.

For ATA/SATA SIMs respective check was instead implemented at SIM level.
Newer SCSI SIMs expected to have these checks at driver or firmware level.
Some older SCSI SIMs have no this check and the issues will get back there.

MFC r207490:
Add xpt_schedule_dev_sendq() call, lost at r203108. It is not needed in
usual operation, but required in some conditions to make queue running
after being shrinked.

Add xpt_schedule_dev_sendq() call, lost at r203108. It is not needed in
usual operation, but required in some conditions to make queue running
after being shrinked.

MFC after: 3 days

Revert r198705.
As scottl@ noticed, max_target/max_lun was intended to be only a hint for
existing bus scanner. Some FC/SAS SIMs report fake values there, that are
smaller then maximum supported IDs. In that case this check makes impossible
manual scan outside hinted range.

For ATA/SATA SIMs respective check was instead implemented at SIM level.
Newer SCSI SIMs expected to have these checks at driver or firmware level.
Some older SCSI SIMs have no this check and the issues will get back there.

This is an MFC of 205993

For unhandled actions in xpt_action_default, remember to call
xpt_done for queued requests. This solves the problem of
indefinite hangs for unspecified transports when XPT_SCAN_BUS
is called.

A few minor cosmetics elsewhere.

For unhandled actions in xpt_action_default, remember to call
xpt_done for queued requests. This solves the problem of
indefinite hangs for unspecified transports when XPT_SCAN_BUS
is called.

A few minor cosmetics elsewhere.

MFC after: 1 week

MFC r204253:
Make xpt_rescan() more correct, as it was before r197208: do not use
XPT_SCAN_LUN for wildcard LUN, fall back to XPT_SCAN_BUS.

MFC r204220, r204353:
Virtualize transport part of periph announcement.

MFC r204195:
Improve output for controllers that doesn't report SATA speed.

Fix newlines broken at r204220.

Make xpt_rescan() more correct, as it was before r197208: do not use
XPT_SCAN_LUN for wildcard LUN, fall back to XPT_SCAN_BUS.

Virtualize transport part of periph announcement.

Improve output for controllers that doesn't report SATA speed.

MFC r203386:
Change the way in which fake async events generated. Do not use
taskqueue for lock decoupling, as it causes unwanted races.

MFC r203376, r203384:
- Give ATA/SATA SIMs info about ATAPI packet size, supported by device.
- Make ATA XPT to reject longer SCSI CDBs then supported by device, or
any SCSI CDBs, if device doesn't support ATAPI.

MFC r203108:
Large set of CAM improvements:
- Unify bus reset/probe sequence. Whenever bus attached at boot or later,
CAM will automatically reset and scan it. It allows to remove duplicate
code from many drivers.
- Any bus, attached before CAM completed it's boot-time initialization,
will equally join to the process, delaying boot if needed.
- New kern.cam.boot_delay loader tunable should help controllers that
are still unable to register their buses in time (such as slow USB/
PCCard/ CardBus devices), by adding one more event to wait on boot.
- To allow synchronization between different CAM levels, concept of
requests priorities was extended. Priorities now split between several
"run levels". Device can be freezed at specified level, allowing higher
priority requests to pass. For example, no payload requests allowed,
until PMP driver enable port. ATA XPT negotiate transfer parameters,
periph driver configure caching and so on.
- Frozen requests are no more counted by request allocation scheduler.
It fixes deadlocks, when frozen low priority payload requests occupying
slots, required by higher levels to manage theit execution.
- Two last changes were holding proper ATA reinitialization and error
recovery implementation. Now it is done: SATA controllers and Port
Multipliers now implement automatic hot-plug and should correctly
recover from timeouts and bus resets.
- Improve SCSI error recovery for devices on buses without automatic sense
reporting, such as ATAPI or USB. For example, it allows CAM to wait, while
CD drive loads disk, instead of immediately return error status.
- Decapitalize diagnostic messages and make them more readable and sensible.
- Teach PMP driver to limit maximum speed on fan-out ports.
- Make boot wait for PMP scan completes, and make rescan more reliable.
- Fix pass driver, to return CCB to user level in case of error.
- Increase number of retries in cd driver, as device may return several UAs.

Disable kern.cam.power_down ATM. It doesn't work fine on some controllers.

MFp4:
Make CAM to stop all attached devices on system shutdown.
It allows devices to park heads, reducing stress on power loss.
Add `kern.cam.power_down` tunable and sysctl to controll it.

Change the way in which fake async events generated. Do not use
taskqueue for lock decoupling, as it causes unwanted races.

Some missed parts for r203376.

MFp4: Large set of CAM inprovements.

- Unify bus reset/probe sequence. Whenever bus attached at boot or later,
CAM will automatically reset and scan it. It allows to remove duplicate
code from many drivers.
- Any bus, attached before CAM completed it's boot-time initialization,
will equally join to the process, delaying boot if needed.
- New kern.cam.boot_delay loader tunable should help controllers that
are still unable to register their buses in time (such as slow USB/
PCCard/ CardBus devices), by adding one more event to wait on boot.
- To allow synchronization between different CAM levels, concept of
requests priorities was extended. Priorities now split between several
"run levels". Device can be freezed at specified level, allowing higher
priority requests to pass. For example, no payload requests allowed,
until PMP driver enable port. ATA XPT negotiate transfer parameters,
periph driver configure caching and so on.
- Frozen requests are no more counted by request allocation scheduler.
It fixes deadlocks, when frozen low priority payload requests occupying
slots, required by higher levels to manage theit execution.
- Two last changes were holding proper ATA reinitialization and error
recovery implementation. Now it is done: SATA controllers and Port
Multipliers now implement automatic hot-plug and should correctly
recover from timeouts and bus resets.
- Improve SCSI error recovery for devices on buses without automatic sense
reporting, such as ATAPI or USB. For example, it allows CAM to wait, while
CD drive loads disk, instead of immediately return error status.
- Decapitalize diagnostic messages and make them more readable and sensible.
- Teach PMP driver to limit maximum speed on fan-out ports.
- Make boot wait for PMP scan completes, and make rescan more reliable.
- Fix pass driver, to return CCB to user level in case of error.
- Increase number of retries in cd driver, as device may return several UAs.

MFC r199747, r199799, r199821:
- Extend XPT-SIM transfer settings control API. Now it allows to report to
SATA SIM number of tags supported by each device, implement ATA mode and
SATA revision negotiation for both SATA and PATA SIMs.
- Make ahci(4) and siis(4) to use submitted maximum tag number, when
scheduling requests. It allows to support NCQ on devices with lower tags
count then controller supports.
- Make PMP driver to report attached devices connection speeds.
- Implement ATA mode negotiation between user settings, device and
controller capabilities.
- Improve ATA mode/SATA revision control.

MFC r199279, r199280, r199281:
- Fix several device freeze counting bugs.
- Remove code that years ago was closing race between request submission
to SIM and device/SIM freeze. That race become impossible after moving from
spl to mutex locking, while this workaround causes some unexpected effects.

MFp4:
- Extend XPT-SIM transfer settings control API. Now it allows to report to
SATA SIM number of tags supported by each device, implement ATA mode and
SATA revision negotiation for both SATA and PATA SIMs.
- Make ahci(4) and siis(4) to use submitted maximum tag number, when
scheduling requests. It allows to support NCQ on devices with lower tags
count then controller supports.
- Make PMP driver to report attached devices connection speeds.
- Implement ATA mode negotiation between user settings, device and
controller capabilities.

MFC r199178:
- Move tagged queueing control from ADA to ATA XPT. It allows to control
device command queue length correctly. First step to support < 32 tags.
- Limit queue for non-tagged devices by 2 slots for ahci(4) and siis(4).
- Implement quirk matching for ATA devices.
- Move xpt_schedule_dev_sendq() from header to source file.
- Move delayed queue shrinking to the more expected place - element freeing.
- Remove some SCSIsms in ATA.

MFC r198899:
- Remove CAM_PERIPH_POLLED flag. It is broken by design. Polling can't be
periph flag. May be SIM, may be CCB, but now it works fine just without it.
- Remove check unused for at least five years. If we will ever have non-BIO
devices in CAM, this check is smallest of what we will need.
- If several controllers complete requests same time, call swi_sched()
only once.

MFC r198748, r198782:
Fix reference counting bug, when device unreferenced before
invalidated. To do it, do not handle validity flag as another
reference, but explicitly modify reference count each time flag is
modified.
The async callback could free the device. If it is a broadcast async,
it doesn't hold device reference, so take our own reference.

MFC r198708:
- Reduce code duplication in ATA XPT and PMP driver.
- Move PIO size setting from ada driver to ATA XPT. It is XPT business
to negotiate transfer details. ada driver is now stateless.
- Report PIO size to SIM. It is required for correct PATA SIM operation.
- Tune PMP scan timings. It workarounds some problems with SiI.
- If reset hapens during PMP initialization - restart it.
- Introduce early-initialized periph drivers, which are used during initial
scan process. Use it for xpt, probe, aprobe and pmp. It gives pmp chance
to finish scan before mountroot and numerate devices in right order.

MFC r198705:
Ensure target/lun passed from user-level supported on this bus.
Scanning unsupported IDs causes different issues from duplicate
devices to system crash.

MFC r198382, r198385:
Replace most of priority numbers with defines. No logical changes.

MFC r198372, r198377:
Implement cam_ccbq_fini().
This is effectively NULL change, but makes this API a bit more consistent.

MFC r198333:
Do not search for bus when it is not needed.

MFC r196903:
Remove duplicate qfrozen_cnt variable from struct cam_ed.
ccbq.queue.qfrozen_cnt should be used instead.

MFC r196901:
Remove unneeded CAM_SIM_MPSAFE check.

MFC r196898:
Report scbusX in xpt_announce_periph() to less confuse users by two
different bus addressing schemes.

MFC r196900:
s/bus %d/scbus%d/ in some messages to correct terminology.

MFC r196897:
Avoid extra swi_sched() call, if this SIM is already queued.
It reduces overhead for coalesced command completions.

MFp4:
Remove code that years ago was closing race between request submission
to SIM and device/SIM freeze. That race become impossible after moving from
spl to mutex locking, while this workaround causes some unexpected effects.

MFp4:
- Move tagged queueing control from ADA to ATA XPT. It allows to control
device command queue length correctly. First step to support < 32 tags.
- Limit queue for non-tagged devices by 2 slots for ahci(4) and siis(4).
- Implement quirk matching for ATA devices.
- Move xpt_schedule_dev_sendq() from header to source file.
- Move delayed queue shrinking to the more expected place - element freeing.
- Remove some SCSIsms in ATA.

MFp4:
- Remove CAM_PERIPH_POLLED flag. It is broken by design. Polling can't be
periph flag. May be SIM, may be CCB, but now it works fine just without it.
- Remove check unused for at least five years. If we will ever have non-BIO
devices in CAM, this check is smallest of what we will need.
- If several controllers complete requests same time, call swi_sched()
only once.

The async callback could free the device. If it is a broadcast async,
it doesn't hold device reference, so take our own reference.

Submitted by: thompsa

MFp4:
Fix reference counting bug, when device unreferenced before then
invalidated. To do it, do not handle validity flag as another
reference, but explicitly modify reference count each time flag is
modified.

Discovered by: thompsa

MFp4:
- Reduce code duplication in ATA XPT and PMP driver.
- Move PIO size setting from ada driver to ATA XPT. It is XPT business
to negotiate transfer details. ada driver is now stateless.
- Report PIO size to SIM. It is required for correct PATA SIM operation.
- Tune PMP scan timings. It workarounds some problems with SiI.
- If reset hapens during PMP initialization - restart it.
- Introduce early-initialized periph drivers, which are used during initial
scan process. Use it for xpt, probe, aprobe and pmp. It gives pmp chance
to finish scan before mountroot and numerate devices in right order.

MFp4:
Ensure target/lun passed from user-level supported on this bus.
Scanning unsupported IDs causes different issues from duplicate
devices to system crash.

Fix the build.

Replace most of priority numbers with defines. No logical changes.

After thinking again, implement cam_ccbq_fini().
This is effectively NULL change, but makes this API a bit more consistent.

MFp4:
Do not search for bus when it is not needed,

Merge change r197208 from head to stable/8:

Fix some instances where CAM rescans get hung up or take a long time to
complete.

Also, allow xpt_rescan() to rescan a LUN instead of a full bus.

Sponsored by: Copan Systems, Inc.
Approved by: re (kib)

Fix some instances where CAM rescans get hung up or take a long time to
complete.

Also, allow xpt_rescan() to rescan a LUN instead of a full bus.

MFC after: 3 days
Sponsored by: Copan Systems, Inc.

MFp4:
Remove duplicate qfrozen_cnt variable from struct cam_ed.
ccbq.queue.qfrozen_cnt should be used instead.

Remove unneeded CAM_SIM_MPSAFE check.

s/bus %d/scbus%d/ in some messages to correct terminology.

MFp4:
Report scbusX in xpt_announce_periph() to less confuse users by two
different bus addressing schemes.

Avoid extra swi_sched() call, if this SIM is already queued.
It reduces overhead for coalesced command completions.

MFC r196352:
Fix iSCSI initiator and vpo driver operation, broken by CAM changes.

Reviewed by: scottl, Danny Braniss
Approved by: re (rwatson)

Fix iSCSI initiator and vpo driver operation, broken by CAM changes.

Reviewed by: scottl, Danny Braniss
Approved by: re (rwatson)

Add 8Gb support (isp_2500). Fix a fair number of configuration and
firmware loading bugs.

Target mode support has received some serious attention to make it
more usable and stable.

Some backward compatible additions to CAM have been made that make
target mode async events easier to deal with have also been put
into place.

Further refinement and better support for NP-IV (N-port Virtualization)
is now in place.

Code for release prior to RELENG_7 has been stripped away for code clarity.

Sponsored by: Copan Systems

Reviewed by: scottl, ken, jung-uk kim
Approved by: re

Separate the parallel scsi knowledge out of the core of the XPT, and
modularize it so that new transports can be created.

Add a transport for SATA

Add a periph+protocol layer for ATA

Add a driver for AHCI-compliant hardware.

Add a maxio field to CAM so that drivers can advertise their max
I/O capability. Modify various drivers so that they are insulated
from the value of MAXPHYS.

The new ATA/SATA code supports AHCI-compliant hardware, and will override
the classic ATA driver if it is loaded as a module at boot time or compiled
into the kernel. The stack now support NCQ (tagged queueing) for increased
performance on modern SATA drives. It also supports port multipliers.

ATA drives are accessed via 'ada' device nodes. ATAPI drives are
accessed via 'cd' device nodes. They can all be enumerated and manipulated
via camcontrol, just like SCSI drives. SCSI commands are not translated to
their ATA equivalents; ATA native commands are used throughout the entire
stack, including camcontrol. See the camcontrol manpage for further
details. Testing this code may require that you update your fstab, and
possibly modify your BIOS to enable AHCI functionality, if available.

This code is very experimental at the moment. The userland ABI/API has
changed, so applications will need to be recompiled. It may change
further in the near future. The 'ada' device name may also change as
more infrastructure is completed in this project. The goal is to
eventually put all CAM busses and devices until newbus, allowing for
interesting topology and management options.

Few functional changes will be seen with existing SCSI/SAS/FC drivers,
though the userland ABI has still changed. In the future, transports
specific modules for SAS and FC may appear in order to better support
the topologies and capabilities of these technologies.

The modularization of CAM and the addition of the ATA/SATA modules is
meant to break CAM out of the mold of being specific to SCSI, letting it
grow to be a framework for arbitrary transports and protocols. It also
allows drivers to be written to support discrete hardware without
jeopardizing the stability of non-related hardware. While only an AHCI
driver is provided now, a Silicon Image driver is also in the works.
Drivers for ICH1-4, ICH5-6, PIIX, classic IDE, and any other hardware
is possible and encouraged. Help with new transports is also encouraged.

Submitted by: scottl, mav
Approved by: re

Revert r190676,190677

The geom and CAM changes for root_hold are the wrong solution for USB design
quirks.

Requested by: scottl

Add interleaving root hold tokens from the CAM probe to disk_create and geom
provider tasting. This is needed for disk attachments that happen after threads
are running in the boot process.

Tested by: rnoland

GCC attacks!

Fix parallel SCSI negotiation in the CAM_NEW_TRAN_CODE world order.
Overzealous sanity checks were locking the sync_rate and offset values to
zero, thanks to a twisty maze of recursive code.

Instrument the probe state machine so that things like DV can be tracked.

In the case that the probe has determined that it can't query the device for
a serial number, fall through to the next case so that initial negotiation
still happens. Without this, devices were showing up with only 1 available
tag opening, leading to observations of very poor I/O performance.

This should fix problems reported with VMWare Fusion and ESX. Early
generation MPT-SAS controllers with SATA disks might also be affected.
HP CISS controllers are also likely affected, as are many other
pseudo-scsi disk subsystems.

Remove an overzealous check.

Submitted by: das
Reviewed by: scottl
Approved by: rwatson (mentor, implicit)
Sponsored by: FreeBSD Foundation

Protect against NULL pointer dereference.

Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation

Guard against NULL pointer dereference.

Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation
Found with: Coverity Prevent(tm)
CID: 1847

Fix use after free.

Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation
Found with: Coverity Prevent(tm)
CID: 3712

Make "kldunload atapicam" return EBUSY instead of deadlocking when a device
created by atapicam is being kept opened or mounted. This is probably just
a temporary solution until we invent something better.

Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation
Reported by: Jaakko Heinonen

cam: Retry TEST UNIT READY command if not successful.

This fixes problems with discovering some USB devices that are very slow to
respond during initialisation.

When a USB device is inserted, CAM performs the sequence:
1) INQUIRY
2) INQUIRY (second time with other parameters)
3) TEST UNIT READY
4) READ CAPACITY

Before this change CAM didn't check if TEST UNIT READY was successful and went
on blindly to the next state and sent READ CAPACITY. If the device was still
not ready by then, CAM ended with error message. This patch adds checking for the
status of TEST UNIT READY command and retrying up to 10 times with 0.5 sec
interval.

Submitted by: Grzegorz Bernacki gjb ! semihalf dot com
Reviewed by: scottl

Periph driver fixes, second try.

Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation

Revert r186186 for now; it breaks stuff.

Approved by: rwatson (mentor)

Fix locking in periph drivers - don't try to unlock periph
that was already deallocated.

Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation

Add SIM refcounting. This is slightly different from what DragonFly
does - in DragonFly, it's cam_sim_release() what actually frees the
SIM; cam_sim_free does nothing more than calling cam_sim_release().
Here, we drain in cam_sim_free, waiting for refcount to drop to zero.
We cannot do the same think DragonFly does, because after cam_sim_free
returns, client would destroy the sim->mtx, and CAM would trip over
an initialized mutex.

Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation

Get rid of dead_sim. There is no way to make it work - any attempt
to actually use it would panic on mtx operation, as dead_sim doesn't
have a proper mutex. Even if it had a properly initialized mutex,
it wouldn't have properly locked and owned one.

Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation

- Import the HEAD csup code which is the basis for the cvsmode work.

Fix whitespace.

When probing a newly found device, don't automatically assume that the
device supports retrieving a serial number. Instead, first query the
list of VPD pages it does support, and only query the serial number if
it's supported, else silently move on. This eliminates a lot of noise
during verbose booting, and will likely eliminate the need for most
NOSERIAL quirks.

Add a missing unlock to an error path fro XPT_DEBUG handling.

Correct the kernel config hint in a printf statement.

Submitted by: R.Mahmatkhanov <cvs-src@yandex.ru>
Reviewed by: scottl
Approved by: rwatson (mentor)
MFC after: 3 days

style(9)

Add a missing mutex unlock.

Reported by: Michael Jung

Rename the kthread_xxx (e.g. kthread_create()) calls
to kproc_xxx as they actually make whole processes.
Thos makes way for us to add REAL kthread_create() and friends
that actually make theads. it turns out that most of these
calls actually end up being moved back to the thread version
when it's added. but we need to make this cosmetic change first.

I'd LOVE to do this rename in 7.0 so that we can eventually MFC the
new kthread_xxx() calls.

Quirk for WD Mybook. Seems they do not like serial number queries.

Submitted by: Jason Harris
PR: 107495
Approved by: re@ (blanket)

Prepare for future integration between CAM and newbus. xpt_bus_register
now takes a device_t to be the parent of the bus that is being created.
Most SIMs have been updated with a reasonable argument, but a few exceptions
just pass NULL for now. This argument isn't used yet and the newbus
integration likely won't be ready until after 7.0-RELEASE.

Despite several examples in the kernel, the third argument of
sysctl_handle_int is not sizeof the int type you want to export.
The type must always be an int or an unsigned int.

Remove the instances where a sizeof(variable) is passed to stop
people accidently cut and pasting these examples.

In a few places this was sysctl_handle_int was being used on 64 bit
types, which would truncate the value to be exported. In these
cases use sysctl_handle_quad to export them and change the format
to Q so that sysctl(1) can still print them.

Release the bus reference that is acquired when doing a CAMIOCOMMAND
ioctl. This should make SIM removal a bit more robust.

Remove an assertion in xpt_print_path() that makes debugging code much more
difficult.

Add a helper function for registering async callbacks. Besides
eliminating a lot of duplicated code, this also fixes a locking edge case.

Eliminate the use of M_TEMP.

Only schedule the xpt_finishconfig_task once. This fixes some potential
panics on boot.

Retire the spl() markers. Add in some minor missed locking as a result.

Inline cam_periph_lock|unlock to make debugging easier. Use more
CAM_SIM_LOCK() more uniformly.

Zero the CCBs when mallocing them.

Split the camisr into per-SIM done queues. This optimizes the locking a
little bit and allows for direct dispatch of the doneq from certain
contexts that would otherwise face recursive locking problems.

Revert a driver API change to xpt_alloc_ccb that isn't necessary. Fix a
couple of associated error checks.

For the XPT_SASYNC_CB operation, only decouple the broadcast to the bus
and device lists instead of decoupling the whole operation. This avoids
problems with SIMs going away.

Remove Giant from CAM. Drivers (SIMs) now register a mutex that CAM will
use to synchornize and protect all data objects that are used for that
SIM. Drivers that are not yet MPSAFE register Giant and operate as
usual. RIght now, no drivers are MPSAFE, though a few will be changed
in the coming week as this work settles down.

The driver API has changed, so all CAM drivers will need to be recompiled.
The userland API has not changed, so tools like camcontrol do not need to
be recompiled.

A silly buglet found by Coverity- check the return value from
cam_periph_acquire.

Add an xpt_rescan function and a thread that will field
rescan requests. The purpose of this is to allow a SIM
(or other entities) to request a bus rescan and have it
then fielded in a different (process) context from the
caller.

There are probably better ways to accomplish this, but
it's a very small change that helps solve a number of
problems.

Reviewed by: Justin, Ken and Scott.
MFC after: 2 weeks

fix botch

Pay attention to return value from xpt_bus_register in xpt_init.

Obtained from: Xin Li (Coverity)
MFC after: 3 days

Make cam_xpt's pronouncements match camcontrol
(Tagged -> Command) Queueing.

Add an xpt_print function to reduce most of the xpt_print_path/printf
pairs. Convert the core code to use it.

Reviewed by: scsi (various folks weighing in with good advice)
MFC after: 2 weeks

Initial cut at Basic Domain Validation- just add some states to
traverse through during probing.

Reviewed by: scsi (scottl)

XPT_GET_TRAN_SETTINGS can fail- so check for an error and
don't blindly assume it succeeded.

Fix the hanging chad if the NEW_TRAN_CODE change and change the
qualifier to force async from cur_spi to spi.

2nd and final commit that moves us to CAM_NEW_TRAN_CODE
as the default.

Reviewed by multitudes.

New Dell 1950/2950 SES backplane drops off the bus if you poke
at greater then lun 0.

MFC after: 1 week

null commit to provide commit message to previous
at the request of Sam Leffler: The previous commit
established min and maxtags for VMware pseudo disks
to fix a submitted PR.

PR: 103130
Submitted by: Shusuke Shinomiya
MFC after: 1 day

Grumble. VMWare ESX and VMWare WorkStation have *slightly* different
inquiry data (pointlessly different I might add). Pick the common
parts.

MFC after: 1 day

VMware disk volumes are only on LUN 0, and
some VMware HBAs have known bugs with commands
sent to other than LUN 0.

MFC after: 1 day

Add PIM_SEQSCAN for HBA misc flags and code that understands
what to do with it.

This forces us to scan targets sequentially, not in parallel.
The reason we might want to do this is that SPI negotiation
might not work right at the SIM level if we try to do it
in parallel. We *could* fix this for each SIM where this is
broken, but it's a lot harder to do that when we can simply
ask CAM to probe sequentially.

If PIM_SEQSCAN is not set (default), the original behaviour for
probing is unchanged.

LUN probing is still done in parallel for each target in either
case.

While we're at it, clean up some resource leakage for error
cases.

Reviewed by: ken, scott, scsi@
MFC after: 1 week

Handle some of the inquiry flags that have come into
usage as of SPC2r20. Specifically, handle the BQueue
flag which will indicate that a device supports the
Basic Queueing model (no Head of Queue or Ordered tags).
When this flag is set, SID_CmdQueue is clear. This has
causes FreeBSD to assume that the device did not support
tagged operations.

MFC after: 1 month

CAM_NEW_TRAN_CODE placeholders for SAS support.

Reviewed by: nate, ken, justin
MFC after: 1 month

When deregistering a bus, attempt to flush out all outstanding
operations before returning. Point the bus at a dummy cam_sim
structure so that any CCBs will complete immediately with a
CAM_DEV_NOT_THERE status, and ensure that any xpt_schedule() calls
on the bus's devices will immediately call the peripheral's
periph_start() routine. Also repeat the async messages because
devices that were part of the way through being probed may appear
after the original AC_LOST_DEVICE was sent, and would otherwise
never go away.

These changes make it possible to deregister a bus and free the SIM
at most stages during bus probing without the usual crashes in
camisr(). In particular, plugging in a umass device and then
unplugging it as soon as the first probe messages appeared would
almost always result in a crash. Now the device just goes away with
a few CAM errors and all references to the CAM bus, target and
device are dropped correctly.

Make the exploring of all luns supported by an HBA more of a
tunable (until we get REPORT LUNS in place).

If we're probing luns, and each probe succeeds, we keep going past
lun 7 if we're a SCSI3 or better device (until we fail to probe).

If we're probing luns, and a probe fails, we only keep going if
we're quirked *for* it (CAM_QUIRK_HILUNS), and if we're not quirked
*against* it (CAM_QUIRK_NOHILUNS), or we're a SCSI3 or better device
and the tunable (kern.cam.cam_srch_hi) is set non-zero.

Reviewed by: nate@rootlabs.org, gibbs@scsiguy.com, ken@kdm.com, scottl@samsco.org
MFC after: 1 week

- Providing fine-grained malloc statistic by replacing M_DEVBUF with
module-specific malloc types. These should help us to pinpoint the
possible memory leakage in the future.
- Implementing xpt_alloc_ccb_nowait() and replacing all malloc/free based
CCB memory management with xpt_alloc_ccb[_nowait]/xpt_free_ccb. Hopefully
this would be helpful if someday we move the CCB allocator to use UMA
instead of malloc().

Encouraged by: jeffr, rwatson
Reviewed by: gibbs, scottl
Approved by: re (scottl)

Fixing a memory leak in xpt_release_device(), which can be quickly
(depends on how many memory you have) observed through "tar -tvf /dev/sa0."

Without this patch, RELENG_5 and HEAD panics with something like:

kmem_malloc(4096): kmem_map too small: 42258432 total allocated

RELENG_4 doesn't panic but spews following errors:

camq_init: - cannot malloc array!

Reviewed by: gibbs, scottl
Approved by: re (scottl)
MFC after: 3 days

Do not initialize path variable with useless value just before
xpt_create_path overwrites it anyway.

Noticed by: Coverity Prevent analysis tool

avoid use after free

Submitted by: gibbs
Reviewed by: mdodd
Noticed by: Coverity Prevent analysis tool

Restore queue depth settings across tag disable events. The system often
disables tag queuing temporarily in order to allow controllers a window
to safely perform transfer negotiation with non-compliant devices. Before
this change, CAM would restore the queue depth to the controller specified
maximum or device quirk level rather than any depth determined by reactions
to QUEUE FULL/BUSY events or an explicit user setting.

During device probe, initialize the flags field for XPT_SCAN_BUS.
The uninitialized value often confused CAM into not bothering to
issue an AC_FOUND_DEVICE async event for new devices. The reason
this bug wasn't reported earlier is that CAM manually announces
devices after the initial system bus scans.

MFC: 3 days

Check the correct periph pointer after search for it.

Submitted by: Coverity Prevent analysis tool

Provide locking for the ccb_bioq. This allows xpt_done() to be called without
Giant held. In camisr(), move the ccb_bioq elements to a temporary local list
and then process the elements off of that list. This enables the list to be
processed by only taking the ccb_bioq_lock once and only for a very short
time.

ccb_bioq_lock is a leaf mutex, so it's fine to call xpt_done() with other
locks held. This is just a very minor step in the work to lock CAM, but
it allows us to avoid some messy locking/unlock dances in certain drivers.

Add quirk for easyRAID ER5 alias areca ARC-6010.
It reports itself as SCSI-3 but doesnt like getting probed on high luns
because it hangs hard after finding itself again on lun 32...

Suggested by: Kenneth Merry

This is a somewhat imperfect means to try and bring FreeBSD forward in
its ability to automatically scan and attach luns for modern storage
which has luns in the 0..1000 range, not 0..7.

The correct thing would be to do REPORT LUNS for devices whose LUN0
version shows a version >= SCSI3, but lacking that we should be able
to search higher than LUN 7 if we're >= SCSI3 with no ill effects.

This change keeps all of the QUIRK_HILUNS quirks, obeys the QUIRK_NOLUNS,
and introduces a QUIRK_NOHILUNS which will keep searches above LUN 7
happening for devices that report >= SCSI3 compliance. I doubt the latter
will be needed, but you never know.

This allowed me to randomly scan and attach > 500 disks at a time in
a situation where quirking for QUIRK_HILUNS wasn't practical (the
vendor id and product id changes of the virtualization changes
constantly).

Reviewed by: ken@freebsd.org, scottl@freebsd.org, gibbs@freebsd.org
MFC after: 2 weeks

Start each of the license/copyright comments with /*-

Correct another location where inquiry length as calculated from the
inquiry additional length field is off by one.

MFC: 3 days

Correct an off-by-one in the calculation of the inquiry buffer length
to request from devices during the "long inquiry" portion of our probe.
This same bug was fixed in the 4.x stream a few years ago, but the fix
was never propogated to -current.

This fix is slightly different than in -stable:

o Use offsetof() instead of a hard coded constant so as the make
the code more self-explainatory.

o Round odd long inquiry lengths up so as to avoid tickling ignore
wide residue bugs in broken parallel SCSI devices running with a
wide transfer negotiation.

MFC: 3 days

Remove the camnet swi and CAM_PERIPH_NET. It has never been used, and given
that netowrk-over-scsi never really took off, there is little chance that
it will ever be needed.

Do a pass over all modules in the kernel and make them return EOPNOTSUPP
for unknown events.

A number of modules return EINVAL in this instance, and I have left
those alone for now and instead taught MOD_QUIESCE to accept this
as "didn't do anything".

Revert rev 1.140, it was a bit pre-mature.

Add a quirk for ciss to stop trying to read the serial number off
of logical volumes.

Do the dreaded s/dev_t/struct cdev */
Bump __FreeBSD_version accordingly.