cam: Let cam_periph_unmapmem() return an error

As of commit b059686a71c8, cam_periph_unmapmem() can legitimately fail
if the copyout() operation fails. However, this failure was never
signaled to upper layers. In practice it is unlikely to occur
since cap_periph_mapmem() would most likely fail in such
circumstances anyway, but an error is nonetheless possible.

However, some code reading revealed a few paths where the return value
of cam_periph_mapmem() is not checked, and this is definitely a bug.
Add error checking there and let cam_periph_unmapmem() return errors
from copyout().

Reviewed by: dab, mav
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D43201

CAM: Replace random sbuf_printf() with cheaper cat/putc.

cam: Remove left-over sys/cdefs.h in sys/cam

These weren't removed when $FreeBSD$ was removed. They aren't needed and
now are a style(9) nonconformity.

Sponsored by: Netflix

cam: Add nvme error devctl publishing

Start reporting nvme errors from devices, like we report ata and scsi
errors.

Sponsored by: Netflix
Reviewed by: mav, jhb
Differential Revision: https://reviews.freebsd.org/D41086

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

cam: Log more error codes

Log CAM_DEV_NOT_THERE status CCBs because they get dropped if a drive
disappears and these requests timeout or are cancelled. It's useful to
know the outstanding commands for failure analysis. Log
CAM_NVME_STATUS_ERROR status CCBs to bring in NVMe errors (this will be
more important in future commits that expand the information logged).

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D41168

cam: Migrate to modern uintXX_t from u_intXX_t

As per https://lists.freebsd.org/archives/freebsd-scsi/2023-July/000257.html
move to the modern uintXX_t.

MFC After: 3 days
Sponsored by: Netflix

cam_periph: Comment about why we need to reset cbfcnp

Just spent a few minutes puzzling out why we do this. Add a comment to
remind my future self (and other intersted folk) why we do the reset
here when we'd set it a few lines above.

Sponsored by: Netflix

cam_periph: Fix a comment

Add a couple of words so that this sentence makes sense.

Sponsored by: Netflix

cam: Add CAM_NVME_STATUS_ERROR error code

Add CAM_NVME_STATUS_ERROR error code. Flag all NVME commands that
completed with an error status as CAM_NVME_STATUS_ERROR (a new value)
instaead of CAM_REQ_CMP_ERR. This indicates to the upper layers of CAM
that the 'cpl' field for nvmeio CCBs is valid and can be examined for
error recovery, if desired.

No functional change. nda will still see these as errors, call
ndaerror() to get the error recovery action, etc. cam_periph_error will
select the same case as before (even w/o the change, though the change
makes it explicit).

Sponsored by: Netflix
Reviewed by: chuck, mav, jhb
Differential Revision: https://reviews.freebsd.org/D41085

cam: Be explict about CAM_SMP_STATUS_ERROR

This is normally caught by default:, but no harm in making it explicit
that we'll retry valid periphs.

Sponsored by: Netflix
Reviewed by: mav
Differential Revision: https://reviews.freebsd.org/D41083

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

cam: Provide compatibility for CAMGETPASSTHRU for periph drivers

The CAM version bump 0x19 -> 0x1a changed the CAMGETPASSTHRU definition,
so applications using the old ioctl are broken. However, that version
change did not affect anything relating to the ioctl implementation for
periphs.

Fixes: 8f9be1eed11c ("cam(4): Improve XPT_DEV_MATCH")
PR: 264709
Tested by: andreas.mahling@googlemail.com
Reviewed by: imp
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D36389

Delay GEOM disk_create() until CAM periph probe completes.

Before this patch CAM periph drivers called both disk_alloc() and
disk_create() same time on periph creation. But then prevented disks
from opening until the periph probe completion with cam_periph_hold().
As result, especially if disk misbehaves during the probe, GEOM event
thread, triggered to taste the disk, got blocked on open attempt,
potentially for a long time, unable to process other events.

This patch moves disk_create() call from periph creation to the end of
the probe. To allow disk_create() calls from non-sleepable CAM contexts
some of its duties requiring memory allocations are moved either back
to disk_alloc() or forward to g_disk_create(), so now disk_alloc() and
disk_add_alias() are the only disk methods that require sleeping. If
disk fails during the probe disk_create() may just be skipped, going
directly to disk_destroy(). Other method calls during that time are
just ignored. Since GEOM may now see the disks after CAM bus scan is
already completed, introduce per-periph boot hold functions. Enclosure
driver already had such mechanism, so just generalize it.

Reviewed by: imp
MFC after: 1 month
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D35784

cam_periph: Return ENXIO when peripheral is invalidated

When the peripheral is invalidated, no further I/O is possible. Signal
this up the stack with ENXIO now that upper layers of the stack
differentiate sometimes. In order for there to be further I/O, and new
open is required for any block device that a future periph might
instantiate for devices at this location that might return or otherwise
become available. The I/O scheduler flushes its I/O with the ENXIO error
for pending I/O that didn't make it to the device, so this makes the two
paths match.

MFC After: 3 days
Sponsored by: Netflix
Reviewed by: chs, mav
Differential Revision: https://reviews.freebsd.org/D35093

CAM: Keep periph_links when restoring CCB in camperiphdone().

While recovery command executed, some other commands from the periph
may complete, that may affect periph_links of this CCB. So restoring
original CCB we must keep current periph_links as more up to date.

I've found this triggering assertions with debug kernel and suspect
some memory corruptions otherwise when spun down disk receives two
or sometimes more concurrent requests.

MFC after: 1 week
Sponsored by: iXsystems, Inc.

cam: Fix wiring fence post error

If the last matching device entry partially matched in camperiphunit,
but then hit a continue case, we'd mistakenly think we had a match on
that entry. This lead to a number of problems downstream (usually a
belief that we had a duplicate wiring hint because unit = 0 is the
default). Fix this by using a for loop that does the assignment before
the loop termination test.

Sponsored by: Netflix
Reviewed by: jhb
Differential Revision: https://reviews.freebsd.org/D33873

cam: Initialize wired to false

As part of converting the code to a while loop, the unconditional
initialization of wired to false was lost.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D33163

cam_periph: wired is really a bool, update it to a bool.

Sponsored by: Netflix
Reviewed by: scottl
Differential Revision: https://reviews.freebsd.org/D32823

cam_periph: Add ability to wire units to a serial number

For scsi, ata and nvme, at least, we read a serial number from the
device (if the device supports it, some scsi drives do not) and record
it during the *_xpt probe device state machine before it posts the
AC_FOUND_DEVICE async event. For mmc, no serial number is ever
retrieved, so it's always NULL. Add the ability to match this serial
number during device wiring.

This mechanism is competely optional, and often times using a label
and/or some other attribute of the device is easier. However, other
times wiring a unit to a serial number simplifies management as most
monitoring tools require the *daX device and having it stable from boot
to boot helps with data continuity. It can be especially helpful for
nvme where no other means exists to reliably tie a ndaX device to an
underlying nvme drive and namespace.

A similar mechanism exists in Linux to mange device unit numbers with
udev.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D32683

cam_periph: fix bug in camperiphunitnext logic

If we assigned just a lun as a wired unit (something that camperiphunit
will accept), we failed to properly skip over that unit when computing a
next unit number. Add lun so the code matches the comments that we have
to skip all the same criteria that camperiphunit uses to select wired
units for a driver.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D32682

cam_periph: switch from negative logic to positive logic

When scanning the resources that are wired for this driver, skip any
that whose number doesn't match newunit. They aren't relevant. Switch to
positive logic to break out of the loop (and thus go to the next unit)
if we find either a target resource or an at resource. This makes the
code easier to read and modify.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D32681

cam_periph: Remove vestigial "scbus" comparison

The code in camperiphunit rejects "scbus" as an 'at' location that would
allow any other wiring to use that unit number. Yet in
camperiphunitnext, if we have a no target and the 'at' location of
'scbus' it would be excluded on the basis that it's a wiring
cadidate. This is improper and appears to be a hold-over of the
pre-hints / pre-newbus config system, so remove it.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D32680

cam_periph: style change

wrap a long line at 80 columns

Sponsored by: Netflix
Reviewed by: chs
Differential Revision: https://reviews.freebsd.org/D32679

cam(4): preserve alloc_flags when copying CCBs

Before UMA CCBs, all CCBs were of the same size, and could
be trivially copied using bcopy(9). Now we have to preserve
alloc_flags, otherwise we might end up attempting to free
stack-allocated CCB to UMA; we also need to take CCB size
into account.

This fixes kernel panic which would occur when trying to access
a stopped (as in, SCSI START STOP, also "ctladm stop") SCSI device.

Reported By: Gary Jennejohn <gljennjohn@gmail.com>
Tested By: Gary Jennejohn <gljennjohn@gmail.com>
Reviewed By: imp
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D31054

cam: make sure to clear CCBs allocated on the stack

This is required for small CCBs support, where we need to track
whether the CCB was allocated from an UMA zone or not. There are
no (intended) functional changes with the current source.

Reviewed By: imp
Sponsored by: NetApp, Inc.
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D29484

cam: Permit non-pollable sims.

Some CAM sim drivers do not support polling (notably iscsi(4)).
Rather than using a no-op poll routine that always times out requests,
permit a SIM to set a NULL poll callback. cam_periph_runccb() will
fail polled requests non-pollable sims immediately as if they had
timed out.

Reviewed by: scottl, mav (earlier version)
Reviewed by: imp
MFC after: 2 weeks
Sponsored by: Chelsio
Differential Revision: https://reviews.freebsd.org/D28453

Remove alignment requirements for KVA buffer mapping.

After r368124 vmapbuf() should happily map misaligned maxphys-sized buffers
thanks to extra page added to pbuf_zone.

Make MAXPHYS tunable. Bump MAXPHYS to 1M.

Replace MAXPHYS by runtime variable maxphys. It is initialized from
MAXPHYS by default, but can be also adjusted with the tunable kern.maxphys.

Make b_pages[] array in struct buf flexible. Size b_pages[] for buffer
cache buffers exactly to atop(maxbcachebuf) (currently it is sized to
atop(MAXPHYS)), and b_pages[] for pbufs is sized to atop(maxphys) + 1.
The +1 for pbufs allow several pbuf consumers, among them vmapbuf(),
to use unaligned buffers still sized to maxphys, esp. when such
buffers come from userspace (*). Overall, we save significant amount
of otherwise wasted memory in b_pages[] for buffer cache buffers,
while bumping MAXPHYS to desired high value.

Eliminate all direct uses of the MAXPHYS constant in kernel and driver
sources, except a place which initialize maxphys. Some random (and
arguably weird) uses of MAXPHYS, e.g. in linuxolator, are converted
straight. Some drivers, which use MAXPHYS to size embeded structures,
get private MAXPHYS-like constant; their convertion is out of scope
for this work.

Changes to cam/, dev/ahci, dev/ata, dev/mpr, dev/mpt, dev/mvs,
dev/siis, where either submitted by, or based on changes by mav.

Suggested by: mav (*)
Reviewed by: imp, mav, imp, mckusick, scottl (intermediate versions)
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D27225

vmapbuf: don't smuggle address or length in buf

Instead, add arguments to vmapbuf. Since this argument is
always a pointer use a type of void * and cast to vm_offset_t in
vmapbuf. (In CheriBSD we've altered vm_fault_quick_hold_pages to
take a pointer and check its bounds.)

In no other situtation does b_data contain a user pointer and vmapbuf
replaces b_data with the actual mapping.

Suggested by: jhb
Reviewed by: imp, jhb
Obtained from: CheriBSD
MFC after: 1 week
Sponsored by: DARPA
Differential Revision: https://reviews.freebsd.org/D26784

cam: clean up empty lines in .c and .h files

Use devctl.h instead of bus.h to reduce newbus pollution.

There's no need for these parts of the kernel to know about newbus,
so narrow what is included to devctl.h for device_notify_*.

Suggested by: kib@

Remove stale comment

There's no useracc here, and even if there was it shouldn't be here. vmapbuf is
sufficient and as the comment says, useracc is racy.

Add comment about how the deferred callback for AC_FOUND_DEVICE we
generate for a race where a device goes away, we start to tear down
the periph state for the device, and then the device suddently
reappears. The key that makes it work is removal of periph from the
drv list before calling the deferred callback.

Hat tip to: mav@

Reword a comment to describe what's actually going on. We can call invalidate
several times potentially. We just don't do anything on the second and
subsequent calls.

The KASSERT is too strict: revert r357897

It's valid for a periph to be removed with outstanding transactions on the
device. In CAM, multiple periphs attach to a single device. There's no interlock
to prevent one of these going away while other periphs have outstanding CCBs and
it's not an error either. Remove this overly agressive KASSERT to prevent
false-positive panics when devices depart.

Add a KASSERT that there's no outstanding CCBs when we call camperiphfree. We
know that if there are any outstanding CCBs, then when they dereference the path
that's freed at the bottom of camperiphfree there will be some flavor of
panic. This moves that eventual panic to a traceback of when we free the last
reference on the device, which is earlier but may not be early enough.

Make pass(4) handle misaligned buffers of MAXPHYS size.

Since we are already using malloc()+copyin()/copyout() for smaller data
blocks, and since new asynchronous API does it always, I see no reason
to keep this ugly artificial size/alignment limitation in old API.

Tape applications suffer enough from the MAXPHYS limitations by itself,
and additional alignment requirement, often halving effectively usable
block size, does not help.

It would be good to use unmapped I/O here instead, but it require some
HBA drivers polishing first to support non-BIO unmapped buffers.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Do not retry long ready waits if previous gave nothing.

I have some disks reporting "Logical unit is in process of becoming ready"
for about half an hour before finally reporting failure. During that time
CAM waits for the readiness during ~2 minutes for each request, that makes
system boot take very long time.

This change reduces wait times for the following requests to ~1 second if
previously long wait for that device has timed out.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Fix assumptions of only one device per SES slot.

It is typical to have one, but no longer true for multi-actuator HDDs
with separate LUN for each actuator.

MFC after: 4 days
Sponsored by: iXsystems, Inc.

Unify SCSI_STATUS_BUSY retry handling with other cases.

- Do not retry if periph was invalidated.
- Do not decrement retry_count if already zero.
- Report action_string when applicable.

MFC after: 2 weeks

Do not map small IOCTL buffers to KVA, but copy.

CAM IOCTL interfaces traditionally mapped user-space data buffers to KVA.
It was nice originally, but now it takes too much to handle respective
TLB shootdowns, while small kernel memory allocations up to 64KB backed
by UMA and accompanied by copyin()/copyout() can be much cheaper.

For large buffers mapping still may have sense, and unmapped I/O would
be even better, but the last unfortunately is more tricky, since unmapped
I/O API is too specific to struct bio now.

MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Allocate pager bufs from UMA instead of 80-ish mutex protected linked list.

o In vm_pager_bufferinit() create pbuf_zone and start accounting on how many
pbufs are we going to have set.
In various subsystems that are going to utilize pbufs create private zones
via call to pbuf_zsecond_create(). The latter calls uma_zsecond_create(),
and sets a limit on created zone. After startup preallocate pbufs according
to requirements of all pbuf zones.

Subsystems that used to have a private limit with old allocator now have
private pbuf zones: md(4), fusefs, NFS client, smbfs, VFS cluster, FFS,
swap, vnode pager.

The following subsystems use shared pbuf zone: cam(4), nvme(4), physio(9),
aio(4). They should have their private limits, but changing that is out of
scope of this commit.

o Fetch tunable value of kern.nswbuf from init_param2() and while here move
NSWBUF_MIN to opt_param.h and eliminate opt_swap.h, that was holding only
this option.
Default values aren't touched by this commit, but they probably should be
reviewed wrt to modern hardware.

This change removes a tight bottleneck from sendfile(2) operation, that
uses pbufs in vnode pager. Other pagers also would benefit from faster
allocation.

Together with: gallatin
Tested by: pho

Add comments explaining what hold/unhold do

They act as a simple one-deep semaphore to keep open/close/probe from
running at the same time to avoid races that creates.

Stop further SCSI recovery attempts after one has failed.

We've got a set of probably damaged hard disks, reporting 0x04,0x02
("Logical unit not ready, initializing command required") in response
to READ CAPACITY(16), where attempts to use START STOP UNIT for recovery
results in 0x44,0x00 ("Internal target failure") after ~1 second delay.
As result of all recovery retries, device open attempt took ~3 seconds
before finally reporting to GEOM that device is opened, but has no media.
If the open was for writing and since it hasn't formally failed, following
close triggered GEOM retaste, opening device few more times with respective
delays.

This change reduces whole time of this cycle from ~12 seconds to ~3 by
giving up on recovery after the first failure.

Reviewed by: ken
MFC after: 2 weeks
Sponsored by: iXsystems, Inc.

Add and fix comments for cam_periph_runccb()

Sponsored by: Netflix

Create a sysctl kern.cam.{,a,n}da.X.invalidate

kern.cam.{,a,n}da.X.invalidate=1 forces *daX to detach by calling
cam_periph_invalidate on the underlying periph. This is for testing
purposes only. Include only with options CAM_TEST_FAILURE and rename
the former [AN]DA_TEST_FAILURE, and fix nda to compile with it set.
We're using it at work to harden geom and the buffer cache to be
resilient in the face of drive failure. Today, it far too often
results in a panic. While much work was done on SIM initiated removal
for the USB thumnb drive removal work, little has been done for periph
initiated removal. This simulates what *daerror() does for some errors
nicely: we get the same panics with it that we do with failing drives.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D14581

Report the number of remaining retries when we have an error that
we're retrying.

Return a C errno for cam_periph_acquire().

There's no compelling reason to return a cam_status type for this
function and doing so only creates confusion with normal C
coding practices. It's technically an API change, but the periph API
isn't widely used. No efffective change to operation.

Reviewed by: imp, mav, ken
Sponsored by: Netflix
Differential Revision: D14063

When we crash, we'll stop the scheduler before we call the
shutdown_post_sync event. For adashutdown, this causes problems
because we need to poll for completion of the commands, but we're not
yet officially dumping yet, so the code from r326964 assumed we could
use the interrupt-driven commands rather than the polled ones. This
lead to a hang. Prevent this by also checking to see if the scheduler
is stopped to do the polling.

Reported by: markj@
Sponsored by: Netflix
Differential Review: https://reviews.freebsd.org/D13845

Don't hold the periph lock when calling into cam_periph_runccb()
from the ada and da dump routines. This avoids difficult locking
problems from needing to be handled. While it might seem like this
would leave the periphs unprotected during dump, they were aleady
at risk of unexpected removal due to the dump functions not
keeping refcount state across the many calls that come in during
a dump. This is an exercise for future work.

Obtained from: Netflix

When doing a dump, the scheduler is normally not running, so this
changed worked to capture dumps for me. However, the test for
SCHEDULER_STOPPED() isn't right. We can also call the dump routine
from ddb, in which case the scheduler is still running. This leads to
an assertion panic that we're sleeping when we shouldn't. Instead, use
the proper test for dumping or not. This brings us in line with other
places that do special things while we're doing polled I/O like this.

Noticed by: pho@
Differential Revision: https://reviews.freebsd.org/D13531

Define xpt_path_inq.

This provides a nice wrarpper around the XPT_PATH_INQ ccb creation and
calling.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D13387

Make cam_periph_runccb be safe to call when we can only do polling.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D13388

Remove unused 4th argument to match the standard error routines.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D13386

Add NVME as a known device type for devstat processing.

Also, reduce the amount of cut and pasted code a little since only two
args are different in the devstat_end_transaction calls.

Sponsored by: Netflix

sys/cam: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

This adds CAM pass(4) support for NVMe IO's. Applications indicate
the IO type (Admin or NVM) using XPT op-codes XPT_NVME_ADMIN or
XPT_NVME_IO.

Submitted by: Chuck Tuffli <chuck@tuffli.net>
Differential Revision: https://reviews.freebsd.org/D10247

An MMC/SD/SDIO stack using CAM

Implement the MMC/SD/SDIO protocol within a CAM framework. CAM's
flexible queueing will make it easier to write non-storage drivers
than the legacy stack. SDIO drivers from both the kernel and as
userland daemons are possible, though much of that functionality will
come later.

Some of the CAM integration isn't complete (there are sleeps in the
device probe state machine, for example), but those minor issues can
be improved in-tree more easily than out of tree and shouldn't gate
progress on other fronts. Appologies to reviews if specific items
have been overlooked.

Submitted by: Ilya Bakulin
Reviewed by: emaste, imp, mav, adrian, ian
Differential Review: https://reviews.freebsd.org/D4761

merge with first commit, various compile hacks.

Fix a panic in camperiphfree().

If a peripheral driver (e.g. da, sa, cd) is added or removed from the
peripheral driver list while an unrelated peripheral driver instance (e.g.
da0, sa5, cd2) is going away and is inside camperiphfree(), we could
dereference an invalid pointer.

When peripheral drivers are added or removed (see periphdriver_register()
and periphdriver_unregister()), the peripheral driver array is resized
and existing entries are moved.

Although we hold the topology lock while we traverse the peripheral driver
list, we retain a pointer to the location of the peripheral driver pointer
and then drop the topology lock. So we are still vulnerable to the list
getting moved around while the lock is dropped.

To solve the problem, cache a copy of the peripheral driver pointer. If
its storage location in the list changes while we have the lock dropped, it
won't have any effect.

This doesn't solve the issue that peripheral drivers ("da", "cd", as opposed
to individual instances like "da0", "cd0") are not generally part of a
reference counting scheme to guard against deregistering them while there
are instances active. The caller (generally the person unloading a module)
has to be aware of active drivers and not unload something that is in use.

sys/cam/cam_periph.c:
In camperiphfree(), cache a pointer to the peripheral driver
instance to avoid holding a pointer to an invalid memory location
in the event that the peripheral driver list changes while we have
the topology lock dropped.

PR: kern/219701
Submitted by: avg
MFC after: 3 days
Sponsored by: Spectra Logic

Fix an unsafe malloc usage with sbufs.

Reported by: ken
Sponsored by: Netflix

Add infrastructure to the ATA and SCSI transports that supports
using a driver-supplied sbuf for printing device discovery
announcements. This helps ensure that messages to the console
will be properly serialized (through sbuf_putbuf) and not be
truncated and interleaved with other messages. The
infrastructure mirrors the existing xpt_announce_periph()
entry point and is opt-in for now. No content or formatting
changes are visible to the operator other than the new coherency.

While here, eliminate the stack usage of the temporary
announcement buffer in some of the drivers. It's moved to the
softc for now, but future work will eliminate it entirely by
making the code flow more linear. Future work will also address
locking so that the sbufs can be dynamically sized.

The scsi_da, scs_cd, scsi_ses, and ata_da drivers are converted
at this point, other drivers can be converted at a later date.
A tunable+sysctl, kern.cam.announce_nosbuf, exists for testing
purposes but will be removed later.

TODO:
Eliminate all of the code duplication and temporary buffers. The
old printf-based methods will be retired, and xpt_announce_periph()
will just be a wrapper that uses a dynamically sized sbuf. This
requires that the register and deregister paths be made malloc-safe,
which they aren't currently.

Sponsored by: Netflix

Add mechanism to unload CAM periph drivers.

For now it allows to unload CTL kernel module if there are no target-capable
SIMs in CAM. As next step full teardown of CAM targets can be implemented.

Improve CAM_CDB_POINTER support.

MFC after: 2 weeks

Add BUF_TRACKING and FULL_BUF_TRACKING buffer debugging

Upstream the BUF_TRACKING and FULL_BUF_TRACKING buffer debugging code.
This can be handy in tracking down what code touched hung bios and bufs
last. The full history is especially useful, but adds enough bloat that
it shouldn't be enabled in release builds.

Function names (or arbitrary string constants) are tracked in a
fixed-size ring in bufs. Bios gain a pointer to the upper buf for
tracking. SCSI CCBs gain a pointer to the upper bio for tracking.

Reviewed by: markj
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8366

cam_periph_ccbwait could return while ccb in progress

In cam_periph_runccb, cam_periph_ccbwait was using the value of the ccb
pinfo.index and status fields to determine whether the ccb was done,
but these fields are updated without a contending lock and could glitch
into states that would be erroneously interpreted as done. Instead,
have cam_periph_ccbwait look for the explicit result of the function
cam_periph_done.

Submitted by: Ryan Libby <rlibby@gmail.com>
Reviewed by: mav
MFC after: 3 weeks
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D8020

Don't report to devd statuses that CAM doesn't consider errors.

Some statuses, such as "ATA pass through information available", are part
part of absolutely normal operation and do not worth reporting.

MFC after: 2 weeks

Don't repeat the the word 'the'

(one manual change to fix grammar)

Confirmed With: db
Approved by: secteam (not really, but this is a comment typo fix)

Fix a memory leak in the devctl notify code.

Submitted by: markj
MFC after: asap

sys/cam: spelling fixes in comments.

No functional change.

tag_action is not used at all in ata. It's set to 1 for ordered
transactions, but that value isn't used. It's bogusly used to report
in devstat, due to a cut and paste error from SCSI. Mark it as unused
in cam_fill_ataio. Reclaim the memory as a new ata_flags. In addition,
tag_id and init_id are completely unused, so reclaim those as 'unused'
now too. These were needlessly copied when ata was split from scsi.

This allows us, in the future, to create structures that can
communicate AUXILIARY regsiter to the SIMs, which cannot be done now.

Differential Revision: https://reviews.freebsd.org/D5598

Add a devctl/devd notification conduit for CAM errors that happen at the
periph level. When a relevant error is reported to the periph, some
amplifying information is gathered, and the error and information are fed
to devctl with the attributes / keys system=CAM, subsystem=periph. The
'type' key will be either 'error' or 'timeout', and based on this, various
other keys are also populated.

The purpose of this is to provide a concise mechanism for error reporting
that is less noisy than the system console but higher in resolution and
fidelity than simple sysctl counters. We will be using it at Netflix to
populate a structured log and database to track errors and error trends
across our world-wide population of drives.

Submitted by: imp, scottl
Approved by: kenm
MFC after: 3 days
Sponsored by: Netflix
Differential Revision: D5943

Make pass, sg and targ drivers respect HBA's maxio.

Previous limitation of 64K (DFLTPHYS) is quite annoying.

Refactor unmapped buffer address handling.
- Use pointer assignment rather than a combination of pointers and
flags to switch buffers between unmapped and mapped. This eliminates
multiple flags and generally simplifies the logic.
- Eliminate b_saveaddr since it is only used with pager bufs which have
their b_data re-initialized on each allocation.
- Gather up some convenience routines in the buffer cache for
manipulating buf space and buf malloc space.
- Add an inline, buf_mapped(), to standardize checks around unmapped
buffers.

In collaboration with: mlaier
Reviewed by: kib
Tested by: pho (many small revisions ago)
Sponsored by: EMC / Isilon Storage Division

Make periphdriver_register() take XPT lock when modifying the periph_drivers
array.

This fixes a panic that sometimes occured when kldloading ctl.ko.

Reviewed by: mav@
MFC after: 1 month
Sponsored by: The FreeBSD Foundation

Fix I/O size calculation for pass(4) driver requests and add latency
tracking.

It is important to subtract the residual from the requested
transfer size to see how much data was actually transferred. With
tape drives in particular, it is common to request more data than is
returned.

Also, add I/O latency tracking for CAM requests issued by
cam_periph_runccb().

If the caller supplies a struct devstat, and the I/O is a SCSI or
ATA I/O, we will track the elapsed time to provide I/O latency
statistics for the request.

sys/cam/scsi/cam_periph.c:
In cam_periph_runccb(), subtract the residual when reporting I/O
totals to devstat(9) for SCSI and ATA passthrough requests.

In cam_periph_runccb(), grab the I/O start time and supply
the start time to devstat_end_transaction() so that it can
calculate the elapsed I/O time.

Sponsored by: Spectra Logic
MFC after: 1 week

Retry indefinitely on SCSI BUSY status from VMware disks and CDs.

VMware returns BUSY status when storage has transient connectivity issues.
It is often better to wait and let VM admin fix the problem then crash.

Discussed with: ken
MFC after: 1 week

Don't denounce peripherals on system shutdown. Together with r267321,
we're now back to the pre-r228483 level of default verbosity. This in
turn again typically allows for reading information that userland might
have printed on the screen before initiating a halt, but still permits
to debug potential device shutdown problems on system shutdown via
CAM_DEBUG etc.

Reviewed by: mav
MFC after: 3 days
Sponsored by: Bally Wulff Games & Entertainment GmbH

Do not retry on CAM_FUNC_NOTAVAIL error, but return immediately.

MFC after: 2 weeks

Take additional reference on SCSI probe periph to cover its freeze count.

Otherwise periph may be invalidated and freed before single-stepping freeze
is dropped, causing use after free panic.

Fix memory and references leak due to unfreed path.

Coverity CID: 1054773

Merge CAM locking changes from the projects/camlock branch to radically
reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.

Replace big per-SIM locks with bunch of smaller ones:
- per-LUN locks to protect device and peripheral drivers state;
- per-target locks to protect list of LUNs on target;
- per-bus locks to protect reference counting;
- per-send queue locks to protect queue of CCBs to be sent;
- per-done queue locks to protect queue of completed CCBs;
- remaining per-SIM locks now protect only HBA driver internals.

While holding LUN lock it is allowed (while not recommended for performance
reasons) to take SIM lock. The opposite acquisition order is forbidden.
All the other locks are leaf locks, that can be taken anywhere, but should
not be cascaded. Many functions, such as: xpt_action(), xpt_done(),
xpt_async(), xpt_create_path(), etc. are no longer require (but allow) SIM
lock to be held.

To keep compatibility and solve cases where SIM lock can't be dropped, all
xpt_async() calls in addition to xpt_done() calls are queued to completion
threads for async processing in clean environment without SIM lock held.

Instead of single CAM SWI thread, used for commands completion processing
before, use multiple (depending on number of CPUs) threads. Load balanced
between them using "hash" of the device B:T:L address.

HBA drivers that can drop SIM lock during completion processing and have
sufficient number of completion threads to efficiently scale to multiple
CPUs can use new function xpt_done_direct() to avoid extra context switch.
Make ahci(4) driver to use this mechanism depending on hardware setup.

Sponsored by: iXsystems, Inc.
MFC after: 2 months

Unify periph invalidation and destruction reporting.
Print message containing device model and serial number on invalidation.

Requested by: glebius
MFC after: 1 week

Add debug trace points for freeze/release device queue.

Remove droping topology mutex after iterating 100 periphs in CAMGETPASSTHRU.
That is not so slow and so often operation to handle unneeded otherwise
xsoftc.xpt_generation and respective locking complications.

Make some improvements to r253322 to really rescan target, not a bus.
Add there and in two more places checks for NULL on xpt_alloc_ccb_nowait().

Improve handling of 0x3F/0x0E "Reported LUNs data has changed" and 0x25/0x00
"Logical unit not supported" errors. First initiates specific target rescan,
second -- destroys specific LUN. That allows to automatically detect changes
in list of device LUNs. This mechanism doesn't work when target is completely
idle, but probably that is all what can be done without active polling.

Reviewed by: ken
Sponsored by: iXsystems, Inc.

Simplify the checking of flags for cam_periph_mapmem(). This gets rid of
a lot of code redundancy and grossness at very minor expense.

Reviewed by: smh
Obtained from: Netflix
MFC after: 3 days

MFprojects/camlock r249541:
Give periph validity flag own periph reference. That slightly simplifies
the release logic and covers hypothetical case if lock is dropped inside
the periph_oninval() method.

MFprojects/camlock r248890, r248897, r248898, r248900, r248903, r248905,
r248917, r248918, r248978, r249001, r249014, r249030:

Remove multilevel freezing mechanism, implemented to handle specifics of
the ATA/SATA error recovery, when post-reset recovery commands should be
allocated when queues are already full of payload requests. Instead of
removing frozen CCBs with specified range of priorities from the queue
to provide free openings, use simple hack, allowing explicit CCBs over-
allocation for requests with priority higher (numerically lower) then
CAM_PRIORITY_OOB threshold.

Simplify CCB allocation logic by removing SIM-level allocation queue.
After that SIM-level queue manages only CCBs execution, while allocation
logic is localized within each single device.

Suggested by: gibbs

MFprojects/camlock:
r249017:
Some cosmetic things:
- Unify device to target insertion inside xpt_alloc_device() instead of
duplicating it three times.
- Remove extra checks for empty lists of devices and targets on release
since zero refcount check also implies it.
- Reformat code to reduce indentation.

r249103:
- Add lock assertions to every point where reference counters are modified.
- When reference counters are reaching zero, add assertions that there are
no children items left.
- Add a bit more locking to the xptpdperiphtraverse().

Fix panic in the error path caused by recursive acquisition of XPT topology
lock.

Reviewed by: ken

Do not remap usermode pages into KVA for physio.

Sponsored by: The FreeBSD Foundation
Tested by: pho

Assert that a ccb passed to cam_periph_mapmem() for XPT_SCSI_IO and
XPT_ATA_IO holds virtual buffer address.

Sponsored by: The FreeBSD Foundation
Tested by: pho

Fix a panic during CAM EDT traversal.

The problem was a race condition between the EDT traversal used by
things like 'camcontrol devlist', and CAM peripheral driver
removal.

The EDT traversal code holds the CAM topology lock, and wants
to show devices that have been invalidated. It acquires a
reference to the peripheral to make sure the peripheral it is
examining doesn't go away.

However, because the peripheral removal code in camperiphfree()
drops the CAM topology lock to call the peripheral's destructor
routine, we can run into a situation where the EDT traversal
increments the peripheral reference count after free process is
already in progress. At that point, the reference count is
ignored, because it was 0 when we started the process.

Fix this race by setting a flag, CAM_PERIPH_FREE, that I previously
added and checked in xptperiphtraverse() and xptpdperiphtravsere(),
but failed to use. If the EDT traversal code sees that flag,
it will know that the peripheral free process has already started,
and that it should not access that peripheral.

Also, fix an inconsistency in the locking between
xptpdperiphtraverse() and xptperiphtraverse(). They now both
hold the CAM topology lock while calling the peripheral traversal
function.

cam_xpt.c: Change xptperiphtraverse() to hold the CAM topology
lock across calls to the traversal function.

Take out the comment in xptpdperiphtraverse() that
referenced the locking inconsistency.

cam_periph.c: Set the CAM_PERIPH_FREE flag when we are in the
process of freeing a peripheral driver.

Sponsored by: Spectra Logic Corporation
MFC after: 1 week

Add explicit check for not set time inside cam_periph_freeze_after_event().

System time is set later on boot process then initial bus scan by CAM.
Until that moment microtime() is equal to microuptime(), and if system
boots quickly, the value can be close to zero. That causes settle time
waiting even for buses that don't use reset during probe.

On my test system this reduces boot time by 1 second if USB enabled, or
by 4 seconds if USB disabled. CAM waited for ctl2cam0 bus "settle".

Implement media change notification for DA and CD removable media devices.
It includes three parts:
1) Modifications to CAM to detect media media changes and report them to
disk(9) layer. For modern SATA (and potentially UAS) devices it utilizes
Asynchronous Notification mechanism to receive events from hardware.
Active polling with TEST UNIT READY commands with 3 seconds period is used
for incapable hardware. After that both CD and DA drivers work the same way,
detecting two conditions: "NOT READY: Medium not present" after medium was
detected previously, and "UNIT ATTENTION: Not ready to ready change, medium
may have changed". First one reported to disk(9) as media removal, second
as media insert/change. To reliably receive second event new
AC_UNIT_ATTENTION async added to make UAs broadcasted to all periphs by
generic error handling code in cam_periph_error().
2) Modifications to GEOM core to handle media remove and change events.
Media removal handled by spoiling all consumers attached to the provider.
Media change event also schedules provider retaste after spoiling to probe
new media. New flag G_CF_ORPHAN was added to consumers to reflect that
consumer is in process of destruction. It allows retaste to create new
geom instance of the same class, while previous one is still dying.
3) Modifications to some GEOM classes: DEV -- to report media change
events to devd; VFS -- to handle spoiling same as orphan to prevent
accessing replaced media. PART class already handles spoiling alike to
orphan.

Reviewed by: silence on geom@ and scsi@
Tested by: avg
Sponsored by: iXsystems, Inc. / PC-BSD
MFC after: 2 months

Fix a typo in a panic() call.

PR: kern/169497
Submitted by: Steven Hartland <steven.hartland@multiplay.co.uk>
MFC after: 3 days

Add scsi_extract_sense_ccb() -- wrapper around scsi_extract_sense_len().
It allows to remove number of duplicate checks from several places.

Don't print SCSI Queue Full and CAM_REQUEUE_REQ statuses as errors if they
were handled and retried. They are part of normal operation for SCSI TCQ.

MFC after: 3 days

Make cam_periph_hold() behavior consistent: drop taken reference and
return ENXIO if periph was invalidated while we were waiting for it.

MFC after: 1 week

Fix several reference counting and object lifetime issues between
the pass(4) and enc(4) drivers and devfs.

The pass(4) driver uses the destroy_dev_sched() routine to
schedule its device node for destruction in a separate thread
context. It does this because the passcleanup() routine can get
called indirectly from the passclose() routine, and that would
cause a deadlock if the close routine tried to destroy its own
device node.

In any case, once a particular passthrough driver number, e.g.
pass3, is destroyed, CAM considers that unit number (3 in this
case) available for reuse.

The problem is that devfs may not be done cleaning up the previous
instance of pass3, and will panic if isn't done cleaning up the
previous instance.

The solution is to get a callback from devfs when the device node
is removed, and make sure we hold a reference to the peripheral
until that happens.

Testing exposed some other cases where we have reference counting
issues, and those were also fixed in the pass(4) driver.

cam_periph.c: In camperiphfree(), reorder some of the operations.

The peripheral destructor needs to be called before
the peripheral is removed from the peripheral is
removed from the list. This is because once we
remove the peripheral from the list, and drop the
topology lock, the peripheral number may be reused.
But if the destructor hasn't been called yet, there
may still be resources hanging around (like devfs
nodes) that haven't been fully cleaned up.

cam_xpt.c: Add an argument to xpt_remove_periph() to indicate
whether the topology lock is already held.

scsi_enc.c: Acquire an extra reference to the peripheral during
registration, and release it once we get a callback
from devfs indicating that the device node is gone.

Call destroy_dev_sched_cb() in enc_oninvalidate()
instead of calling destroy_dev() in the cleanup
routine.

scsi_pass.c: Add reference counting to handle peripheral and
devfs object lifetime issues.

Add a reference to the peripheral and the devfs
node in the peripheral registration.

Don't attempt to add a physical path alias if the
peripheral has been marked invalid.

Release the devfs reference once the initial
physical path alias taskqueue run has completed.

Schedule devfs node destruction in the
passoninvalidate(), and release our peripheral
reference in a new routine, passdevgonecb() once
the devfs node is gone. This allows the peripheral
to fully go away, and the peripheral destructor,
passcleanup(), will get called.

MFC after: 3 days
Sponsored by: Spectra Logic

One more major cam_periph_error() rewrite to improve error handling and
reporting. It includes:
- removing of error messages controlled by bootverbose, replacing them
with more universal and informative debugging on CAM_DEBUG_INFO level,
that is now built into the kernel by default;
- more close following to the arguments submitted by caller, such as
SF_PRINT_ALWAYS, SF_QUIET_IR and SF_NO_PRINT; consumer knows better which
errors are usual/expected at this point and which are really informative;
- adding two new flags SF_NO_RECOVERY and SF_NO_RETRY to allow caller
specify how much assistance it needs at this point; previously consumers
controlled that by not calling cam_periph_error() at all, but that made
behavior inconsistent and debugging complicated;
- tuning debug messages and taken actions order to make debugging output
more readable and cause-effect relationships visible;
- making camperiphdone() (common device recovery completion handler) to
also use cam_periph_error() in most cases, instead of own dumb code;
- removing manual sense fetching code from cam_periph_error(); I was told
by number of people that it is SIM obligation to fetch sense data, so this
code is useless and only significantly complicates recovery logic;
- making ada, da and pass driver to use cam_periph_error() with new limited
recovery options to handle error recovery and debugging in common way;
as one of results, CAM_REQUEUE_REQ and other retrying statuses are now
working fine with pass driver, that caused many problems before.
- reverting r186891 by raj@ to avoid burning few seconds in tight DELAY()
loops on device probe, while device simply loads media; I think that problem
may already be fixed in other way, and even if it is not, solution must be
different.

Sponsored by: iXsystems, Inc.
MFC after: 2 weeks

Add CAM_DEBUG_INFO debug messages for periph created/invalidated/destroyed
and for asyncs sent.

Fix a bug introduced in r230000. We were eliminating all LUNs on a target
in response to CAM_DEV_NOT_THERE, instead of just the LUN in question.

This will now just eliminate the specified LUN in response to
CAM_DEV_NOT_THERE.

Reported by: Richard Todd <rmtodd@servalan.servalan.com>
MFC after: 3 days

Fix a race condition in CAM peripheral free handling, locking
in the CAM XPT bus traversal code, and a number of other periph level
issues.

cam_periph.h,
cam_periph.c: Modify cam_periph_acquire() to test the CAM_PERIPH_INVALID
flag prior to allowing a reference count to be gained
on a peripheral. Callers of this function will receive
CAM_REQ_CMP_ERR status in the situation of attempting to
reference an invalidated periph. This guarantees that
a peripheral scheduled for a deferred free will not
be accessed during its wait for destruction.

Panic during attempts to drop a reference count on
a peripheral that already has a zero reference count.

In cam_periph_list(), use a local sbuf with SBUF_FIXEDLEN
set so that mallocs do not occur while the xpt topology
lock is held, regardless of the allocation policy of the
passed in sbuf.

Add a new routine, cam_periph_release_locked_buses(),
that can be called when the caller already holds
the CAM topology lock.

Add some extra debugging for duplicate peripheral
allocations in cam_periph_alloc().

Treat CAM_DEV_NOT_THERE much the same as a selection
timeout (AC_LOST_DEVICE is emitted), but forgo retries.

cam_xpt.c: Revamp the way the EDT traversal code does locking
and reference counting. This was broken, since it
assumed that the EDT would not change during
traversal, but that assumption is no longer valid.

So, to prevent devices from going away while we
traverse the EDT, make sure we properly lock
everything and hold references on devices that
we are using.

The two peripheral driver traversal routines should
be examined. xptpdperiphtraverse() holds the
topology lock for the entire time it runs.
xptperiphtraverse() is now locked properly, but
only holds the topology lock while it is traversing
the list, and not while the traversal function is
running.

The bus locking code in xptbustraverse() should
also be revisited at a later time, since it is
complex and should probably be simplified.

scsi_da.c: Pay attention to the return value from cam_periph_acquire().

Return 0 always from daclose() even if the disk is now gone.

Add some rudimentary error injection support.

scsi_sg.c: Fix reference counting in the sg(4) driver.

The sg driver was calling cam_periph_release() on close,
but never called cam_periph_acquire() (which increments
the reference count) on open.

The periph code correctly complained that the sg(4)
driver was trying to decrement the refcount when it
was already 0.

Sponsored by: Spectra Logic
MFC after: 2 weeks

Mark MALLOC_DEFINEs static that have no corresponding MALLOC_DECLAREs.

This means that their use is restricted to a single C file.

Add descriptor sense support to CAM, and honor sense residuals properly in
CAM.

Desriptor sense is a new sense data format that originated in SPC-3. Among
other things, it allows for an 8-byte info field, which is necessary to
pass back block numbers larger than 4 bytes.

This change adds a number of new functions to scsi_all.c (and therefore
libcam) that abstract out most access to sense data.

This includes a bump of CAM_VERSION, because the CCB ABI has changed.
Userland programs that use the CAM pass(4) driver will need to be
recompiled.

camcontrol.c: Change uses of scsi_extract_sense() to use
scsi_extract_sense_len().

Use scsi_get_sks() instead of accessing sense key specific
data directly.

scsi_modes: Update the control mode page to the latest version (SPC-4).

scsi_cmds.c,
scsi_target.c: Change references to struct scsi_sense_data to struct
scsi_sense_data_fixed. This should be changed to allow the
user to specify fixed or descriptor sense, and then use
scsi_set_sense_data() to build the sense data.

ps3cdrom.c: Use scsi_set_sense_data() instead of setting sense data
manually.

cam_periph.c: Use scsi_extract_sense_len() instead of using
scsi_extract_sense() or accessing sense data directly.

cam_ccb.h: Bump the CAM_VERSION from 0x15 to 0x16. The change of
struct scsi_sense_data from 32 to 252 bytes changes the
size of struct ccb_scsiio, but not the size of union ccb.
So the version must be bumped to prevent structure
mis-matches.

scsi_all.h: Lots of updated SCSI sense data and other structures.

Add function prototypes for the new sense data functions.

Take out the inline implementation of scsi_extract_sense().
It is now too large to put in a header file.

Add macros to calculate whether fields are present and
filled in fixed and descriptor sense data

scsi_all.c: In scsi_op_desc(), allow the user to pass in NULL inquiry
data, and we'll assume a direct access device in that case.

Changed the SCSI RESERVED sense key name and description
to COMPLETED, as it is now defined in the spec.

Change the error recovery action for a number of read errors
to prevent lots of retries when the drive has said that the
block isn't accessible. This speeds up reconstruction of
the block by any RAID software running on top of the drive
(e.g. ZFS).

In scsi_sense_desc(), allow for invalid sense key numbers.
This allows calling this routine without checking the input
values first.

Change scsi_error_action() to use scsi_extract_sense_len(),
and handle things when invalid asc/ascq values are
encountered.

Add a new routine, scsi_desc_iterate(), that will call the
supplied function for every descriptor in descriptor format
sense data.

Add scsi_set_sense_data(), and scsi_set_sense_data_va(),
which build descriptor and fixed format sense data. They
currently default to fixed format sense data.

Add a number of scsi_get_*() functions, which get different
types of sense data fields from either fixed or descriptor
format sense data, if the data is present.

Add a number of scsi_*_sbuf() functions, which print
formatted versions of various sense data fields. These
functions work for either fixed or descriptor sense.

Add a number of scsi_sense_*_sbuf() functions, which have a
standard calling interface and print the indicated field.
These functions take descriptors only.

Add scsi_sense_desc_sbuf(), which will print a formatted
version of the given sense descriptor.

Pull out a majority of the scsi_sense_sbuf() function and
put it into scsi_sense_only_sbuf(). This allows callers
that don't use struct ccb_scsiio to easily utilize the
printing routines. Revamp that function to handle
descriptor sense and use the new sense fetching and
printing routines.

Move scsi_extract_sense() into scsi_all.c, and implement it
in terms of the new function, scsi_extract_sense_len().
The _len() version takes a length (which should be the
sense length - residual) and can indicate which fields are
present and valid in the sense data.

Add a couple of new scsi_get_*() routines to get the sense
key, asc, and ascq only.

mly.c: Rename struct scsi_sense_data to struct
scsi_sense_data_fixed.

sbp_targ.c: Use the new sense fetching routines to get sense data
instead of accessing it directly.

sbp.c: Change the firewire/SCSI sense data transformation code to
use struct scsi_sense_data_fixed instead of struct
scsi_sense_data. This should be changed later to use
scsi_set_sense_data().

ciss.c: Calculate the sense residual properly. Use
scsi_get_sense_key() to fetch the sense key.

mps_sas.c,
mpt_cam.c: Set the sense residual properly.

iir.c: Use scsi_set_sense_data() instead of building sense data by
hand.

iscsi_subr.c: Use scsi_extract_sense_len() instead of grabbing sense data
directly.

umass.c: Use scsi_set_sense_data() to build sense data.

Grab the sense key using scsi_get_sense_key().

Calculate the sense residual properly.

isp_freebsd.h: Use scsi_get_*() routines to grab asc, ascq, and sense key
values.

Calculate and set the sense residual.

MFC after: 3 days
Sponsored by: Spectra Logic Corporation

In some cases failed SATA disks may report their presence, but don't
respond to any commands. I've found that because of multiple command
retries, each of which cause 30s timeout, bus reset and another retry or
requeue for many commands, it may take ages to eventually drop the
failed device. The odd thing is that those retries continue even after
XPT considered device as dead and invalidated it.

This patch makes cam_periph_error() to block any command retries after
periph was marked as invalid. With that patch all activity completes in
1-2 minutes, just after several timeouts, required to consider device
death. This should make ZFS, gmirror, graid, etc. operation more robust.

Reviewed by: mjacob@ on scsi@

Approved by: re (kib)

Lay groundwork in CAM for recording and reporting physical path and
other device attributes stored in the CAM Existing Device Table (EDT).
This includes some infrastructure requried by the enclosure services
driver to export physical path information.

Make the CAM device advanced info interface accept store requests.

sys/cam/scsi/scsi_all.c:
sys/cam/scsi/scsi_all.h:
- Replace scsi_get_sas_addr() with a scsi_get_devid() which takes
a callback that decides whether to accept a particular descriptor.
Provide callbacks for NAA IEEE Registered addresses and for SAS
addresses, replacing the old function. This is needed because
the old function doesn't work for an enclosure address for a SAS
device, which is not flagged as a SAS address, but is NAA IEEE
Registered. It may be worthwhile merging this interface with the
devid match interface.
- Add a few more defines for some device ID fields.

sbin/camcontrol/camcontrol.c:
- Update for the CCB_DEV_ADVINFO interface change.

cam/cam_xpt_internal.h:
- Add the new fields for the physical path string to the CAM EDT.
cam/cam_ccb.h:
- Rename CCB_GDEV_ADVINFO to simply CCB_DEV_ADVINFO, and the ccb
structure to ccb_dev_advinfo.
- Add a flag that changes this CCB's action to store, rather than
the default, retrieve.
- Add a new buffer type, CDAI_TYPE_PHYS_PATH, for the new CAM EDT
physpath field.
- Remove the never-implemented transport & proto flags.
cam/cam_xpt.c:
cam/cam_xpt.h:
- Add xpt_getattr(), which provides a wrapper for fetching a device's
attribute using the GEOM strings as key. This method currently
supports "GEOM::ident" and "GEOM::physpath".

Submitted by: will
Reviewed by : gibbs

Extend the XPT_DEV_MATCH api to allow a device search by device ID.
As far as the API is concerned, device ID is a binary blob to be
interpreted by the transport layer. The SCSI implementation assumes
it is an array of VPD device ID descriptors.

sys/cam/cam_ccb.h:
Create a new structure, device_id_match_pattern, and
update the XPT_DEV_MATCH datastructures and flags so
that this pattern type can be used.

sys/cam/cam_xpt.c:
- A single pattern matching on both inquiry data and device
ID is invalid. Report any violators.
- Pass device ID match requests through to the new routine
scsi_devid_match(). The direct call of a SCSI routine is
a layering violation, but no worse than the one a few
lines up that checks inquiry data. Defer cleaning this
up until our future, larger, rototilling of CAM.
- Zero out cam_ed and cam_et nodes on allocation. Prior to
this change, device_id_len and device_id were not inialized,
preventing proper detection of the presence of this
information.

sys/cam/scsi/scsi_all.c:
sys/cam/scsi/scsi_all.h:
Add the scsi_match_devid() routine.

Add a helper function for extracting peripherial driver names

sys/cam/cam_periph.c:
sys/cam/cam_periph.h:
Add the cam_periph_list() method which fills an sbuf
with a comma delimited list of the peripheral instances
associated with a given CAM path.

Add a helper functions for SCSI commands used by the SES driver.

sys/cam/scsi/scsi_all.c:
sys/cam/scsi/scsi_all.h:
Add structure definitions and csio filling functions for
the receive diagnostic results and send diagnostic commands.

Misc CAM XPT cleanups.

sys/cam/cam_xpt.c:
Broadcast AC_FOUND_DEVICE and AC_PATH_REGISTERED
events at the time async event handlers are attached
even when registering just for events on a partitular
SIM. Previously, you had to register for these
events on all SIMs in the system in order to get
the initial broadcast even though subsequent device
and path arrivals would be delivered.

sys/cam/cam_xpt.c:
Remove SIM mutex held asserts from path accessors.
CAM paths are reference counted and it is this
reference count, not the sim mutex, that garantees
they are stable.

Sponsored by: Spectra Logic Corporation

Remove always false "< 0" check for unsgined int variable. This check is
also duplicate, as the value was already checked for 0 before decrementing.

Reported by: rpaulo

Don't automatically send a START UNIT to sequential access devices-
this might cause them to load the tape unintentionally.

Reviewed by: gibbs
MFC after: 1 month

Remove unneeded includes of <sys/linker_set.h>. Other headers that use
it internally contain nested includes.

Reviewed by: bde

Fix a few issues related to the XPT_GDEV_ADVINFO CCB.

camcontrol.c: In buildbusdevlist(), don't attempt to get call
getdevid() for an unconfigured device, even when the
verbose flag is set. The cam_open_btl() call will almost
certainly fail.

Probe for the buffer size when issuing the XPT_GDEV_ADVINFO
CCB. Probing for the buffer size first helps us avoid
allocating the maximum buffer size when it really may not
be necessary. This also helps avoid errors from
cam_periph_mapmem() if we attempt to map more than MAXPHYS.

cam_periph.c: In cam_periph_mapmem(), if the XPT_GDEV_ADVINFO CCB
shows a bufsiz of 0, we don't have anything to map,
so just return.

Also, set the maximum mapping size to MAXPHYS
instead of DFLTPHYS for XPT_GDEV_ADVINFO CCBs,
since they don't actually go down to the hardware.

scsi_pass.c: Don't bother mapping the buffer in XPT_GDEV_ADVINFO
CCBs if bufsiz is 0.

Add Serial Management Protocol (SMP) passthrough support to CAM.

This includes support in the kernel, camcontrol(8), libcam and the mps(4)
driver for SMP passthrough.

The CAM SCSI probe code has been modified to fetch Inquiry VPD page 0x00
to determine supported pages, and will now fetch page 0x83 in addition to
page 0x80 if supported.

Add two new CAM CCBs, XPT_SMP_IO, and XPT_GDEV_ADVINFO. The SMP CCB is
intended for SMP requests and responses. The ADVINFO is currently used to
fetch cached VPD page 0x83 data from the transport layer, but is intended
to be extensible to fetch other types of device-specific data.

SMP-only devices are not currently represented in the CAM topology, and so
the current semantics are that the SIM will route SMP CCBs to either the
addressed device, if it contains an SMP target, or its parent, if it
contains an SMP target. (This is noted in cam_ccb.h, since it will change
later once we have the ability to have SMP-only devices in CAM's topology.)

smp_all.c,
smp_all.h: New helper routines for SMP. This includes
SMP request building routines, response parsing
routines, error decoding routines, and structure
definitions for a number of SMP commands.

libcam/Makefile: Add smp_all.c to libcam, so that SMP functionality
is available to userland applications.

camcontrol.8,
camcontrol.c: Add smp passthrough support to camcontrol. Several
new subcommands are now available:

'smpcmd' functions much like 'cmd', except that it
allows the user to send generic SMP commands.

'smprg' sends the SMP report general command, and
displays the decoded output. It will automatically
fetch extended output if it is available.

'smppc' sends the SMP phy control command, with any
number of potential options. Among other things,
this allows the user to reset a phy on a SAS
expander, or disable a phy on an expander.

'smpmaninfo' sends the SMP report manufacturer
information and displays the decoded output.

'smpphylist' displays a list of phys on an
expander, and the CAM devices attached to those
phys, if any.

cam.h,
cam.c: Add a status value for SMP errors
(CAM_SMP_STATUS_ERROR).

Add a missing description for CAM_SCSI_IT_NEXUS_LOST.

Add support for SMP commands to cam_error_string().

cam_ccb.h: Rename the CAM_DIR_RESV flag to CAM_DIR_BOTH. SMP
commands are by nature bi-directional, and we may
need to support bi-directional SCSI commands later.

Add the XPT_SMP_IO CCB. Since SMP commands are
bi-directional, there are pointers for both the
request and response.

Add a fill routine for SMP CCBs.

Add the XPT_GDEV_ADVINFO CCB. This is currently
used to fetch cached page 0x83 data from the
transport later, but is extensible to fetch many
other types of data.

cam_periph.c: Add support in cam_periph_mapmem() for XPT_SMP_IO
and XPT_GDEV_ADVINFO CCBs.

cam_xpt.c: Add support for executing XPT_SMP_IO CCBs.

cam_xpt_internal.h: Add fields for VPD pages 0x00 and 0x83 in struct
cam_ed.

scsi_all.c: Add scsi_get_sas_addr(), a function that parses
VPD page 0x83 data and pulls out a SAS address.

scsi_all.h: Add VPD page 0x00 and 0x83 structures, and a
prototype for scsi_get_sas_addr().

scsi_pass.c: Add support for mapping buffers in XPT_SMP_IO and
XPT_GDEV_ADVINFO CCBs.

scsi_xpt.c: In the SCSI probe code, first ask the device for
VPD page 0x00. If any VPD pages are supported,
that page is required to be implemented. Based on
the response, we may probe for the serial number
(page 0x80) or device id (page 0x83).

Add support for the XPT_GDEV_ADVINFO CCB.

sys/conf/files: Add smp_all.c.

mps.c: Add support for passing in a uio in mps_map_command(),
so we can map a S/G list at once.

Add support for SMP passthrough commands in
mps_data_cb(). SMP is a special case, because the
first buffer in the S/G list is outbound and the
second buffer is inbound.

Add support for warning the user if the busdma code
comes back with more buffers than will work for the
command. This will, for example, help the user
determine why an SMP command failed if busdma comes
back with three buffers.

mps_pci.c: Add sys/uio.h.

mps_sas.c: Add the SAS address and the parent handle to the
list of fields we pull from device page 0 and cache
in struct mpssas_target. These are needed for SMP
passthrough.

Add support for the XPT_SMP_IO CCB. For now, this
CCB is routed to the addressed device if it supports
SMP, or to its parent if it does not and the parent
does. This is necessary because CAM does not
currently support SMP-only nodes in the topology.

Make SMP passthrough support conditional on
__FreeBSD_version >= 900026. This will make it
easier to MFC this change to the driver without
MFCing the CAM changes as well.

mps_user.c: Un-staticize mpi_init_sge() so we can use it for
the SMP passthrough code.

mpsvar.h: Add a uio and iovecs into struct mps_command for
SMP passthrough commands.

Add a cm_max_segs field to struct mps_command so
that we can warn the user if busdma comes back with
too many segments.

Clear the cm_reply when a command gets freed. If
it is not cleared, reply frames will eventually get
freed into the pool multiple times and corrupt the
pool. (This fix is from scottl.)

Add a prototype for mpi_init_sge().

sys/param.h: Bump __FreeBSD_version to 900026 for the for the
inclusion of the XPT_GDEV_ADVINFO and XPT_SMP_IO
CAM CCBs.

When requesting sense data for SIM not doing it automatically (such as
ATAPI or USB), request only as much data as requested by consumer.
On the way back -- report how much sense data we have actually received.

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Protect periph drivers list and rearrange things to minimize the chance of
stepping oneself during probing.

Don't blindly decrement a periph probe count.

Reviewed by: scsi@
Obtained from: Alexander Motin, Atillio Rao, Others
MFC after: 1 month

MFC r203108:
Large set of CAM improvements:
- Unify bus reset/probe sequence. Whenever bus attached at boot or later,
CAM will automatically reset and scan it. It allows to remove duplicate
code from many drivers.
- Any bus, attached before CAM completed it's boot-time initialization,
will equally join to the process, delaying boot if needed.
- New kern.cam.boot_delay loader tunable should help controllers that
are still unable to register their buses in time (such as slow USB/
PCCard/ CardBus devices), by adding one more event to wait on boot.
- To allow synchronization between different CAM levels, concept of
requests priorities was extended. Priorities now split between several
"run levels". Device can be freezed at specified level, allowing higher
priority requests to pass. For example, no payload requests allowed,
until PMP driver enable port. ATA XPT negotiate transfer parameters,
periph driver configure caching and so on.
- Frozen requests are no more counted by request allocation scheduler.
It fixes deadlocks, when frozen low priority payload requests occupying
slots, required by higher levels to manage theit execution.
- Two last changes were holding proper ATA reinitialization and error
recovery implementation. Now it is done: SATA controllers and Port
Multipliers now implement automatic hot-plug and should correctly
recover from timeouts and bus resets.
- Improve SCSI error recovery for devices on buses without automatic sense
reporting, such as ATAPI or USB. For example, it allows CAM to wait, while
CD drive loads disk, instead of immediately return error status.
- Decapitalize diagnostic messages and make them more readable and sensible.
- Teach PMP driver to limit maximum speed on fan-out ports.
- Make boot wait for PMP scan completes, and make rescan more reliable.
- Fix pass driver, to return CCB to user level in case of error.
- Increase number of retries in cd driver, as device may return several UAs.

MFp4: Large set of CAM inprovements.

- Unify bus reset/probe sequence. Whenever bus attached at boot or later,
CAM will automatically reset and scan it. It allows to remove duplicate
code from many drivers.
- Any bus, attached before CAM completed it's boot-time initialization,
will equally join to the process, delaying boot if needed.
- New kern.cam.boot_delay loader tunable should help controllers that
are still unable to register their buses in time (such as slow USB/
PCCard/ CardBus devices), by adding one more event to wait on boot.
- To allow synchronization between different CAM levels, concept of
requests priorities was extended. Priorities now split between several
"run levels". Device can be freezed at specified level, allowing higher
priority requests to pass. For example, no payload requests allowed,
until PMP driver enable port. ATA XPT negotiate transfer parameters,
periph driver configure caching and so on.
- Frozen requests are no more counted by request allocation scheduler.
It fixes deadlocks, when frozen low priority payload requests occupying
slots, required by higher levels to manage theit execution.
- Two last changes were holding proper ATA reinitialization and error
recovery implementation. Now it is done: SATA controllers and Port
Multipliers now implement automatic hot-plug and should correctly
recover from timeouts and bus resets.
- Improve SCSI error recovery for devices on buses without automatic sense
reporting, such as ATAPI or USB. For example, it allows CAM to wait, while
CD drive loads disk, instead of immediately return error status.
- Decapitalize diagnostic messages and make them more readable and sensible.
- Teach PMP driver to limit maximum speed on fan-out ports.
- Make boot wait for PMP scan completes, and make rescan more reliable.
- Fix pass driver, to return CCB to user level in case of error.
- Increase number of retries in cd driver, as device may return several UAs.

MFC r199279, r199280, r199281:
- Fix several device freeze counting bugs.
- Remove code that years ago was closing race between request submission
to SIM and device/SIM freeze. That race become impossible after moving from
spl to mutex locking, while this workaround causes some unexpected effects.

MFC r198947:
Implement device stats accounting for ATA commands.

MFC r198849:
Improve reporting ATA Status error details.

MFC r198394:
Make "Retrying Command" to be printed before actual retrying.
It should make debug/error log a bit more readable.

MFC r198382, r198385:
Replace most of priority numbers with defines. No logical changes.

MFC r197421:
If on sense request device returns no sence, give up and return,
or we may loop forever.

MFp4:
Some more missed parts from previous commits.

MFp4:
Fix several device freeze counting bugs.

MFp4:
Implement device stats accounting for ATA commands.

MFp4:
Improve reporting ATA Status error details.

Make "Retrying Command" to be printed before actual retrying.
It should make debug/error log a bit more readable.

Replace most of priority numbers with defines. No logical changes.

MFp4:
If on sense request device returns no sence, give up and return,
or we may loop forever.

Separate the parallel scsi knowledge out of the core of the XPT, and
modularize it so that new transports can be created.

Add a transport for SATA

Add a periph+protocol layer for ATA

Add a driver for AHCI-compliant hardware.

Add a maxio field to CAM so that drivers can advertise their max
I/O capability. Modify various drivers so that they are insulated
from the value of MAXPHYS.

The new ATA/SATA code supports AHCI-compliant hardware, and will override
the classic ATA driver if it is loaded as a module at boot time or compiled
into the kernel. The stack now support NCQ (tagged queueing) for increased
performance on modern SATA drives. It also supports port multipliers.

ATA drives are accessed via 'ada' device nodes. ATAPI drives are
accessed via 'cd' device nodes. They can all be enumerated and manipulated
via camcontrol, just like SCSI drives. SCSI commands are not translated to
their ATA equivalents; ATA native commands are used throughout the entire
stack, including camcontrol. See the camcontrol manpage for further
details. Testing this code may require that you update your fstab, and
possibly modify your BIOS to enable AHCI functionality, if available.

This code is very experimental at the moment. The userland ABI/API has
changed, so applications will need to be recompiled. It may change
further in the near future. The 'ada' device name may also change as
more infrastructure is completed in this project. The goal is to
eventually put all CAM busses and devices until newbus, allowing for
interesting topology and management options.

Few functional changes will be seen with existing SCSI/SAS/FC drivers,
though the userland ABI has still changed. In the future, transports
specific modules for SAS and FC may appear in order to better support
the topologies and capabilities of these technologies.

The modularization of CAM and the addition of the ATA/SATA modules is
meant to break CAM out of the mold of being specific to SCSI, letting it
grow to be a framework for arbitrary transports and protocols. It also
allows drivers to be written to support discrete hardware without
jeopardizing the stability of non-related hardware. While only an AHCI
driver is provided now, a Silicon Image driver is also in the works.
Drivers for ICH1-4, ICH5-6, PIIX, classic IDE, and any other hardware
is possible and encouraged. Help with new transports is also encouraged.

Submitted by: scottl, mav
Approved by: re

Change cam_periph_ioctl() to take 'cmd' and a u_long instead of an
int. All of its callers pass in cmd as a u_long, so this has
always been a dangerous type demotion. It was spooted by clang/llvm
trying to do a type promotion and sign extension within
cam_periph_ioctl.

Submitted by: rdivacky

Add missing free(9) in error case.

Found with: Coverity Prevent(tm)
CID: 4224

cam_periph_alloc: fix "invalid periph name" error condition

Found with: Coverity Prevent(tm)
CID: 130

Now that mtx_sleep/msleep can accept Giant as the interlock, simplify the
CAM locking code slightly to no longer special case sleeping when a sim
uses Giant for its lock.

Tested by: trasz

Guard against NULL pointer dereference.

Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation
Found with: Coverity Prevent(tm)
CID: 130

Periph driver fixes, second try.

Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation

Move mtx_assert lower, when we can be sure that the pointer
to the mutex is valid.

Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation

Revert r186186 for now; it breaks stuff.

Approved by: rwatson (mentor)

Fix locking in periph drivers - don't try to unlock periph
that was already deallocated.

Reviewed by: scottl
Approved by: rwatson (mentor)
Sponsored by: FreeBSD Foundation

- Import the HEAD csup code which is the basis for the cvsmode work.

Fix typo.

Approved by: cognet (mentor)

Eliminate the use of M_TEMP.

Retire the spl() markers. Add in some minor missed locking as a result.

Inline cam_periph_lock|unlock to make debugging easier. Use more
CAM_SIM_LOCK() more uniformly.

Drop the topology lock before calling the periph oninvalidate and dtor
vectors.

Remove Giant from CAM. Drivers (SIMs) now register a mutex that CAM will
use to synchornize and protect all data objects that are used for that
SIM. Drivers that are not yet MPSAFE register Giant and operate as
usual. RIght now, no drivers are MPSAFE, though a few will be changed
in the coming week as this work settles down.

The driver API has changed, so all CAM drivers will need to be recompiled.
The userland API has not changed, so tools like camcontrol do not need to
be recompiled.

Add an xpt_print function to reduce most of the xpt_print_path/printf
pairs. Convert the core code to use it.

Reviewed by: scsi (various folks weighing in with good advice)
MFC after: 2 weeks

Put a bit of hysteresis into both BUSY SCSI status returns
and CAM_RESRC_UNAVAIL returns. Delay a tunable amount for
either between retries.

This came up because the MPT IOC was returning "IOC out of
resources" for some user and this caused a CAM_RESRC_UNAVAIL
return. Putting a bit of delay between retries helped them
out.

There was some discussion that an async event should be used
to clear CAM_RESRC_UNAVAIL. That's probably a better notion
eventually.

Reviewed by: scsi@freebsd.org (ade, scott)
MFC after: 1 week

Make physical buffers in cam_periph_mapmem owned by the kernel in case we
return to user space w/o waiting for I/O to complete.

I tried to get several folks who know this code better than me to review it
with no luck. I *do* know that w/o this code, using the SCSI target driver
panics in userret (if it doesn't panic in knote first).

There should be no need to retry when the CCB status code is
CAM_LUN_INVALID or CAM_TID_INVALID. Retries were being triggered
here when a umass device was unplugged, and while the retries
themselves are probably harmless, they complicated finding the real
SIM removal problems.

- Providing fine-grained malloc statistic by replacing M_DEVBUF with
module-specific malloc types. These should help us to pinpoint the
possible memory leakage in the future.
- Implementing xpt_alloc_ccb_nowait() and replacing all malloc/free based
CCB memory management with xpt_alloc_ccb[_nowait]/xpt_free_ccb. Hopefully
this would be helpful if someday we move the CCB allocator to use UMA
instead of malloc().

Encouraged by: jeffr, rwatson
Reviewed by: gibbs, scottl
Approved by: re (scottl)

Protect against trying to free a non-existant peripheral.

Submitted by: Coverity Prevent analysis tool
MFC After: 3 days

Start each of the license/copyright comments with /*-

Correct a typo in a comment: alloation->allocation

Reported by: Jens Schweikhardt <schweikh@schweikhardt.net>

Don't be so chatty when performing manual sense. This should make ATAPICAM
devices a lot more quiet.

In camperiphdone(), make sure we check for fatal errors and bail out
instead of retrying them blindly.

This should fix some of the problems people have been having with cdrom
drives taking a long time to probe. This should also eliminate the need
for the initial TUR in cdsize().

cam_periph.c: Don't keep retrying if the error we get back is a fatal
error. This should help us detect the transition from
"Logical unit not ready, cause not reportable" to "Medium
not present" in the "TUR many" handler. (The TUR many
handler gets triggered for Logical unit not ready, cause
not reportable errors.)

scsi_cd.c: Remove the initial test unit ready in cdsize(). Hopefully
it isn't necessary after the above change.

Submitted by: gibbs (mostly)
Tested by: peter
MFC After: 2 weeks

There is no need to muck about with the B_PHYS flag here. We never even
get close to DEV_STRATEGY() which is the only place it is relevant.

(camperiphdone): When the cam_periph layer performs sense recovery,
completion of recovery is indicated by positioning the CAM_AUTOSNS_VALID
bit in the status field of the CCB, not in the flags field.
This fixes an endless loop of sense recovery actions.

Reviewed by: ken

In case vmapbuf() fails, release all of the held resources.

Submitted by: tegge

Use __FBSDID().

Add /* FALLTHROUGH */

Found by: FlexeLint

Add /* FALLTHROUGH */

Found by: FlexeLint

Sufficient access checks are performed by vmapbuf() that calling useracc()
is pointless. Remove the calls to useracc().

Run a revision of the devstat interface:

Kernel:

Change statistics to use the *uptime() timescale (ie: relative to
boottime) rather than the UTC aligned timescale. This makes the
device statistics code oblivious to clock steps.

Change timestamps to bintime format, they are cheaper.

Remove the "busy_count", and replace it with two counter fields:
"start_count" and "end_count", which are updated in the down and
up paths respectively. This removes the locking constraint on
devstat.

Add a timestamp argument to devstat_start_transaction(), this will
normally be a timestamp set by the *_bio() function in bp->bio_t0.
Use this field to calculate duration of I/O operations.

Add two timestamp arguments to devstat_end_transaction(), one is
the current time, a NULL pointer means "take timestamp yourself",
the other is the timestamp of when this transaction started (see
above).

Change calculation of busy_time to operate on "the salami principle":
Only when we are idle, which we can determine by the start+end
counts being identical, do we update the "busy_from" field in the
down path. In the up path we accumulate the timeslice in busy_time
and update busy_from.

Change the byte_* and num_* fields into two arrays: bytes[] and
operations[].

Userland:

Change the misleading "busy_time" name to be called "snap_time" and
make the time long double since that is what most users need anyway,
fill it using clock_gettime(CLOCK_MONOTONIC) to put it on the same
timescale as the kernel fields.

Change devstat_compute_etime() to operate on struct bintime.

Remove the version 2 legacy interface: the change to bintime makes
compatibility far too expensive.

Fix a bug in systat's "vm" page where boot relative busy times would
be bogus.

Bump __FreeBSD_version to 500107

Review & Collaboration by: ken

Back out M_* changes, per decision of the TRB.

Approved by: trb

Remove M_TRYWAIT/M_WAITOK/M_WAIT. Callers should use 0.
Merge M_NOWAIT/M_DONTWAIT into a single flag M_NOWAIT.

Close the remaining user address mapping races for physical
I/O, CAM, and AIO. Still TODO: streamline useracc() checks.

Reviewed by: alc, tegge
MFC after: 7 days

Panic message strings do not need a trailing \n.

Reviewed by: ken

Make SCSI_DELAY setable at boot time and runtime via the
kern.cam.scsi_delay tunable/sysctl.

Reviewed by: mdodd, njl

Add some \n's to printf()s

Fix unit wiring. Also, change the variable "hit" to "wired" so that it
is more obvious.

Obtained from: gibbs/ken

If we boot verbose, then print out 'interesting' CAM errors that otherwise
would hide problems (like Selection Timeout).

Kill the command (don't rerun it) if we had an AUTOSENSE failure.
If we had an AUTOSENSE failure, we don't know what SENSE DATA
we had for a CHECK CONDITION. It's far better to assume failure
in this case.

Hints overhaul:
- Replace some very poorly thought out API hacks that should have been
fixed a long while ago.
- Provide some much more flexible search functions (resource_find_*())
- Use strings for storage instead of an outgrowth of the rather
inconvenient temporary ioconf table from config(). We already had a
fallback to using strings before malloc/vm was running anyway.

In camperiphscsisenseerror(), don't return an error when the error action
is SS_NOP.

Submitted by: joerg

Fix an off-by-2 error in periphdriver_register(). The read side of the
bcopy would go off the end of the array by two elements, which sometimes
causes a panic if it happens to cross into a page that isn't mapped.

Submitted by: gibbs
Reviewed by: peter

If we have and error and are booting verbosely, don't be complaining
if this was a non-retryable selection timeout- wading through 256 targets
worth of Fibre Channel 'selection timeouts' is tedious at best.

Catch up to header include changes:
- <sys/mutex.h> now requires <sys/systm.h>
- <sys/mutex.h> and <sys/sx.h> now require <sys/lock.h>

Rewrite of the CAM error recovery code.

Some of the major changes include:

- The SCSI error handling portion of cam_periph_error() has
been broken out into a number of subfunctions to better
modularize the code that handles the hierarchy of SCSI errors.
As a result, the code is now much easier to read.

- String handling and error printing has been significantly
revamped. We now use sbufs to do string formatting instead
of using printfs (for the kernel) and snprintf/strncat (for
userland) as before.

There is a new catchall error printing routine,
cam_error_print() and its string-based counterpart,
cam_error_string() that allow the kernel and userland
applications to pass in a CCB and have errors printed out
properly, whether or not they're SCSI errors. Among other
things, this helped eliminate a fair amount of duplicate code
in camcontrol.

We now print out more information than before, including
the CAM status and SCSI status and the error recovery action
taken to remedy the problem.

- sbufs are now available in userland, via libsbuf. This
change was necessary since most of the error printing code
is shared between libcam and the kernel.

- A new transfer settings interface is included in this checkin.
This code is #ifdef'ed out, and is primarily intended to aid
discussion with HBA driver authors on the final form the
interface should take. There is example code in the ahc(4)
driver that implements the HBA driver side of the new
interface. The new transfer settings code won't be enabled
until we're ready to switch all HBA drivers over to the new
interface.

src/Makefile.inc1,
lib/Makefile: Add libsbuf. It must be built before libcam,
since libcam uses sbuf routines.

libcam/Makefile: libcam now depends on libsbuf.

libsbuf/Makefile: Add a makefile for libsbuf. This pulls in the
sbuf sources from sys/kern.

bsd.libnames.mk: Add LIBSBUF.

camcontrol/Makefile: Add -lsbuf. Since camcontrol is statically
linked, we can't depend on the dynamic linker
to pull in libsbuf.

camcontrol.c: Use cam_error_print() instead of checking for
CAM_SCSI_STATUS_ERROR on every failed CCB.

sbuf.9: Change the prototypes for sbuf_cat() and
sbuf_cpy() so that the source string is now a
const char *. This is more in line wth the
standard system string functions, and helps
eliminate warnings when dealing with a const
source buffer.

Fix a typo.

cam.c: Add description strings for the various CAM
error status values, as well as routines to
look up those strings.

Add new cam_error_string() and
cam_error_print() routines for userland and
the kernel.

cam.h: Add a new CAM flag, CAM_RETRY_SELTO.

Add enumerated types for the various options
available with cam_error_print() and
cam_error_string().

cam_ccb.h: Add new transfer negotiation structures/types.

Change inq_len in the ccb_getdev structure to
be "reserved". This field has never been
filled in, and will be removed when we next
bump the CAM version.

cam_debug.h: Fix typo.

cam_periph.c: Modularize cam_periph_error(). The SCSI error
handling part of cam_periph_error() is now
in camperiphscsistatuserror() and
camperiphscsisenseerror().

In cam_periph_lock(), increase the reference
count on the periph while we wait for our lock
attempt to succeed so that the periph won't go
away while we're sleeping.

cam_xpt.c: Add new transfer negotiation code. (ifdefed
out)

Add a new function, xpt_path_string(). This
is a string/sbuf analog to xpt_print_path().

scsi_all.c: Revamp string handing and error printing code.
We now use sbufs for much of the string
formatting code. More of that code is shared
between userland the kernel.

scsi_all.h: Get rid of SS_TURSTART, it wasn't terribly
useful in the first place.

Add a new error action, SS_REQSENSE. (Send a
request sense and then retry the command.)
This is useful when the controller hasn't
performed autosense for some reason.

Change the default actions around a bit.

scsi_cd.c,
scsi_da.c,
scsi_pt.c,
scsi_ses.c: SF_RETRY_SELTO -> CAM_RETRY_SELTO. Selection
timeouts shouldn't be covered by a sense flag.

scsi_pass.[ch]: SF_RETRY_SELTO -> CAM_RETRY_SELTO.

Get rid of the last vestiges of a read/write
interface.

libkern/bsearch.c,
sys/libkern.h,
conf/files: Add bsearch.c, which is needed for some of the
new table lookup routines.

aic7xxx_freebsd.c: Define AHC_NEW_TRAN_SETTINGS if
CAM_NEW_TRAN_CODE is defined.

sbuf.h,
subr_sbuf.c: Add the appropriate #ifdefs so sbufs can
compile and run in userland.

Change sbuf_printf() to use vsnprintf()
instead of kvprintf(), which is only available
in the kernel.

Change the source string for sbuf_cpy() and
sbuf_cat() to be a const char *.

Add __BEGIN_DECLS and __END_DECLS around
function prototypes since they're now exported
to userland.

kdump/mkioctls: Include stdio.h before cam.h since cam.h now
includes a function with a FILE * argument.

Submitted by: gibbs (mostly)
Reviewed by: jdp, marcel (libsbuf makefile changes)
Reviewed by: des (sbuf changes)
Reviewed by: ken

Change the peripheral driver list from a linker set to module driven
driver registration. This should allow things like da, sa, cd etc to be
in seperate KLD's to the cam core and make them preloadable.

Another round of the <sys/queue.h> FOREACH transmogriffer.

Created with: sed(1)
Reviewed by: md5(1)

Mechanical change to use <sys/queue.h> macro API instead of
fondling implementation details.

Created with: sed(1)
Reviewed by: md5(1)

Remove unused #includes

Reviewed by: ken

Change the "Duplicate Wired Device entry" printout in camperiphnextunit to
also mention the peripheral name, bus, target and lun of the device we
attempted to put in that slot. This gives the user a little more
information about what is going on.

Tested by: Andre Albsmeier <andre.albsmeier@mchp.siemens.de>
Discussed with: gibbs

Separate the struct bio related stuff out of <sys/buf.h> into
<sys/bio.h>.

<sys/bio.h> is now a prerequisite for <sys/buf.h> but it shall
not be made a nested include according to bdes teachings on the
subject of nested includes.

Diskdrivers and similar stuff below specfs::strategy() should no
longer need to include <sys/buf.> unless they need caching of data.

Still a few bogus uses of struct buf to track down.

Repocopy by: peter

Remove B_READ, B_WRITE and B_FREEBUF and replace them with a new
field in struct buf: b_iocmd. The b_iocmd is enforced to have
exactly one bit set.

B_WRITE was bogusly defined as zero giving rise to obvious coding
mistakes.

Also eliminate the redundant struct buf flag B_CALL, it can just
as efficiently be done by comparing b_iodone to NULL.

Should you get a panic or drop into the debugger, complaining about
"b_iocmd", don't continue. It is likely to write on your disk
where it should have been reading.

This change is a step in the direction towards a stackable BIO capability.

A lot of this patch were machine generated (Thanks to style(9) compliance!)

Vinum users: Greg has not had time to test this yet, be careful.

When autosense fails, make sure we retry the command.

PR: kern/15975
Submitted by: Carl Mascott <cmascott@world.std.com>, n_hibma

A minimal update to the device wiring code so that it looks at the
dynamic resource tables rather than relying on a duplicated cam-specific
table generated by config(8) in ioconf.c. This was a major holdup to
getting loader / userconfig driven configuration of scsi wiring.

Change useracc() and kernacc() to use VM_PROT_{READ|WRITE|EXECUTE} for the
"rw" argument, rather than hijacking B_{READ|WRITE}.

Fix two bugs (physio & cam) resulting by the confusion caused by this.

Submitted by: Tor.Egge@fast.no
Reviewed by: alc, ken (partly)

Put an upper bound on the number of BUSY status retries we'll do (use the
retry count for the ccb). This is probably not quite the right thing, but it
is better than silently hanging on (possibly broken) h/w which is what we
do now.

Reviewed by:Justin/Ken: they weren't entirely happy about it but didn't say no.

$Id$ -> $FreeBSD$

int/long printf format problems on the Alpha.

Convert buffer locking from using the B_BUSY and B_WANTED flags to using
lockmgr locks. This commit should be functionally equivalent to the old
semantics. That is, all buffer locking is done with LK_EXCLUSIVE
requests. Changes to take advantage of LK_SHARED and LK_RECURSIVE will
be done in future commits.

Run xpt_action at splsoftcam() so that it cannot be re-entered by our SWI.
The XPT doesn't have a problem with this itself, but some controllers
drivers may have been caught off guard by the old behavior.

XPT_CONT_TARGET_IO is also a valid ccb type for cam_periph_unmapmem.

Fix the read/write permission checks in cam_periph_mapmem(). If we're
writing, we want to be able to read the buffer. If we're reading, we want
to be able to write to the buffer.

PR: kern/11870
Submitted by: Andrew Mobbs <amobbs@allstor-sw.co.uk>

Rely on the statistics in XPT_GDEV_STATS instead of the versions still
retained in XPT_GDEV_TYPE for binary compatibility. Mark the legacy
structure values for removal when we bump the major CAM revision.

Add the XPT_PATH_STATS and XPT_GDEV_STATS function codes. These ccb
types allow the reporting of error counts and other statistics. Currently
we provide information on the last BDR or bus reset as well as active
transaction inforamtion, but this will be expanded as more information is
added to aid in error recovery.

Use the 'last reset' information to better handle bus settle delays.
Peripheral drivers now control whether a bus settle delay occurs and
for how long. This allows target mode peripheral drivers to avoid
having their device queue frozen by the XPT for what shoudl only be
initiator type behavior.

Don't perform a bus reset if the target device is incapable of performing
transfer negotiation (e.g. Fiber Channel).

If we don't perform a bus reset but the controller is capable of transfer
negotiations, force negotiations on the first transaction to go to the
device. This ensures that we aren't tripped up by a left over negotiation
from the prom, BIOS, loader, etc.

Add a default async handler funstion to cam_periph.c to remove duplicated
code in all initiator type peripheral drivers.

Allow mapping of XPT_CONT_TARGET_IO ccbs from userland. They are
itentical to XPT_SCSI_IO ccbs as far as data mapping is concerned.

Add a facility in the CAM error handling code to retry selection timeouts.
If the client requests that the error recovery code retry a selection
timeout, it will be retried after half a second. The delay is to give the
device time to recover.

For most of these drivers, I only added selection timeout retries where
they were also retrying unit attention type errors. The sa(4) driver calls
saerror() in a number of places, but most of them don't request retrying
unit attentions.

Also, bump the default minimum CD changer timeout from 2 to 5 seconds and
the maximum timeout from 10 to 15 seconds. Some Pioneer changers seem to
have trouble with the shorter timeout.

Reviewed by: gibbs

cam_periph.c:
Move handling of CAM_AUTOSENSE_FAIL into block dealing with
all other scsi status errors.

cam_queue.c:
cam_queue.h:
Fix 'off by one' heap bug in a more efficient manner. Since
heap algorithms like to deal with indexes started from 1,
offset our heap array pointer at allocation time to make this
so for a C environment. This makes the implementation of the
algorithm a bit more efficient.

cam_xpt.c:
Use macros for accessing the head of the heap so that code
is isolated from implementation details of the heap.

Use PHOLD/PRELE rather than P_PHYSIO.

This is a rather large commit that encompasses the new swapper,
changes to the VM system to support the new swapper, VM bug
fixes, several VM optimizations, and some additional revamping of the
VM code. The specific bug fixes will be documented with additional
forced commits. This commit is somewhat rough in regards to code
cleanup issues.

Reviewed by: "John S. Dyson" <root@dyson.iquest.net>, "David Greenman" <dg@root.com>

Replace includes of <sys/kernel.h> with includes of
<sys/linker_set.h> in those files that use only the linker set
definitions.

At Justin's request, limit the size of buffers that can be mapped into
and out of kernel address space (via the pass(4) and xpt(4) peripheral
drivers) to 64K (DFLTPHYS). Some controllers, like the Adaptec 1542,
don't support more than 64K transactions.

We plan on eventually having the capability of limiting this size based
on min(MAXPHYS, controller max), but since that capability isn't here yet,
limit things to the lowest common denominator.

Probable fix for the "cdda2wav" panics that various people have been
reporting since this past summer. (I think Daniel O'Conner was the first.)

The problem appears to have been something like this:

- cdda2wav by default passes in a buffer that is close to the 128K MAXPHYS
limit.
- many times, the buffer is not page aligned
- vmapbuf() truncates the address, so that it is page aligned
- that causes the total size of the buffer to be greater than MAXPHYS,
which of course is a bad thing.

Here's a quote from the PR (kern/9067):

==================
In particular, note bp->b_bufsize = 0x0001f950 and bp->b_data = 0xf2219960
(which does not start on a page boundary). vunmapbuf() loops through all
the pages without any difficulty until addr reaches 0xf2239000, and then
the panic occurs. This seems to indicate that we are exceeding MAXPHYS
since we actually started from the middle of a page (the data is being
transfered to a non page aligned location).

To complete the description, note that the system call originates from
ReadCddaMMC12() (in scsi_cmds.c of cdda2wav) with a request to read 55
audio sectors of 2352 bytes (which is calculated to fall under MAXPHYS).
This in turn ends up calling scsi_send() (in scsi-bsd.c) which calls
cam_fill_csio() and cam_send_ccb(). This results in a CAMIOCOMMAND ioctl
with a ccb function code of XPT_SCSI_IO.
==================

The fix is to change the size check in cam_periph_mapmem() so that it is
like the one in minphys(). In particular, it is something like:

if ((buffer_length + (buf_ptr & PAGE_MASK)) > MAXPHYS)
buffer is too big

My fix is based on the one in the PR, but I cleaned up a fair number of
things in cam_periph_mapmem(). The checks for each buffer to be mapped
are now in a separate loop from the actual mapping operation. With the new
arrangement, we don't have to bother with unmapping any previously mapped
buffers if one of the checks fails.

Many thanks to James Liu for tracking this down. I'd appreciate it if some
vm-savvy folks would look this over. I believe this fix is correct, but I
could be wrong.

PR: kern/9067 (also, kern/8112)
Reviewed by: gibbs
Submitted by: "James T. Liu" <jtliu@phlebas.rockefeller.edu>

Fix a problem with the way we handled device invalidation when attaching
to a device failed.

In theory, the same steps that happen when we get an AC_LOST_DEVICE async
notification should have been taken when a driver fails to attach. In
practice, that wasn't the case.

This only affected the da, cd and ch drivers, but the fix affects all
peripheral drivers.

There were several possible problems:
- In the da driver, we didn't remove the peripheral's softc from the da
driver's linked list of softcs. Once the peripheral and softc got
removed, we'd get a kernel panic the next time the timeout routine
called dasendorderedtag().
- In the da, cd and possibly ch drivers, we didn't remove the
peripheral's devstat structure from the devstat queue. Once the
peripheral and softc were removed, this could cause a panic if anyone
tried to access device statistics. (one component of the linked list
wouldn't exist anymore)
- In the cd driver, we didn't take the peripheral off the changer run
queue if it was scheduled to run. In practice, it's highly unlikely,
and maybe impossible that the peripheral would have been on the
changer run queue at that stage of the probe process.

The fix is:
- Add a new peripheral callback function (the "oninvalidate" function)
that is called the first time cam_periph_invalidate() is called for a
peripheral.

- Create new foooninvalidate() routines for each peripheral driver. This
routine is always called at splsoftcam(), and contains all the stuff
that used to be in the AC_LOST_DEVICE case of the async callback
handler.

- Move the devstat cleanup call to the destructor/cleanup routines, since
some of the drivers do I/O in their close routines.

- Make sure that when we're flushing the buffer queue, we traverse it at
splbio().

- Add a check for the invalid flag in the pt driver's open routine.

Reviewed by: gibbs

Clean up some unused variables.

Reviewed by: ken
Submitted by: phk

Fix a bug in the error recovery code. It was possible to have more than
one error recovery action oustanding for a given peripheral.

This is bad for several reasons. The first problem is that the error
recovery actions would likely be to fix the same problem. (e.g., we
queue 5 CCBs to a disk, and the first one comes back with 0x04,0x02. We
start error recovery, and the second one comes back with the same status.
Then the third one comes back, and so on. Each one causes the drive to get
nailed with a start unit, when we really only need one.)

The other problem is that we only have space to store one CCB while we're
doing error recovery. The subsequent error recovery actions that got
started were over-writing the CCBs from previous error recovery actions,
but we still tried to call the done routine N times for N error recovery
actions. Each call to dadone() was done with the same CCB, though. So on
the second one, we got a "biodone: buffer not busy" panic, since the buffer
in question had already been through biodone().

In any case, this fixes things so that any any given time, there's only one
error recovery action outstanding for any given peripheral driver.

Reviewed by: gibbs
Reported by: Philippe Regnauld <regnauld@deepo.prosa.dk>
[ Philippe wins the "bug finder of the week" award ]

Fixed printf format errors. u_long is not necessarily suitable for casting
pointers to, and %d is not suitable for printing uint32_t's.

cam_xpt.c:
Add quirk entry for a Samsung drive that doesn't like experiencing
the queue full condition.

Bump the timeouts for all probe activities to 60s. We don't know
what the seletion timeout (or equivelent on other mediums) is
for controllers, which can make the transactions at the tail
end of a parallel probe take a while to complete. The DPT
seems to be a card that takes a long time to see a selection timeout.

cam_periph.c:
Don't call a device "gone" after a single selection timeout. We
need to come up with a better policy. Until that time, you'll
have to manually re-scan a bus via camcontrol for the system to
decide that a device is really gone. This should give devices
experiencing temporary insanity to escape death.

CAM Transport Layer (XPT).

Submitted by: The CAM Team