ahci: Use device_set_desc(f)()

No functional change intended.

MFC after: 1 week

new-bus: Remove the 'rid' and 'type' arguments from BUS_RELEASE_RESOURCE

The public bus_release_resource() API still accepts both forms, but
the internal kobj method no longer passes the arguments.
Implementations which need the rid or type now use rman_get_rid() or
rman_get_type() to fetch the value from the allocated resource.

Reviewed by: imp
Differential Revision: https://reviews.freebsd.org/D44131

sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, drop -FreeBSD

The SPDX folks have obsoleted the BSD-2-Clause-FreeBSD identifier. Catch
up to that fact and revert to their recommended match of BSD-2-Clause.

Discussed with: pfg
MFC After: 3 days
Sponsored by: Netflix

ahci: increase timout

For some devices, like Marvell 88SE9230, it takes more time
to connect to the device. This patch introduces a special flag
that extends the timeout from around 100ms to around 500ms.

This change is based on the work of: Peter Eriksson <pen@lysator.liu.se>

PR: 243401
Reviewed by: imp
Tested by: dch
MFC after: 3 days
Sponsored by: Equinix
Sponsored by: SkunkWerks, GmbH
Sponsored by: Klara, Inc.
Differential Revision: https://reviews.freebsd.org/D38413

ahci(4): Fix a typo in a source code comment

- s/reqests/requests/

MFC after: 3 days

ahci: Remove unused devclass arguments to DRIVER_MODULE.

Remove unused ahci_devclass.

ahci/siis/mvs: Fix panics after 3394d4239b.

Full CCB header overwrites made frees go into wrong zones, causing
kernel panics. Instead of copying full header use xpt_setup_ccb(),
since the only field I see used from all the header is target_id.

PR: 262263

ahci(4): Allow enclosure emulation without hardware.

After 53f5ac1310e allowed SATA device mapping to enclosure slots,
it may have sense to provide enclosure device emulation even without
real hardware interface like SGPIO just for purposes of physical
device location tracking (still assuming straight cabling).

MFC after: 1 week
Sponsored by: iXsystems, Inc.

Mark some sysctls as CTLFLAG_MPSAFE.

MFC after: 2 weeks

newbus: Move from bus_child_{pnpinfo,location}_src to bus_child_{pnpinfo,location} with sbuf

Now that the upper layers all go through a layer to tie into these
information functions that translates an sbuf into char * and len. The
current interface suffers issues of what to do in cases of truncation,
etc. Instead, migrate all these functions to using struct sbuf and these
issues go away. The caller is also in charge of any memory allocation
and/or expansion that's needed during this process.

Create a bus_generic_child_{pnpinfo,location} and make it default. It
just returns success. This is for those busses that have no information
for these items. Migrate the now-empty routines to using this as
appropriate.

Document these new interfaces with man pages, and oversight from before.

Reviewed by: jhb, bcr
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D29937

Make MAXPHYS tunable. Bump MAXPHYS to 1M.

Replace MAXPHYS by runtime variable maxphys. It is initialized from
MAXPHYS by default, but can be also adjusted with the tunable kern.maxphys.

Make b_pages[] array in struct buf flexible. Size b_pages[] for buffer
cache buffers exactly to atop(maxbcachebuf) (currently it is sized to
atop(MAXPHYS)), and b_pages[] for pbufs is sized to atop(maxphys) + 1.
The +1 for pbufs allow several pbuf consumers, among them vmapbuf(),
to use unaligned buffers still sized to maxphys, esp. when such
buffers come from userspace (*). Overall, we save significant amount
of otherwise wasted memory in b_pages[] for buffer cache buffers,
while bumping MAXPHYS to desired high value.

Eliminate all direct uses of the MAXPHYS constant in kernel and driver
sources, except a place which initialize maxphys. Some random (and
arguably weird) uses of MAXPHYS, e.g. in linuxolator, are converted
straight. Some drivers, which use MAXPHYS to size embeded structures,
get private MAXPHYS-like constant; their convertion is out of scope
for this work.

Changes to cam/, dev/ahci, dev/ata, dev/mpr, dev/mpt, dev/mvs,
dev/siis, where either submitted by, or based on changes by mav.

Suggested by: mav (*)
Reviewed by: imp, mav, imp, mckusick, scottl (intermediate versions)
Tested by: pho
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D27225

Fix stupid math mistake in r366922.

MFC after: 3 days

Add icc (Isochronous Command Completion) ccb_ataio field.

MFC after: 1 week

Pass lower 3 bits of sector_count for FPDMA commands.

When this code was written those bits were N/A, but now the lowest bit
is Rebuild Assist Recovery Control (RARC).

MFC after: 1 month

ahci: clean up empty lines in .c and .h files

Add support for AHCI BIOS/OS Handoff.

This allows clean handoff from BIOS implementing some asynchronous I/O to
the OS AHCI driver. During attach driver declares OS ownership request
and waits from 25ms to 2s for BIOS to complete operation and release the
hardware.

MFC after: 2 weeks

Mark more nodes as CTLFLAG_MPSAFE or CTLFLAG_NEEDGIANT (17 of many)

r357614 added CTLFLAG_NEEDGIANT to make it easier to find nodes that are
still not MPSAFE (or already are but aren’t properly marked).
Use it in preparation for a general review of all nodes.

This is non-functional change that adds annotations to SYSCTL_NODE and
SYSCTL_PROC nodes using one of the soon-to-be-required flags.

Mark all obvious cases as MPSAFE. All entries that haven't been marked
as MPSAFE before are by default marked as NEEDGIANT

Approved by: kib (mentor, blanket)
Commented by: kib, gallatin, melifaro
Differential Revision: https://reviews.freebsd.org/D23718

Use a void * argument to callout handlers instead of timeout_t casts.

Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D22684

Fix AHCI Enclosure Management, broken by r351356.

ivars value of -1 was used to distinguish EM device, and r351356 left some
wrong checks for it. Give EM device separate flag there instead.

Create a AHCI attachment for nvme.

Intel has created RST and many laptops from vendors like Lenovo and Asus. It's a
mechanism for creating multiple boot devices under windows. It effectively hides
the nvme drive inside of the ahci controller. The details are supposed to be a
trade secret. However, there's a reverse engineered Linux driver, and this
implements similar operations to allow nvme drives to attach. The ahci driver
attaches nvme children that proxy the remapped resources to the child. nvme_ahci
is just like nvme_pci, except it doesn't do the PCI specific things. That's
moved into ahci where appropriate.

When the nvme drive is remapped, MSI-x interrupts aren't forwarded (the linux
driver doesn't know how to use this either). INTx interrupts are used
instead. This is suboptimal, but usually sufficient for the laptops these parts
are in.

This is based loosely on https://www.spinics.net/lists/linux-ide/msg53364.html
submitted, but not accepted by, Linux. It was written by Dan Williams. These
changes were written from scratch by Olivier Houchard.

Submitted by: cognet@ (Olivier Houchard)

Improve AHCI Enclosure Management and SES interoperation.

Since SES specs do not define mechanism to map enclosure slots to SATA
disks, AHCI EM code I written many years ago appeared quite useless,
that always bugged me. I was thinking whether it was a good idea, but
if LSI HBAs do that, why I shouldn't?

This change introduces simple non-standard mechanism for the mapping
into both AHCI EM and SES code, that makes AHCI EM on capable controllers
(most of Intel's) a first-class SES citizen, allowing it to report disk
physical path to GEOM, show devices inserted into each enclosure slot in
`sesutil map` and `getencstat`, control locate and fault LEDs for specific
devices with `sesutil locate adaX on` and `sesutil fault adaX on`, etc.

I've successfully tested this on Supermicro X10DRH-i motherboard connected
with sideband cable of its S-SATA Mini-SAS connector to SAS815TQ backplane.
It can indicate with LEDs Locate, Fault and Rebuild/Remap SES statuses for
each disk identical to real SES of Supermicro SAS2 backplanes.

MFC after: 2 weeks

Fix use of busdma(9) KPI in ahci(4).

Use BUS_DMA_NOWAIT for loads at initialization time.
Report actual numeric error code if any problem occurs at the
initialization.

Reported and tested by: pho
Reviewed by: mav
Sponsored by: The FreeBSD Foundation
MFC after: 1 week
Differential revision: https://reviews.freebsd.org/D18741

Implement ability to turn on/off PHYs for AHCI devices.

As part of Chuck's work on fixing kernel crashes caused by disk I/O
errors, it is useful to be able to trigger various kinds of
errors. This patch allows causing an AHCI-attached disk to disappear,
by having the driver keep the PHY disabled when the driver would
otherwise enable the PHY. It also allows making the disk reappear by
having the driver go back to setting the PHY enable/disable state as
it normal would and simulating the hardware event that causes a bus
rescan.

Submitted by: Chuck Silvers
Sponsored by: Netflix, Inc
Differential Revision: https://reviews.freebsd.org/D16043

Fixed endianess issue in AHCI driver

There were some bits that were being set in cmd_flags (a field of AHCI's
command list structure) after cmd_flags was converted to little endian.
On a big endian host, such as PowerPC, this would set the wrong bits.
This was preventing AHCI driver from working on these hosts.

Reviewed by: jhibbits
Approved by: jhibbits (mentor)

sys/dev: further adoption of SPDX licensing ID tags.

Mainly focus on files that use BSD 2-Clause license, however the tool I
was using misidentified many licenses so this was mostly a manual - error
prone - task.

The Software Package Data Exchange (SPDX) group provides a specification
to make it easier for automated tools to detect and summarize well known
opensource licenses. We are gradually adopting the specification, noting
that the tags are considered only advisory and do not, in any way,
superceed or replace the license texts.

Define a single instance of ahci_devclass and reference it from all the
attachment code for various SOCs and busses. Remove all the static and
should-have-been-static and named-differently instances of it.

This should eliminate the recently-grown build warnings about multiple
definitions when building arm kernels.

Check if the device is marked as dma-coherent in the FDT, and if so, let
busdma know, so that on architectures where dma isn't always coherent, we
know we don't have to write-back/invalidates cachelines on DMA operations.

Reviewed by: andrew, mav

Block FPDMA TRIM for ASMedia HBAs.

Experiments show FPDMA TRIM command timeouts on ASMedia HBAs, while the
same SSDs working fine on Intel HBAs.

MFC after: 2 weeks

Remove dead mentions of CAM target mode APIs from drivers.

This makes grepping kernel for target mode implementation much easier.

Partially workaround ASMedia HBA error recovery.

Taking closer look on my ASM1062 I found that it has bunch of issues around
error recovery: reported wrong CCS, failed commands reported as completed,
READ LOG EXT times out after NCQ error. This patch workarounds first two
problems, that were making ATAPI devices close to unusable on these HBAs.

MFC after: 2 weeks

Enable optional soft reset in AHCI

It occurred that some Marvell integrated controllers
require additional time after soft reset to work properly.
Introduce new quirk (AHCI_Q_MRVL_SR_DEL), that enable
such operation.

Submitted by: Konrad Adamczyk <ka@semihalf.com>
Obtained from: Semihalf
Sponsored by: Stormshield
Reviewed by: mav
Differential revision: https://reviews.freebsd.org/D9221

Always null-terminate ccb_pathinq.(sim_vid|hba_vid|dev_name)

The sim_vid, hba_vid, and dev_name fields of struct ccb_pathinq are
fixed-length strings. AFAICT the only place they're read is in
sbin/camcontrol/camcontrol.c, which assumes they'll be null-terminated.
However, the kernel doesn't null-terminate them. A bunch of copy-pasted code
uses strncpy to write them, and doesn't guarantee null-termination. For at
least 4 drivers (mpr, mps, ciss, and hyperv), the hba_vid field actually
overflows. You can see the result by doing "camcontrol negotiate da0 -v".

This change null-terminates those fields everywhere they're set in the
kernel. It also shortens a few strings to ensure they'll fit within the
16-character field.

PR: 215474
Reported by: Coverity
CID: 1009997 1010000 1010001 1010002 1010003 1010004 1010005
CID: 1331519 1010006 1215097 1010007 1288967 1010008 1306000
CID: 1211924 1010009 1010010 1010011 1010012 1010013 1010014
CID: 1147190 1010017 1010016 1010018 1216435 1010020 1010021
CID: 1010022 1009666 1018185 1010023 1010025 1010026 1010027
CID: 1010028 1010029 1010030 1010031 1010033 1018186 1018187
CID: 1010035 1010036 1010042 1010041 1010040 1010039
Reviewed by: imp, sephe, slm
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D9037
Differential Revision: https://reviews.freebsd.org/D9038

Process port interrupt even is PxIS register is zero.

ASMedia ASM1062 AHCI chips with some fancy firmware handling PMP inside
seems sometimes forgeting to set bits in PxIS, causing command timeouts.
Removal of this check fixes the issue by the theoretical cost of slightly
higher CPU usage in some odd cases, but this is what Linux does too.

MFC after: 1 month

Fix channel initialization in FBS mode.

Due to reading initialized variable, FIS receive area was always allocated
as 256 bytes, suitable for command-based switching, instead of 4096 bytes,
required for FIS-based switching. This caused memory corruption in case of
port multipliers used on FBS-capable HBAs (Marvell).

MFC after: 1 week

In AHCI_IRQ_MODE_AFTER mode do not clear interrupts below.

This is probably a NOP change since IS register is not activery used for
interrupts below the shared, but it looked odd to clear interrupts we did
not handle.

Do not consider the last interrupt shared if there are enough interrupts
for all channels.

Fix ahci(4) driver attach to controller with 32 ports.

Incorrect sign expansion in variables that supposed to be a bit fields
caused infinite loop. Fixing this allows system properly detect maximal
possible 32 devices configured on AHCI HBA of BHyVe. That case did not
happen in a wild before due to lack of hardware AHCI HBAs with 32 ports.

Approved by: re (gjb@)
MFC after: 1 week

Add support for managing Shingled Magnetic Recording (SMR) drives.

This change includes support for SCSI SMR drives (which conform to the
Zoned Block Commands or ZBC spec) and ATA SMR drives (which conform to
the Zoned ATA Command Set or ZAC spec) behind SAS expanders.

This includes full management support through the GEOM BIO interface, and
through a new userland utility, zonectl(8), and through camcontrol(8).

This is now ready for filesystems to use to detect and manage zoned drives.
(There is no work in progress that I know of to use this for ZFS or UFS, if
anyone is interested, let me know and I may have some suggestions.)

Also, improve ATA command passthrough and dispatch support, both via ATA
and ATA passthrough over SCSI.

Also, add support to camcontrol(8) for the ATA Extended Power Conditions
feature set. You can now manage ATA device power states, and set various
idle time thresholds for a drive to enter lower power states.

Note that this change cannot be MFCed in full, because it depends on
changes to the struct bio API that break compatilibity. In order to
avoid breaking the stable API, only changes that don't touch or depend on
the struct bio changes can be merged. For example, the camcontrol(8)
changes don't depend on the new bio API, but zonectl(8) and the probe
changes to the da(4) and ada(4) drivers do depend on it.

Also note that the SMR changes have not yet been tested with an actual
SCSI ZBC device, or a SCSI to ATA translation layer (SAT) that supports
ZBC to ZAC translation. I have not yet gotten a suitable drive or SAT
layer, so any testing help would be appreciated. These changes have been
tested with Seagate Host Aware SATA drives attached to both SAS and SATA
controllers. Also, I do not have any SATA Host Managed devices, and I
suspect that it may take additional (hopefully minor) changes to support
them.

Thanks to Seagate for supplying the test hardware and answering questions.

sbin/camcontrol/Makefile:
Add epc.c and zone.c.

sbin/camcontrol/camcontrol.8:
Document the zone and epc subcommands.

sbin/camcontrol/camcontrol.c:
Add the zone and epc subcommands.

Add auxiliary register support to build_ata_cmd(). Make sure to
set the CAM_ATAIO_NEEDRESULT, CAM_ATAIO_DMA, and CAM_ATAIO_FPDMA
flags as appropriate for ATA commands.

Add a new get_ata_status() function to parse ATA result from SCSI
sense descriptors (for ATA passthrough over SCSI) and ATA I/O
requests.

sbin/camcontrol/camcontrol.h:
Update the build_ata_cmd() prototype

Add get_ata_status(), zone(), and epc().

sbin/camcontrol/epc.c:
Support for ATA Extended Power Conditions features. This includes
support for all features documented in the ACS-4 Revision 12
specification from t13.org (dated February 18, 2016).

The EPC feature set allows putting a drive into a power power mode
immediately, or setting timeouts so that the drive will
automatically enter progressively lower power states after various
idle times.

sbin/camcontrol/fwdownload.c:
Update the firmware download code for the new build_ata_cmd()
arguments.

sbin/camcontrol/zone.c:
Implement support for Shingled Magnetic Recording (SMR) drives
via SCSI Zoned Block Commands (ZBC) and ATA Zoned Device ATA
Command Set (ZAC).

These specs were developed in concert, and are functionally
identical. The primary differences are due to SCSI and ATA
differences. (SCSI is big endian, ATA is little endian, for
example.)

This includes support for all commands defined in the ZBC and
ZAC specs.

sys/cam/ata/ata_all.c:
Decode a number of additional ATA command names in ata_op_string().

Add a new CCB building function, ata_read_log().

Add ata_zac_mgmt_in() and ata_zac_mgmt_out() CCB building
functions. These support both DMA and NCQ encapsulation.

sys/cam/ata/ata_all.h:
Add prototypes for ata_read_log(), ata_zac_mgmt_out(), and
ata_zac_mgmt_in().

sys/cam/ata/ata_da.c:
Revamp the ada(4) driver to support zoned devices.

Add four new probe states to gather information needed for zone
support.

Add a new adasetflags() function to avoid duplication of large
blocks of flag setting between the async handler and register
functions.

Add new sysctl variables that describe zone support and paramters.

Add support for the new BIO_ZONE bio, and all of its subcommands:
DISK_ZONE_OPEN, DISK_ZONE_CLOSE, DISK_ZONE_FINISH, DISK_ZONE_RWP,
DISK_ZONE_REPORT_ZONES, and DISK_ZONE_GET_PARAMS.

sys/cam/scsi/scsi_all.c:
Add command descriptions for the ZBC IN/OUT commands.

Add descriptions for ZBC Host Managed devices.

Add a new function, scsi_ata_pass() to do ATA passthrough over
SCSI. This will eventually replace scsi_ata_pass_16() -- it
can create the 12, 16, and 32-byte variants of the ATA
PASS-THROUGH command, and supports setting all of the
registers defined as of SAT-4, Revision 5 (March 11, 2016).

Change scsi_ata_identify() to use scsi_ata_pass() instead of
scsi_ata_pass_16().

Add a new scsi_ata_read_log() function to facilitate reading
ATA logs via SCSI.

sys/cam/scsi/scsi_all.h:
Add the new ATA PASS-THROUGH(32) command CDB. Add extended and
variable CDB opcodes.

Add Zoned Block Device Characteristics VPD page.

Add ATA Return SCSI sense descriptor.

Add prototypes for scsi_ata_read_log() and scsi_ata_pass().

sys/cam/scsi/scsi_da.c:
Revamp the da(4) driver to support zoned devices.

Add five new probe states, four of which are needed for ATA
devices.

Add five new sysctl variables that describe zone support and
parameters.

The da(4) driver supports SCSI ZBC devices, as well as ATA ZAC
devices when they are attached via a SCSI to ATA Translation (SAT)
layer. Since ZBC -> ZAC translation is a new feature in the T10
SAT-4 spec, most SATA drives will be supported via ATA commands
sent via the SCSI ATA PASS-THROUGH command. The da(4) driver will
prefer the ZBC interface, if it is available, for performance
reasons, but will use the ATA PASS-THROUGH interface to the ZAC
command set if the SAT layer doesn't support translation yet.
As I mentioned above, ZBC command support is untested.

Add support for the new BIO_ZONE bio, and all of its subcommands:
DISK_ZONE_OPEN, DISK_ZONE_CLOSE, DISK_ZONE_FINISH, DISK_ZONE_RWP,
DISK_ZONE_REPORT_ZONES, and DISK_ZONE_GET_PARAMS.

Add scsi_zbc_in() and scsi_zbc_out() CCB building functions.

Add scsi_ata_zac_mgmt_out() and scsi_ata_zac_mgmt_in() CCB/CDB
building functions. Note that these have return values, unlike
almost all other CCB building functions in CAM. The reason is
that they can fail, depending upon the particular combination
of input parameters. The primary failure case is if the user
wants NCQ, but fails to specify additional CDB storage. NCQ
requires using the 32-byte version of the SCSI ATA PASS-THROUGH
command, and the current CAM CDB size is 16 bytes.

sys/cam/scsi/scsi_da.h:
Add ZBC IN and ZBC OUT CDBs and opcodes.

Add SCSI Report Zones data structures.

Add scsi_zbc_in(), scsi_zbc_out(), scsi_ata_zac_mgmt_out(), and
scsi_ata_zac_mgmt_in() prototypes.

sys/dev/ahci/ahci.c:
Fix SEND / RECEIVE FPDMA QUEUED in the ahci(4) driver.

ahci_setup_fis() previously set the top bits of the sector count
register in the FIS to 0 for FPDMA commands. This is okay for
read and write, because the PRIO field is in the only thing in
those bits, and we don't implement that further up the stack.

But, for SEND and RECEIVE FPDMA QUEUED, the subcommand is in that
byte, so it needs to be transmitted to the drive.

In ahci_setup_fis(), always set the the top 8 bits of the
sector count register. We need it in both the standard
and NCQ / FPDMA cases.

sys/geom/eli/g_eli.c:
Pass BIO_ZONE commands through the GELI class.

sys/geom/geom.h:
Add g_io_zonecmd() prototype.

sys/geom/geom_dev.c:
Add new DIOCZONECMD ioctl, which allows sending zone commands to
disks.

sys/geom/geom_disk.c:
Add support for BIO_ZONE commands.

sys/geom/geom_disk.h:
Add a new flag, DISKFLAG_CANZONE, that indicates that a given
GEOM disk client can handle BIO_ZONE commands.

sys/geom/geom_io.c:
Add a new function, g_io_zonecmd(), that handles execution of
BIO_ZONE commands.

Add permissions check for BIO_ZONE commands.

Add command decoding for BIO_ZONE commands.

sys/geom/geom_subr.c:
Add DDB command decoding for BIO_ZONE commands.

sys/kern/subr_devstat.c:
Record statistics for REPORT ZONES commands. Note that the
number of bytes transferred for REPORT ZONES won't quite match
what is received from the harware. This is because we're
necessarily counting bytes coming from the da(4) / ada(4) drivers,
which are using the disk_zone.h interface to communicate up
the stack. The structure sizes it uses are slightly different
than the SCSI and ATA structure sizes.

sys/sys/ata.h:
Add many bit and structure definitions for ZAC, NCQ, and EPC
command support.

sys/sys/bio.h:
Convert the bio_cmd field to a straight enumeration. This will
yield more space for additional commands in the future. After
change r297955 and other related changes, this is now possible.
Converting to an enumeration will also prevent use as a bitmask
in the future.

sys/sys/disk.h:
Define the DIOCZONECMD ioctl.

sys/sys/disk_zone.h:
Add a new API for managing zoned disks. This is very close to
the SCSI ZBC and ATA ZAC standards, but uses integers in native
byte order instead of big endian (SCSI) or little endian (ATA)
byte arrays.

This is intended to offer to the complete feature set of the ZBC
and ZAC disk management without requiring the application developer
to include SCSI or ATA headers. We also use one set of headers
for ioctl consumers and kernel bio-level consumers.

sys/sys/param.h:
Bump __FreeBSD_version for sys/bio.h command changes, and inclusion
of SMR support.

usr.sbin/Makefile:
Add the zonectl utility.

usr.sbin/diskinfo/diskinfo.c
Add disk zoning capability to the 'diskinfo -v' output.

usr.sbin/zonectl/Makefile:
Add zonectl makefile.

usr.sbin/zonectl/zonectl.8
zonectl(8) man page.

usr.sbin/zonectl/zonectl.c
The zonectl(8) utility. This allows managing SCSI or ATA zoned
disks via the disk_zone.h API. You can report zones, reset write
pointers, get parameters, etc.

Sponsored by: Spectra Logic
Differential Revision: https://reviews.freebsd.org/D6147
Reviewed by: wblock (documentation)

sys/dev: minor spelling fixes.

Most affect comments, very few have user-visible effects.

Implement Auxiliary register. Add PIM_ATA_EXT flag to flag that a SIM
can handle it, and add the code to add it to the FIS that's sent to
the drive. The mvs driver is the only other ATA driver in the system,
and its hardware doesn't appear to support setting the Auxiliary
register.

Differential Revision: https://reviews.freebsd.org/D5598

New CAM I/O scheduler for FreeBSD. The default I/O scheduler is the same
as before. The common scheduling bits have moved from inline code in
each of the CAM periph drivers into a library that implements the
default scheduling.

In addition, a number of rate-limiting and I/O preference options can
be enabled by adding CAM_IOSCHED_NETFLIX to your config file. A number
of extra stats are also maintained. CAM_IOSCHED_NETFLIX isn't on by
default because it uses a separate BIO_READ and BIO_WRITE queue, so
doesn't honor BIO_ORDERED between these two types of operations. We
already didn't honor it for BIO_DELETE, and we don't depend on
BIO_ORDERED between reads and writes anywhere in the system (it is
currently used with BIO_FLUSH in ZFS to make sure some writes are
complete before others start and as a poor-man's soft dependency in
one place in UFS where we won't be issuing READs until after the
operation completes). However, out of an abundance of caution, it
isn't enabled by default.

Plus, this also brings in NCQ TRIM support for those SSDs that support
it. A black list is also provided for known rogues that use NCQ trim
as an excuse to corrupt the drive. It was difficult to separate out
into a separate commit.

This code has run in production at Netflix for over a year now.

Sponsored by: Netflix, Inc
Differential Revision: https://reviews.freebsd.org/D4609

Convert a long to rman_res_t, fixing a sign extension bug.

ahci.c had one signed long, which was passed into rman, rather than u_long.
After the switch of rman_res_t from size u_long to size uintmax_t, the sign
extension caused ranges to get messed up, and ahcich* to not attach.

There may be more signed longs used in this way, which will be fixed as they're
reported.

Reported by: pho

Convert rman to use rman_res_t instead of u_long

Summary:
Migrate to using the semi-opaque type rman_res_t to specify rman resources. For
now, this is still compatible with u_long.

This is step one in migrating rman to use uintmax_t for resources instead of
u_long.

Going forward, this could feasibly be used to specify architecture-specific
definitions of resource ranges, rather than baking a specific integer type into
the API.

This change has been broken out to facilitate MFC'ing drivers back to 10 without
breaking ABI.

Reviewed By: jhb
Sponsored by: Alex Perez/Inertial Computing
Differential Revision: https://reviews.freebsd.org/D5075

AHCI: Fix AHCI driver for ARM.
On ARM, we must ensure proper interdevice write ordering.
The AHCI interrupt status register must be updated in HW before
registers in interrupt controller.
Unfortunately, only way how we can do it is readback.

Discussed with: mav
Approved by: kib (mentor)
Differential Revision: https://reviews.freebsd.org/D4240

AHCI: Use bus_dmamap_sync(9) when accessing DMA buffers.

Reviewed by: mav
Approved by: kib (mentor)
Differential Revision: https://reviews.freebsd.org/D4240

Introduce support for MSI-X interrupts in AHCI

- Allocate resources for MSI-X table and PBA if necessary
- Add function ahci_free_mem() to free all resources

Reviewed by: jhb, mav
Obtained from: Semihalf
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D3009

Add AHCI attachment code for Allwinner A10/A20 SoCs.

The Allwinner SoC has an AHCI device on its internal main bus rather
than the PCI bus. This SoC is somewhat underdocumented, and its SATA
controller is no exception. The methods to support this chip were
harvested from the Linux Allwinner SDK, and then constants invented to
describe what's going on based on low-level constants contained in the
SATA standard and guess work.

This SoC requires a specific AHCI channel setup in order to start the
operations on the channel properly.

Clock setup and AHCI channel setup idea came from NetBSD.

Tested on Cubieboard 2 and Banana pi (and attachment on Cubieboard by
Pratik Singhal).

Differential Revision: https://reviews.freebsd.org/D737
Submitted by: imp
Reviewed by: imp, ganbold, mav, andrew

Reduce priority of ATA/SATA drivers.

Legacy ata(4) -> BUS_PROBE_LOW_PRIORITY; more functional ahci(4), siis(4),
mvs(4) -> BUS_PROBE_DEFAULT; BUS_PROBE_VENDOR leave for vendor drivers.

MFC after: 2 weeks

Introduce Annapurna Labs AHCI support

Overview:
* implemented quirk for forcing SATA interface enable
* restore value to status register - this enables link autonegotiation

Modifications:
* devid:vendorid field
* quirk for forcing PI setting (BIOS is doing that on PC-like systems)
* write to capabilites field to enable phy link initialization

Submitted by: Wojciech Macek <wma@semihalf.com>
Reviewed by: imp, mav
Obtained from: Semihalf

For some uniformity move ahci_ch_init() call under the lock.

Submitted by: Dmitry Luhtionov <dmitryluhtionov@gmail.com>
MFC after: 2 weeks

- Const'ify the ahci_ids table.
- Use DEVMETHOD_END.
- Use NULL instead of 0 for pointers.

MFC after: 3 days

style (9) nits

Use return (val); instead of return val;

Differential Revision: D838
MFC after: 2 weeks
Sponsored by: Multiplay

Return the error from ahci_setup_interrupt in ahci_attach

Previously ahci_attach returned a hard coded ENXIO instead of the value
from ahci_setup_interrupt. This is effectively a NOOP change as currently
ahci_setup_interrupt only ever returns 0 or ENXIO, so just there to protect
against any future changes to that.

Differential Revision: D838
MFC after: 2 weeks
Sponsored by: Multiplay

Clamp ahci max irq's to AHCI_MAX_IRQS

This prevents the possiblity of any overruns on the statically allocated
struct irqs field.

Differential Revision: D838
MFC after: 2 weeks
X-MFC-With: r276012
Sponsored by: Multiplay

Prevent overflow issues in timeout processing

Previously, any timeout value for which (timeout * hz) will overflow the
signed integer, will give weird results, since callout(9) routines will
convert negative values of ticks to '1'. For unsigned integer overflow we
will get sufficiently smaller timeout values than expected.

Switch from callout_reset, which requires conversion to int based ticks
to callout_reset_sbt to avoid this.

Also correct isci to correctly resolve ccb timeout.

This was based on the original work done by Eygene Ryabinkin
<rea@freebsd.org> back in 5 Aug 2011 which used a macro to help avoid
the overlow.

Differential Revision: https://reviews.freebsd.org/D1157
Reviewed by: mav, davide
MFC after: 1 month
Sponsored by: Multiplay

A dma tag alignment should be 1 (not 0) when a device can dma at any address.

Use r271207 optimization only for MSI-enabled HBAs.

It was found that VirtualBox' AHCI does not allow nterrupt to be cleared
before the interrupt status register is read, causing interrupt storm.

AHCI specification allows to skip this register use when multi-vector MSI
is enabled and so interrupting port is known. For single-vector MSI that
is not stated explicitly, but if the port is only one, it is obviously
known too.

Use a local STAILQ for unlocked done CCB processing in ahci direct mode

Previously it was possible for issues e.g. use after free, to result
from processing the done queue while not holding the channel lock.

While this should never happen in practice, unexpected code flows
which result in two threads processing from the same queue may
be possible.

We now use a local STAILQ to prevent this ever being an issue.

Sponsored by: Multiplay

Initialize variables before resource_int_value().

Submitted by: Dmitry Luhtionov <dmitryluhtionov@gmail.com>

Bunch of microoptimizations to reduce dereferences and cache collisions.

Save one register read (AHCI_IS) for AHCI controllers with only one port.

For controllers with only one port (like PCIe or M.2 SSDs) interrupt can
come from only one source, and skipping read saves few percents of CPU time.

MFC after: 1 month
H/W donated by: I/O Switch

Restore order of interrupt setup. Minor problems can result by
setting up the interrupts too early:

Reviewed by: mav@
Sponsored by: Netflix

Fix typo in comments.

Submitted by: Benedict Reuschling <bcr@FreeBSD.org>
MFC after: 6 days

Invert AHCI_Q_NOBSYRES quirk meaning, waiting for readiness by default.

I gave up to update list of Marvell chips that require this quirk.
The final nail was growing number of PCIe/M.2 SSDs where Marvell chips
have PCI IDs of different vendors.

MFC after: 1 week
H/W donated by: I/O Switch

Separate out PCI attachment from the main AHCI driver. Move checks of
PCI IDs into quirks, which mostly fit (though you'd get no argument
from me that AHCI_Q_SATA1_UNIT0 is oddly specific). Set these quirks
in the PCI attachment. Make some shared functions public so that PCI
and possibly other bus attachments can use them.

The split isn't perfect yet, but it is functional. The split will be
perfected as other bus attachments for AHCI are written.

Sponsored by: Netflix
Reviewed by: kan, mav
Differential Revision: https://reviews.freebsd.org/D699

We were returning 20 bytes as the FIS size to send, but only
initializing 16. Initialize all 20 so we don't send garbage in the
Auxiliary register. The SATA standard mandates a 5 dword length for
the Host to Device FIS.

Sponsored by: Netflix

Don't bother clearing maps for static DMA allocations to NULL. Instead,
leave them as purely opaque values that are only set by bus_dmamem_alloc().

Correct AMD chipsets identification.

Submitted by: Dmitry Luhtionov <dmitryluhtionov@gmail.com>
MFC after: 2 weeks

Add ID for one more ASMedia AHCI-compatible controller.

Reported by: ignace.peeters@gmail.com
MFC after: 2 weeks

Revert r260165: Proper configuration of unmapped_buf_allowed should be used

To avoid failures in bus_dmamap_sync() on ARM unmapped_buf_allowed should
be set to 0. Hence, ARM-specific changes in AHCI should not be applied.

Use only mapped BIOs on ARM

Using unmapped BIOs causes failure inside bus_dmamap_sync, since
this function requires valid MVA address, which is not present
if mapping is not set up.

Submitted by: Wojciech Macek <wma@semihalf.com>
Obtained from: Semihalf

Do not attach to PCI bridges in AHCI driver

Some vendors use the same VID:PID combination in AHCI and PCI bridge cards

Submitted by: Wojciech Macek <wma@semihalf.com>
Obtained from: Semihalf

Add few more minor parts of DevSleep support from AHCI 1.3.1 proposal.

Add some more IDs for Intel ATA, AHCI and USB controllers.

Merge CAM locking changes from the projects/camlock branch to radically
reduce lock congestion and improve SMP scalability of the SCSI/ATA stack,
preparing the ground for the coming next GEOM direct dispatch support.

Replace big per-SIM locks with bunch of smaller ones:
- per-LUN locks to protect device and peripheral drivers state;
- per-target locks to protect list of LUNs on target;
- per-bus locks to protect reference counting;
- per-send queue locks to protect queue of CCBs to be sent;
- per-done queue locks to protect queue of completed CCBs;
- remaining per-SIM locks now protect only HBA driver internals.

While holding LUN lock it is allowed (while not recommended for performance
reasons) to take SIM lock. The opposite acquisition order is forbidden.
All the other locks are leaf locks, that can be taken anywhere, but should
not be cascaded. Many functions, such as: xpt_action(), xpt_done(),
xpt_async(), xpt_create_path(), etc. are no longer require (but allow) SIM
lock to be held.

To keep compatibility and solve cases where SIM lock can't be dropped, all
xpt_async() calls in addition to xpt_done() calls are queued to completion
threads for async processing in clean environment without SIM lock held.

Instead of single CAM SWI thread, used for commands completion processing
before, use multiple (depending on number of CPUs) threads. Load balanced
between them using "hash" of the device B:T:L address.

HBA drivers that can drop SIM lock during completion processing and have
sufficient number of completion threads to efficiently scale to multiple
CPUs can use new function xpt_done_direct() to avoid extra context switch.
Make ahci(4) driver to use this mechanism depending on hardware setup.

Sponsored by: iXsystems, Inc.
MFC after: 2 months

Support the PCI-Express SSD in the new MacBook Air (model A1465)

Submitted by: Johannes Lundberg <johannes brilliantservice.co.jp>
MFC after: 3 days

Decode some bits defined in AHCI 1.3.1 Device Sleep Technical Proposal.
For now this is only conmetics to report HBA capabilities (Haswell?).

Submitted by: Dmitry Luhtionov <dmitryluhtionov@gmail.com>

Add new Coleto Creek device support: SATA, SMBus, and Watchdog devices.

MFC after: 1 week

Add ID for Marvell 88SE9215 AHCI SATA controller.

MFC after: 3 days

Add one more ID of Marvell SATA controller.

Submitted by: marck
MFC after: 3 days

Added output of device QUIRKS for CAM and AHCI devices during boot.

Reviewed by: mav
Approved by: pjd (mentor)
MFC after: 2 weeks

Add IDs for HighPoint RocketRAID 640L/642L/644L -- new series of 4-port
6Gbps PCIe 2.0 x2 SATA controllers, based on Marvell 88SE9235 chips.

MFC after: 1 week

Create controller-level DMA tag, handling range of supported addresses.
That simplifies logic for channels and gives the bus information about what
device actually allocated the tag.

Submitted by: jhb@

Read Asynchronous Notification statuses only if Port Multiplier or ATAPI
device are connected. ATA disks are not using ANs, while the extra register
read operation is quite expensive.

Depending on combination of running commands (NCQ/non-NCQ) try to avoid
extra read from PxCI/PxSACT registers. If only NCQ commands are running, we
don't really need PxCI. If only non-NCQ commands are running we don't need
PxSACT. Mixed set may happen only on controllers with FIS-based switching
when port multiplier is attached, and then we have to read both registers.

MFC after: 1 month

No need to erase all 64 bytes of CFIS area if we never use more then 16.

ahci(4) and siis(4) are ready to process the unmapped i/o requests

Sponsored by: The FreeBSD Foundation
Tested by: pho
Submitted by: bf (siis patch)

Reform the busdma API so that new types may be added without modifying
every architecture's busdma_machdep.c. It is done by unifying the
bus_dmamap_load_buffer() routines so that they may be called from MI
code. The MD busdma is then given a chance to do any final processing
in the complete() callback.

The cam changes unify the bus_dmamap_load* handling in cam drivers.

The arm and mips implementations are updated to track virtual
addresses for sync(). Previously this was done in a type specific
way. Now it is done in a generic way by recording the list of
virtuals in the map.

Submitted by: jeff (sponsored by EMC/Isilon)
Reviewed by: kan (previous version), scottl,
mjacob (isp(4), no objections for target mode changes)
Discussed with: ian (arm changes)
Tested by: marius (sparc64), mips (jmallet), isci(4) on x86 (jharris),
amd64 (Fabian Keil <freebsd-listen@fabiankeil.de>)

Disable MSI interrupts for SB600 chipset. According to the report they are
not functional.

PR: kern/174880, kern/174985, kern/175002
MFC after: 1 week

Add Intel Lynx Point PCH SATA Controller Device IDs

Add IDs for SATA controllers on AMD Hudson-2 series chipsets.
I am not exactly sure about the naming due to lack of specs on AMD site,
but it is better to have some identification then none at all.

MFC after: 1 month

Switch some PCI register reads from using magic numbers to using the names
defined in pcireg.h

MFC after: 1 week

Fix AHCI 1.2 version checks. This should be mostly cosmetic.

Submitted by: Dmitry Luhtionov <dmitryluhtionov@gmail.com>
MFC after: 1 week

Add IDs for JMicron JMB360/JMB362 AHCI SATA controllers.

MFC after: 1 week

Refactor enclosure manegement support in ahci(4). Move it out into separate
subdevice ahciem. Emulate SEMB SES device from AHCI LED interface to expose
it to users in form of ses(4) CAM device. If we ever see AHCI controllers
supporting SES of SAF-TE over I2C as described by specification, they should
fit well into this new picture.

Sponsored by: iXsystems, Inc.

Partially revert r236666:
Return PROTO_ATA protocol in response to XPT_PATH_INQ.

smartmontools uses it to identify ATA devices and I don't know any other
place now where it is important. It could probably use XPT_GDEV_TYPE
instead for more accurate protocol information, but let it live for now.

Reported by: matthew
MFC after: 3 days

Add IDs for Marvell 88SE9220/9230/9235 PCIe 2.0 x2 6Gbps SATA controllers.
Marvell 88SE9230 was confirmed to work, the rest two are just guessed.

MFC after: 1 week

ATA/SATA controllers have no idea about protocol of the connected device
until transport will do some probe actions (at least soft reset).
Make ATA/SATA SIMs to not report bogus and confusing PROTO_ATA protocol.
Make ATA/SATA transport to fill that gap by reporting protocol to SIM with
XPT_SET_TRAN_SETTINGS and patching XPT_GET_TRAN_SETTINGS results if needed.

Add quirk for Marvell based AHCI controller.

MFC after: 3 days
Suggested by: mav @

Add two functions xpt_batch_start() and xpt_batch_done() to the CAM SIM KPI
to allow drivers to handle request completion directly without passing
them to the CAM SWI thread removing extra context switch.
Modify all ATA/SATA drivers to use them.

Reviewed by: gibbs, ken
MFC after: 2 weeks

Fix names of some Marvell SATA chips. It looks like chips with proprietary
interface supported by mvs(4) are 88SX, while AHCI-like chips are 88SE.

PR: kern/165271
Submitted by: Jia-Shiun Li <jiashiun@gmail.com>
MFC after: 1 week

Add 0x2826 device ID for C600 (Patsburg) SATA controller in RAID mode.

Reviewed by: mav
Approved by: scottl

Add hw.ahci.force tunable to control whether AHCI drivers should attach
to known AHCI-capable chips (AMD/NVIDIA), configured for legacy emulation.

Enabled by default to get additional performance and functionality of AHCI
when it can't be enabled by BIOS. Can be disabled to honor BIOS settings if
needed for some reason.

MFC after: 1 month

Rename device_delete_all_children() into device_delete_children().

Suggested by: jhb @ and marius @
MFC after: 1 week

Move the device_delete_all_children() function from usb_util.c
to kern/subr_bus.c. Simplify this function so that it no longer
depends on malloc() to execute. Identify a few other places where
it makes sense to use device_delete_all_children().

MFC after: 1 week

Change the way how "not implemented" AHCI channels handled. Instead of
completely skipping them, create ahcich devices for them to allocate unit
numbers, but mark them as disabled to prevent driver probe and attach.

Last time some BIOSes tend to report unused channels as "not implemented".
This change makes ahcichX devices numbering consistent, independently of
connected disks. It makes per-channel driver hints usable and CAM devices
wiring possible on such systems.

Mark MALLOC_DEFINEs static that have no corresponding MALLOC_DECLAREs.

This means that their use is restricted to a single C file.

Add one more ID for the Marvell 88SE9128 6Gbps SATA controller.

MFC after: 3 days

Add ID for ASMedia ASM1061 2-port PCIe 2.0 x1 6Gb/s SATA controller.

Approved by: re (blackend)
MFC after: 1 week

Do not force AHCI mode on NVIDIA MCP89 SATA controllers. Recent Apple
Mac with this chipset does not initialize AHCI mode unless it is started
from EFI loader. However, legacy ATA mode works.

Submitted by: jkim@ (original version)
Approved by: re (kib)
MFC after: 1 week

In some cases, at least on Marvell 88SE912x controllers, Current Command
Slot field of the PxCMD register may point to an empty command slot.
That breaks command timeout detection logic, making impossible to find
what command actually caused timeout, and leading to infinite wait.
Workaround that by checking whether pointed command slot is really used
and can timeout in its time. And if not, fallback to the dumb algorithm
used with FBS -- let all commands to time out and then fail all of them.

Approved by: re (kib)
MFC after: 1 week

Add ID for Marvell 88SE9125 SATA controller.

PR: kern/157843
MFC after: 1 week

Add Marvell 88SE9172 chip PCI ID.

Marvell 88SE91xx controllers are known to report soft-reset completion
without waiting for device readiness (or at least not updating FIS receive
area in time). To workaround that, special quirk was added earlier to wait
for the FIS receive area update. But it was found that under same PCI ID
0x91231b4b and revision 0x11 there are two completely different chip
versions (firmware?): HBA and RAID. The problem is that RAID version in
some cases, such as hot-plug, does not update FIS receive area at all!

To workaround that, differentiate the chip versions by their capabilities,
and, if RAID version found, skip FIS receive area update waiting and read
device signature from the PxSIG register instead. This method doesn't work
for HBA version when PMP attached, so keep using previous workaround there.

According to SATA specification, when Serial ATA Enclosure Management Bridge
(SEMB) is unable to communicate to Storage Enclosure Processor (SEP), in
response to hard and soft resets it should among other things return value
0x7F in Status register. The weird side is that it means DRQ bit set, which
tells that reset request is not completed. It would be fine if SEMB was the
only device on port. But if SEMB connected to PMP or built into it, it may
block access to other devices sharing same SATA port.

Make some tunings/fixes to soft-reset handling to workaround the issue:
- ahci(4): request CLO on the port after soft reset to ignore DRQ bit;
- siis(4): gracefully reinitialize port after soft reset timeout (hardware
doesn't detect reset request completion in this case);
- mvs(4): if PMP is used, send dummy soft-reset to the PMP port to make it
clear DRQ bit for us.

For now this makes quirks in ata_pmp.c, hiding SEMB ports of SiI3726/SiI4726
PMPs, less important. Further, if hardware permit, I hope to implement real
SEMB support.

Add support for "LED" enclosure management messages, defined by the AHCI.

When supported by hardware, this allows to control per-port activity, locate
and fault LEDs via the led(4) API for localization and status reporting
purposes. Supporting AHCI controllers may transmit that information to the
backplane controllers via SGPIO interface. Backplane controllers interpret
received statuses in some way (IBPI standard) to report them using present
indicators.

Chipset support for the new Intel Panther Point PCH, thanks
to Seth Heasley for preparing the changes.

Add PCI ID for Marvell 88SE9182 -- PCIe 2.x x2 relative of the 88SE912x.

Submitted by: dchagin
MFC after: 1 week

Fix some English grammar.

According to specification. device should respond to COMRESET with COMINIT
in no more then 10ms. If we detected no device presence within that time,
there is no reason to wait longer.

Properly handle memory allocation errors during error recovery.

Handle ready timeout during polled operation same as done in mvs(4) before.

- Tune different wait loops to cut some more milliseconds from reset time.
- Do not call ahci_start() before device signature received. It is required
by the specification and caused non-fatal reset timeouts on AMD chipsets.

Some changes around hot-plug and interface power-management:
- use ATA_SE_EXCHANGED (SError.DIAG.X) bit to detect hot-plug events when
power-management enabled and ATA_SE_PHY_CHANGED (SError.DIAG.N) can't be
trusted;
- on controllers supporting staggered spin-up (SS) put unused channels
into Listen state instead of Off. It should still save some power, but
allow plug-in events to be detected;
- on controllers supporting cold presence detection (CPD), when power
management enabled, use CPD events to detect hot-plug in addition to PHY
events.

Improve SATA Asynchronous Notification feature support in CAM:
- make SATA SIMs announce capabilities to handle SDB with Notification bit;
- make PMP driver honor this SIMs capability;
- make SATA XPT to negotiate and enable this feature for ATAPI devices.

This feature allows supporting SATA ATAPI devices to inform system about
some events happened, that may require attention. In my case this allows
LG GH22LS50 SATA DVR-RW drive to report tray open/close events. Events
reported to CAM in form of AC_SCSI_AEN async. Further they could be used
as a hints for checking device status and reporting media change to upper
layers, for example, via spoiling mechanism of GEOM.

Refactor hard-reset implementation in ahci(4).

Instead of spinning in a tight loop for up to 15 seconds, polling for device
readiness while it spins up, return reset completion just after PHY reports
"connect well" or 100ms connection timeout. If device was found, use callout
for checking device readiness with 100ms period up to full 31 second timeout.

This fixes system freeze for 5-10 seconds on drives hot plug-in.

Implement automatic SCSI sense fetching for ahci(4).

Add one more ID for Marvell 88SE912x chip found on Asus U3S6 card.

Submitted by: Jonas Jonsson <fatbrain@gmail.com>

Add some more IDs of HighPoint RocketRAID 64x.

Restore DH89xxCC/Patsburg chip IDs accentally dropped at r218596.

Disable NCQ for multiport Marvell 88SX61XX SATA controllers. Simultaneous
active I/O to several disks (copying large file on ZFS) causes timeout after
just a few seconds of run. Single port 88SX6111 seems like not affected.

Skip reading transferred bytes count for these controllers. It works for
88SX6111, but 88SX6145 always returns zero there. Haven't tested others,
but better to be safe.

Support for the new Patsburg PCH chipset:
- SMBus Controller
- SATA Controller
- HD Audio Controller
- Watchdog Controller

Thanks to Seth Heasley (seth.heasley@intel.com) for providing us code.

MFC after 3 days

Support for the new DH89xxCC PCH chipset including:
- SATA controller
- Watchdog timer
- SMBus controller

Add IDs for HighPoint RocketRAID 64x controllers.

These controllers consist of two Marvell 88SE9128 6Gbps SATA chips and
PLX PCIe bridge. As result, they seem to be agree to work with ahci(4)
as usual HBAs. The only noticed issue is that RAID BIOS disables all
drive caches during boot, though `camcontrol cmd ...` is able to fix that.

Those who wants RAID functionality can still use closed proprietary driver
from HighPoint site.

MFC after: 1 week

Add IDs of HighPoint RocketRAID 62x cards (Marvell 88SE9128 chips).

PR: kern/152926
Submitted by: Mike Tancsa <mike@sentex.net>
MFC after: 1 week

Fix small typo.

Submitted by: Artem Belevich

Teach ahci(4), siis(4) and ATA_CAM ata(4) wrapper report to CAM residual
I/O length on underruns, that often happens for some SCSI commands.

Add missing mtx_destroy() on channel attach failure.

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Fix panic, when due to some kind of congestion on FIS-based switching
port multiplier some command triggers false positive timeout, but then
completes normally.

MFC after: 2 weeks

MFata(4):
Add Intel Cougar Point PCH SATA Controller DeviceIDs. Correct some existing
entries for Intel Ibex Peak (5 Series/3400 Series) PCH SATA controllers.

Export PCI IDs of ATA/SATA controllers through CAM and ata(4) layers to
GEOM. This information needed for proper soft-RAID's on-disk metadata
reading and writing.

Disable NCQ and PMP support for VIA VT8251 AHCI. It was reported to be
unreliable under load. Linux does the same.

Plug memory leak to silent Coverity. Error is still not really handled.

Found with: Coverity Prevent(tm)
CID: 4130

Fix attach errors handling.

Found with: Coverity Prevent(tm)
CID: 3424

MFC r208410:
Report ATA/SATA channel number to NewBus at location string.

MFC r208414:
Fill rman range start/end values. It makes devinfo output more readable.

MFC r208375, r208393:
Improve suspend/resume support. Make sure controller is idle on suspend
and reset it on resume.

Fill rman range start/end values. It makes devinfo output more readable.

Report ATA/SATA channel number to NewBus at location string.

Improve suspend/resume support. Make sure controller is idle on suspend
and reset it on resume.

MFC r207511:
Enable PCI busmastering explicitly to be sure.

MFC r207499:
Make SATA XPT negotiate and enable some additional SATA features, such as:
- device initiated power management (some devices support only this way);
- Automatic Partial to Slumber Transition (more power saving);
- DMA auto-activation (expected to slightly improve performance).
More features could be added later, when hardware supports.

MFC r207430:
Add Target/LUN ID checks and deny access to targets 1-14 when PMP absent.
Enforce PMA bit clearing when PMP detached to avoid further scan timeouts.

Enable PCI busmastering explicitly to be sure.

Make SATA XPT negotiate and enable some additional SATA features, such as:
- device initiated power management (some devices support only this way);
- Automatic Partial to Slumber Transition (more power saving);
- DMA auto-activation (expected to slightly improve performance).
More features could be added later, when hardware supports.

Add Target/LUN ID checks and deny access to targets 1-14 when PMP absent.
Enforce PMA bit clearing when PMP detached to avoid further scan timeouts.

Revert r206755. It causes some laptops to stop booting.

Add another ICH7M chipset that works.

MFC after: 1 week

MFC r205422:
- Spec tells that CCC interrupt is edge triggered. Acknowledge it as such.
- Do not try to enable CCC if it is not supported.

- Spec tells that CCC interrupt is edge triggered. Acknowledge it as such.
- Do not try to enable CCC if it is not supported.

MFC r203873:
With FBS enabled, we have no idea what command caused timeout.
Implement same logic as in siis(4) - wait for other commands
complete or timeout and then give some more time.

MFC r203426:
Disable PHY of unconnected ports when interface power management enabled.
It allows to save a bit more power (about 0.5W on 2 unused ports of ICH8M).

MFC r203165:
Reset port on disconnect event, to abort any running requests.

MFC r203123:
Add FIS-based switching support. If controller supports FBS, it allows
several devices beyond Port Multiplier to work simultaneously, substantially
increasing performance.

MFC r203376, r203384:
- Give ATA/SATA SIMs info about ATAPI packet size, supported by device.
- Make ATA XPT to reject longer SCSI CDBs then supported by device, or
any SCSI CDBs, if device doesn't support ATAPI.

MFC r203108:
Large set of CAM improvements:
- Unify bus reset/probe sequence. Whenever bus attached at boot or later,
CAM will automatically reset and scan it. It allows to remove duplicate
code from many drivers.
- Any bus, attached before CAM completed it's boot-time initialization,
will equally join to the process, delaying boot if needed.
- New kern.cam.boot_delay loader tunable should help controllers that
are still unable to register their buses in time (such as slow USB/
PCCard/ CardBus devices), by adding one more event to wait on boot.
- To allow synchronization between different CAM levels, concept of
requests priorities was extended. Priorities now split between several
"run levels". Device can be freezed at specified level, allowing higher
priority requests to pass. For example, no payload requests allowed,
until PMP driver enable port. ATA XPT negotiate transfer parameters,
periph driver configure caching and so on.
- Frozen requests are no more counted by request allocation scheduler.
It fixes deadlocks, when frozen low priority payload requests occupying
slots, required by higher levels to manage theit execution.
- Two last changes were holding proper ATA reinitialization and error
recovery implementation. Now it is done: SATA controllers and Port
Multipliers now implement automatic hot-plug and should correctly
recover from timeouts and bus resets.
- Improve SCSI error recovery for devices on buses without automatic sense
reporting, such as ATAPI or USB. For example, it allows CAM to wait, while
CD drive loads disk, instead of immediately return error status.
- Decapitalize diagnostic messages and make them more readable and sensible.
- Teach PMP driver to limit maximum speed on fan-out ports.
- Make boot wait for PMP scan completes, and make rescan more reliable.
- Fix pass driver, to return CCB to user level in case of error.
- Increase number of retries in cd driver, as device may return several UAs.

MFC r203030:
Add support for SATA part of Marvell 88SE912x controllers to ahci(4).
Limit early revisions from 6Gb/s to 3Gb/s by default, or they negotiate
only 1.5Gb/s, when 3Gb/s devices connected.

Add dummy driver for PATA part of these controllers, preventing generic
driver attach them. It causes system freeze when SATA controller used after
PATA was touched.

MFp4:
With FBS enabled, we have no idea what command caused timeout.
Implement same logic as in siis(4) - wait for other commands
complete or timeout and then give some more time.

Disable PHY of unconnected ports when interface power management enabled.
It allows to save a bit more power (about 0.5W on 2 unused ports of ICH8M).

- Give ATA/SATA SIMs info about ATAPI packet size, supported by device.
- Make ATA XPT to reject longer SCSI CDBs then supported by device, or
any SCSI CDBs, if device doesn't support ATAPI.

Reset port on disconnect event, to abort any running requests.

Add FIS-based switching support. If controller supports FBS, it allows
several devices beyond Port Multiplier to work simultaneously, substantially
increasing performance.

MFp4: Large set of CAM inprovements.

- Unify bus reset/probe sequence. Whenever bus attached at boot or later,
CAM will automatically reset and scan it. It allows to remove duplicate
code from many drivers.
- Any bus, attached before CAM completed it's boot-time initialization,
will equally join to the process, delaying boot if needed.
- New kern.cam.boot_delay loader tunable should help controllers that
are still unable to register their buses in time (such as slow USB/
PCCard/ CardBus devices), by adding one more event to wait on boot.
- To allow synchronization between different CAM levels, concept of
requests priorities was extended. Priorities now split between several
"run levels". Device can be freezed at specified level, allowing higher
priority requests to pass. For example, no payload requests allowed,
until PMP driver enable port. ATA XPT negotiate transfer parameters,
periph driver configure caching and so on.
- Frozen requests are no more counted by request allocation scheduler.
It fixes deadlocks, when frozen low priority payload requests occupying
slots, required by higher levels to manage theit execution.
- Two last changes were holding proper ATA reinitialization and error
recovery implementation. Now it is done: SATA controllers and Port
Multipliers now implement automatic hot-plug and should correctly
recover from timeouts and bus resets.
- Improve SCSI error recovery for devices on buses without automatic sense
reporting, such as ATAPI or USB. For example, it allows CAM to wait, while
CD drive loads disk, instead of immediately return error status.
- Decapitalize diagnostic messages and make them more readable and sensible.
- Teach PMP driver to limit maximum speed on fan-out ports.
- Make boot wait for PMP scan completes, and make rescan more reliable.
- Fix pass driver, to return CCB to user level in case of error.
- Increase number of retries in cd driver, as device may return several UAs.

Add support for SATA part of Marvell 88SE912x controllers to ahci(4).
Limit early revisions from 6Gb/s to 3Gb/s by default, or they negotiate
only 1.5Gbps, when 3Gb/s devices connected.

Add dummy driver for PATA part of these controllers, preventing generic
driver attach them. It causes system freeze when SATA controller used after
PATA was touched.

MFC r200814:
Clear all ports interrupt status bits in single write. Clearing one by one
causes additional MSIs messages sent if several ports asked for attention
same time. Time window before clearing is not important, as these interrupts
are level triggered by interrupt source.

Partially revert r202428. There is no bus_describe_intr() on RELENG_8.

MFC r202011:
While AHCI specification tells that multi-vector MSI doesn't use global IS
register, nVidia chipsets have different oppinion, requiring every interrupt
to be acknowledged there.

While there, add interrupt descriptions in multi-vector MSI mode.

While AHCI specification tells that multi-vector MSI doesn't use global IS
register, nVidia chipsets have different oppinion, requiring every interrupt
to be acknowledged there.

While there, add interrupt descriptions in multi-vector MSI mode.

MFC 200977:
Avoid false positive probe on ICH6 chipsets.

Avoid false positive probe on ICH6 chipsets.

Clear all ports interrupt status bits in single write. Clearing one by one
causes additional MSIs messages sent if several ports asked for attention
same time. Time window before clearing is not important, as these interrupts
are level triggered by interrupt source.

MFC r200196:
Add Asynchronous Notification support for controllers without SNTF
capability by snooping SDB FIS receive area. It should be even faster
then regular way, but less reliable.

Add Asynchronous Notification support for controllers without SNTF
capability by snooping SDB FIS receive area. It should be even faster
then regular way, but less reliable.

MFC r199747, r199799, r199821:
- Extend XPT-SIM transfer settings control API. Now it allows to report to
SATA SIM number of tags supported by each device, implement ATA mode and
SATA revision negotiation for both SATA and PATA SIMs.
- Make ahci(4) and siis(4) to use submitted maximum tag number, when
scheduling requests. It allows to support NCQ on devices with lower tags
count then controller supports.
- Make PMP driver to report attached devices connection speeds.
- Implement ATA mode negotiation between user settings, device and
controller capabilities.
- Improve ATA mode/SATA revision control.

MFp4:
Improve ATA mode/SATA revision control.

MFC r199717:
Do not attach JMicrons with single PCI function. They are not working as
AHCI for some reason, even when declaring so. Let atajmicron configure
them for us and provide PATA support.

MFp4:
- Extend XPT-SIM transfer settings control API. Now it allows to report to
SATA SIM number of tags supported by each device, implement ATA mode and
SATA revision negotiation for both SATA and PATA SIMs.
- Make ahci(4) and siis(4) to use submitted maximum tag number, when
scheduling requests. It allows to support NCQ on devices with lower tags
count then controller supports.
- Make PMP driver to report attached devices connection speeds.
- Implement ATA mode negotiation between user settings, device and
controller capabilities.

Do not attach JMicrons with single PCI function. They are not working as
AHCI for some reason, even when declaring so. Let atajmicron configure
them for us and provide PATA support.

MFC r199259, r199262, r199322:
Change the way in which AHCI+PATA combined controllers, such as JMicron
are handled. Instead of trying to attach two different drivers to
single device, wrapping each call, make one of them (atajmicron)
attach do device solely, but create child device for AHCI driver,
passing it all required resources. It is quite easy, as none of
resources are shared, except IRQ.
Add support for AHCI SATA parts of alike SATA+PATA MArvell controllers.
Add IDs of Marvell 88SX6102, 88SX6111. 88SX6141 controllers.

As result, it:
- makes drivers operation more independent and straitforward,
- allows to use new ahci(4) driver with such devices, adding support for
new features, such as PMP and NCQ, same time keeping legacy PATA support,
- will allow to just drop old ataahci driver, when it's time come.

MFC r199278:
Check SNCQ HBA capability bit when reporting NCQ support to CAM.

MFC r199178:
- Move tagged queueing control from ADA to ATA XPT. It allows to control
device command queue length correctly. First step to support < 32 tags.
- Limit queue for non-tagged devices by 2 slots for ahci(4) and siis(4).
- Implement quirk matching for ATA devices.
- Move xpt_schedule_dev_sendq() from header to source file.
- Move delayed queue shrinking to the more expected place - element freeing.
- Remove some SCSIsms in ATA.

MFC r199176:
Add set of chip IDs, known to support AHCI.

MFC r198851:
- Handle timeouts and fatal errors with port hard-reset. The rest of
recovery will be done by XPT on receiving async event. More gracefull
per-device soft-reset recovery can be implemented later.
- Add workaround for ATI SB600/SB700 PMP probe related bug, to speedup boot.

MFC r198390:
Revert interrupt reason check order back.
ATAPI errors may set IF bit together with TFE.

MFC r198319:
On error, freeze device queue, to allow periph driver to do proper recovery.
Freeze SIM queue only in some cases, when it is needed to protect SIM.

Implement better command timeout detection logic for non-queued commands.
This fixes false positives when command with short timeout waiting for the
long one. For example, when hald tastes CD during burning process.

Read and clear SERR register on interrupt.

MFC r197838:
On command timeout handle frozen command first, to not run it inside
XXX_end_transaction().

MFC r198322:
Report real max_target = 15. SIM doesn't need to know that target 15 is PMP.
It is XPT business.

MFC r196907:
To save small bit of CPU time, hide part of SNTF register read latency
behind other reads.

Change the way in which AHCI+PATA combined controllers, such as JMicron
and Marvell handled. Instead of trying to attach two different drivers to
single device, wrapping each call, make one of them (atajmicron, atamarvell)
attach do device solely, but create child device for AHCI driver,
passing it all required resources. It is quite easy, as none of
resources are shared, except IRQ.

As result, it:
- makes drivers operation more independent and straitforward,
- allows to use new ahci(4) driver with such devices, adding support for
new features, such as PMP and NCQ, same time keeping legacy PATA support,
- will allow to just drop old ataahci driver, when it's time come.

MFp4:
Check SNCQ HBA capability bit when reporting NCQ support to CAM.

MFp4:
- Move tagged queueing control from ADA to ATA XPT. It allows to control
device command queue length correctly. First step to support < 32 tags.
- Limit queue for non-tagged devices by 2 slots for ahci(4) and siis(4).
- Implement quirk matching for ATA devices.
- Move xpt_schedule_dev_sendq() from header to source file.
- Move delayed queue shrinking to the more expected place - element freeing.
- Remove some SCSIsms in ATA.

MFp4:
Add set of chip IDs, known to support AHCI.

MFp4:
- Handle timeouts and fatal errors with port hard-reset. The rest of
recovery will be done by XPT on receiving async event. More gracefull
per-device soft-reset recovery can be implemented later.
- Add workaround for ATI SB600/SB700 PMP probe related bug, to speedup boot.

Revert interrupt reason check order back.
ATAPI errors may set IF bit together with TFE.

MFp4:
Report real max_target = 15. SIM doesn't need to know that target 15 is PMP.
It is XPT business.

MFp4:
On error, freeze device queue, to allow periph driver to do proper recovery.
Freeze SIM queue only in some cases, when it is needed to protect SIM.

Implement better command timeout detection logic for non-queued commands.
This fixes false positives when command with short timeout waiting for the
long one. For example, when hald tastes CD during burning process.

Read and clear SERR register on interrupt.

On command timeout handle frozen command first, to not run it inside
XXX_end_transaction().

Submitted by: avg

To save small bit of CPU time, hide part of SNTF register read latency
behind other reads.

MFC r196777, r196796:
ATI SB600 can't handle 256 sectors transfers with FPDMA (NCQ).

Approved by: re (ATA-CAM blanket)

Round maxio for ATI SB600 to 64K.

Submitted by: scottl@

ATI SB600 can't handle 256 sectors transfers with FPDMA (NCQ).

MFC after: 3 days

MFC r196656, r196660:
Update ahci driver:
- Add Command Completion Coalescing support.
- Add SNTF support.
- Add two more power management modes (4, 5), implemented on driver level.
- Fix interface mode setting.
- Reduce interface reset time.
- Do not report meaningless protocol/transport versions.
- Report CAP2 register content.
- Some performance optimizations.

Approved by: re (ATA-CAM blanket)

Fix build with INVARIANTS.

MFp4:
- Add Command Completion Coalescing support.
- Add SNTF support.
- Add two more power management modes (4, 5), implemented on driver level.
- Fix interface mode setting.
- Reduce interface reset time.
- Do not report meaningless protocol/transport versions.
- Report CAP2 register content.
- Some performance optimizations.

Separate the parallel scsi knowledge out of the core of the XPT, and
modularize it so that new transports can be created.

Add a transport for SATA

Add a periph+protocol layer for ATA

Add a driver for AHCI-compliant hardware.

Add a maxio field to CAM so that drivers can advertise their max
I/O capability. Modify various drivers so that they are insulated
from the value of MAXPHYS.

The new ATA/SATA code supports AHCI-compliant hardware, and will override
the classic ATA driver if it is loaded as a module at boot time or compiled
into the kernel. The stack now support NCQ (tagged queueing) for increased
performance on modern SATA drives. It also supports port multipliers.

ATA drives are accessed via 'ada' device nodes. ATAPI drives are
accessed via 'cd' device nodes. They can all be enumerated and manipulated
via camcontrol, just like SCSI drives. SCSI commands are not translated to
their ATA equivalents; ATA native commands are used throughout the entire
stack, including camcontrol. See the camcontrol manpage for further
details. Testing this code may require that you update your fstab, and
possibly modify your BIOS to enable AHCI functionality, if available.

This code is very experimental at the moment. The userland ABI/API has
changed, so applications will need to be recompiled. It may change
further in the near future. The 'ada' device name may also change as
more infrastructure is completed in this project. The goal is to
eventually put all CAM busses and devices until newbus, allowing for
interesting topology and management options.

Few functional changes will be seen with existing SCSI/SAS/FC drivers,
though the userland ABI has still changed. In the future, transports
specific modules for SAS and FC may appear in order to better support
the topologies and capabilities of these technologies.

The modularization of CAM and the addition of the ATA/SATA modules is
meant to break CAM out of the mold of being specific to SCSI, letting it
grow to be a framework for arbitrary transports and protocols. It also
allows drivers to be written to support discrete hardware without
jeopardizing the stability of non-related hardware. While only an AHCI
driver is provided now, a Silicon Image driver is also in the works.
Drivers for ICH1-4, ICH5-6, PIIX, classic IDE, and any other hardware
is possible and encouraged. Help with new transports is also encouraged.

Submitted by: scottl, mav
Approved by: re