MFC r316653: Fix few minor issues found by Clang Analyzer.

MFC r315084: Increase device openings to tagged maximum.

Some SIMs report much less untagged device openings then tagged ones.
Target mode devices are not handled by regular probing routines, and so
there is nothing to increase queue size for them to the SIM's maximum.
To fix that resize the queue explicitly on ctl periph registration.
This radically improves performance of mpt(4) in target mode.

Also fetch and report device queue statistics in `ctladm dumpstructs`,
since regular way of `camcontrol tags` is not usable in target mode.

MFC r315022: Request change of SIM target role only when it is different.

Separate WWNs change into separate request to know what actually failed.

MFC r315030: Abort all ATIOs and INOTs queued to SIM on LUN disable.

Some SIMs may not abort them implicitly, that either fail the LUN disable
request or just make us wait for those CCBs forever. With this change
I can successfully disable LUNs on mpt(4). For isp(4), which aborts them
implicitly, this change should be irrelevant.

MFC r315025: Switch work_queue from TAILQ to STAILQ.

It is mostly FIFO and we don't need random removal there.

MFC r311305 (by asomers):
Always null-terminate ccb_pathinq.(sim_vid|hba_vid|dev_name)

The sim_vid, hba_vid, and dev_name fields of struct ccb_pathinq are
fixed-length strings. AFAICT the only place they're read is in
sbin/camcontrol/camcontrol.c, which assumes they'll be null-terminated.
However, the kernel doesn't null-terminate them. A bunch of copy-pasted code
uses strncpy to write them, and doesn't guarantee null-termination. For at
least 4 drivers (mpr, mps, ciss, and hyperv), the hba_vid field actually
overflows. You can see the result by doing "camcontrol negotiate da0 -v".

This change null-terminates those fields everywhere they're set in the
kernel. It also shortens a few strings to ensure they'll fit within the
16-character field.

PR: 215474
Reported by: Coverity
CID: 1009997 1010000 1010001 1010002 1010003 1010004 1010005
CID: 1331519 1010006 1215097 1010007 1288967 1010008 1306000
CID: 1211924 1010009 1010010 1010011 1010012 1010013 1010014
CID: 1147190 1010017 1010016 1010018 1216435 1010020 1010021
CID: 1010022 1009666 1018185 1010023 1010025 1010026 1010027
CID: 1010028 1010029 1010030 1010031 1010033 1018186 1018187
CID: 1010035 1010036 1010042 1010041 1010040 1010039

MFC r314307: Add support for SIMs without autosense.

If we asked to send sense data by setting CAM_SEND_SENSE, but SIM didn't
confirm transmission by setting CAM_SENT_SENSE, assume it was not sent.
Queue the I/O back to CTL for later REQUEST SENSE with ctl_queue_sense().
This is needed for error reporting on SPI HBAs like ahc(4)/ahd(4).

MFC r314338: Polish handling of different reset flavours.

The biggest change is that ctl_remove_initiator() now generates I_T NEXUS
LOSS event, cleaning part of LUs state related to the initiator.

MFC r314496: Add check missed in r314257.

MFC r314387: Make ctl_queue_sense() not sleep.

It may be called in non-sleepable frontend context.

MFC r314299, r314300: Fix residual length reporting in target mode.

This allows to properly handle cases when target wants to receive or send
more data then initiator wants to send or receive. Previously in such
cases isp(4) returned CAM_DATA_RUN_ERR, while now it returns resid > 0.

MFC r314257: Add reporting SAS protocol, in case we ever have one.

MFC r314255: Reenable CTL_WITH_CA, optimizing it for lower memory usage.

This code was disabled due to its high memory usage. But now we need this
functionality for cfumass(4) frontend, since USB MS BBB transport does not
support autosense.

MFC r314247: Axe out some forever disabled questionable functionality.

This code is complicated enough even in its base shape.

MFC r314246: Improve CAM target frontend reference counting.

Before this change it was possible to trigger some use-after-free panics
by disabling LUNs/ports under heavy load.

MFC r314204: Explicitly abort ATIO if CTIO sending status has failed.

This helps SIM to free related resources in questionable cases.

MFC r314200: We can't access periph after ctlfe_free_ccb().

MFC r314196: Unify ATIO/INOT CCBs requeuing.

MFC r314193: Some code cleanup.

MFC r314027: Do not blindly free completed ATIOs/INOTs on invalidation.

When LUN is disabled, SIM starts returning queued ATIOs/INOTs. But at the
same time there can be some ATIOs/INOTs still carrying real new requests.
If we free those, SIM may leak some resources, forever expecting for any
response from us. So try to be careful, separating ATIOs/INOTs carrying
requests which still must be processed, from ATIOs/INOTs completed with
errors which can be freed.

MFC r313744: No need to erase sense_data when sense_len is set to zero.

MFC r313910: Change XCOPY memory allocations.

Before this change XCOPY code could allocate memory in chunks up to 16-32MB
(VMware does XCOPY in 4MB chunks by default), that could be difficult for
VM subsystem to do due to KVA fragmentation, that sometimes created huge
allocation delays, blocking any I/O for respective LU for that time.

This change limits allocations down to TPC_MAX_IO_SIZE, which is 1MB now.
1MB is also not a cookie, but ZFS also can do that for large blocks, so
it should be less dramatic. As drawback this increases CPU overhead, but
it still look acceptable comparing to time consumed by ZFS read/write.

MFC r313736: Fix panic on shutdown of ramdisk LU with zero capacity.

MFC 313895:

------------------------------------------------------------------------
r313895 | ken | 2017-02-17 13:15:27 -0700 (Fri, 17 Feb 2017) | 9 lines

Make ctl(4) build with CTL_IO_DELAY defined.

sys/cam/ctl/ctl.c:
In ctl_datamove(), inside CTL_IO_DELAY, add a lun variable and fill
it in before trying to dereference it.

Sponsored by: Spectra Logic

------------------------------------------------------------------------

MFC r312694: Make CTL ramdisk backend a real RAM disk.

If "capacity" LU option is set, ramdisk backend now implements featured
thin provisioned disk, storing data in malloc(9) allocated memory blocks
of pblocksize bytes (default PAGE_SIZE or 4KB). Additionally ~0.2% of LU
size is used for indirection tree (bigger pblocksize reduce the overhead).
Backend supports all unmap and anchor operations. If configured capacity
is overflowed, proper error conditions are reported.

If "capacity" LU option is not set, the backend operates mostly the same
as before without allocating real storage: writes go to nowhere, reads
return zeroes, reporting that all LBAs are unmapped.

This backend is still mostly oriented on testing and benchmarking (it is
still a volatile RAM disk), but now it should allow to run real FS tests,
not only simple dumb dd.

MFC r312603: Add initial support for CTL module unloading.

It is only a first step and not perfect, but better then nothing.
The main blocker is CAM target frontend, that can not be unloaded,
since CAM does not have mechanism to unregister periph driver now.

MFC r312348: Remove writing 'residual' field of struct ctl_scsiio.

This field has no practical use and never readed. Initiators already
receive respective residual size from frontends. Removed field had
different semantics, which looks useless, and was never passed through
by any frontend.

While there, fix kern_data_resid field support in case of HA, missed in
r312291.

MFC r312291, r312669:
Make CTL frontends report kern_data_resid for under-/overruns.

It seems like kern_data_resid was never really implemented. This change
finally does it. Now frontends update this field while transferring data,
while CTL/backends getting it can more flexibly handle the result.
At this point behavior should not change significantly, still reporting
errors on write overrun, but that may be changed later, if we decide so.

CAM target frontend still does not properly handle overruns due to CAM API
limitations. We may need to add some fields to struct ccb_accept_tio to
pass information about initiator requested transfer size(s).

MFC r312343: Improve error message on duplicate iSCSI port.

MFC r312232: Add under-/overrun support to IOCTL and CAM SIM frontends.

MFC r312231: When in kernel, map ctl_scsi_zero_io() to ctl_zero_io().

MFC r312026: Improve CAM_CDB_POINTER support.

MFC r311873: Fix malloc(M_WAITOK) under mutex, introduced at r311787.

MFC r311804: Rewrite CTL statistics in more simple and scalable way.

Instead of collecting statistics for each combination of ports and logical
units, that consumed ~45KB per LU with present number of ports, collect
separate statistics for every port and every logical unit separately, that
consume only 176 bytes per each single LU/port. This reduces struct
ctl_lun size down to just 6KB.

Also new IOCTL API/ABI does not hardcode number of LUs/ports, and should
allow handling of very large quantities.

Old API is still enabled in stable branches for compatibility reasons.

MFC r311787: Allocate memory for prevent flags only for removable LUs.

This array takes 64KB of RAM now, that was more then half of struct ctl_lun
size. If at some point we support more ports, this may need another tune.

MFC r311680: Make CTL_GETSTATS ioctl return partial data if buffer is small.

MFC r310778, r310782: Improve use of I/O's private area.

- Since I/Os are allocates from per-port pools, make allocations store
pointer to CTL softc there, and use it where needed instead of global.
- Created bunch of helper macros to access LUN, port and CTL softc.

MFC r311892: Do not wait for HA thread shutdown if scheduler is stopped.

This wait loop made system hang on panic instead of reboot.

MFC r310649: Allow more efficient use of private area.

There are 16 bytes of space, so we may store two pointers in one.

MFC r310646: Do not update "saved" mode page on every MODE SELECT.

We do not have non-volatile memory to really save those values, so we
neither report nor support this capability. Also saved mode pages are
not replicated between HA peers now.

MFC r310644: Fix/synchronize field types in struct ctl_modepage_header.

MFC r310640, r310643:
Add support for revert to defaults (RTD) bit in MODE SELECT.

MFC r310635: Decouple limits on number of LUNs per port and LUs per CTL.

Those two values are not directly related, so make them independent.
This does not change any limits immediately, but makes number of LUNs
per port controllable via tunable/sysctl kern.cam.ctl.lun_map_size.
After this change increasing CTL_MAX_LUNS should be pretty cheap,
and even making it tunable should be easy.

MFC r310575: Fix improperly used nexus.targ_lun.

MFC r310555: Some random code cleaning.

- Reduce indentation.
- Remove extra braces.
- Add few missing savety checks.

MFC r310539: Remove CTL_MAX_LUNS from places where it is not required.

MFC r311446: Fix bootverbose affecting code logic in r294558.

Reported by: Jilles Tjoelker <jilles@stack.nl>

MFC r310534: Improve third-party copy error reporting.

For EXTENDED COPY:
- improve parameters checking to report some errors before copy start;
- forward sense data from copy target as descriptor in case of error;
- report which CSCD reported error in sense key specific information.
For WRITE USING TOKEN:
- pass through real sense data from copy target instead of reporting
our copy error, since for initiator its a "simple" write, not a copy.

MFC r310524: Improve length handling when writing sense data.

- Allow maximal sense size limitation via Control Extension mode page.
- When sense size limited, include descriptors atomically: whole or none.
- Set new SDAT_OVFL bit if some descriptors don't fit the limit.
- Report real written sense length instead of static maximal 252 bytes.

MFC r310478: Add place-holders for TAPE STREAM MIRRORING subcommands of XCOPY.

We do not support tapes, so this is only a comments.

MFC r310390: Add support for REPD bit in RSTMF command.

We have no real timeout values to report there, but its better then error.

MFC r310389:
Fix REPORT SUPPORTED OPERATION CODES for READ/WRITE BUFFER commands.

Specifications require MODE parameter of those commands to be treated
the same as SERVICE ACTION parameter of other commands.

MFC r310373:
Add support for REPORTING OPTIONS == 3 in REPORT SUPPORTED OPERATION CODES.

MFC r310366: Add support for SITUA bit in Logical Block Provisioning mode page.

VMware tries to enable this bit to avoid multiple threshold notifications
in case of multiple initiators connected to the same LUN. Unfortunately
their code sends MODE SELECT(6) request with parameter length hardcoded
for the page without any thresholds. Since we have four threshold and our
page is bigger, this attempt fails, that is correct in my understanding.
So all we can do about this now is to report proper error code and hope
VMware fix their code one day.

MFC r310356: Add support for locally assigned RFC 4122 UUID LUN identifiers.

MFC r310339: Bump specifications support to SAM-6/SPC-5.

MFC r310298: Improve error handling when I/O split between several BIOs.

If we get several error codes, handle one with lowest offset.

MFC r310285:
When reporting "Logical block address out of range" error, report the LBA
in sense data INFORMATION field.

MFC r310275: Fix typo in function name.

MFC r310272: Add new bits into Extended Inquiry VPD page.

MFC r310266: Add support for NUAR bit in Control mode page.

MFC r310265: Add set of macros to simplify code access to mode pages fields.

MFC r310259:
Following SPC-5, make REQUEST SENSE report "Logical unit not supported"
in returned parameter data for not accessible LUNs.

MFC r310257: Improve support for informational exceptions.

While CTL still has no real events to report in this way (like SMART),
it is possible to trigger false event by manually setting TEST bit in
Informational Exceptions Control mode page, that can be useful for
initiator testing. This code supports all flavours of IE reporting:
UNIT ATTENTION, RECOVERED ERROR and NO SENSE sense keys, REQUEST SENSE
command and Informational Exceptions log page.

MFC r298810 (by pfg): sys/cam: spelling fixes in comments.

No functional change.

MFC r295476 (by trasz): Remove stray semicolons from the iSCSI code.

MFC r294558: Hide "soconnect() error" messages under bootverbose.

They can be too noisy.

MFC r307374: Add LU option to control reported provisioning type.

MFC r307350: Add LUN options to limit UNMAP and WRITE SAME sizes.

CTL itself has no limits on on UNMAP and WRITE SAME sizes. But depending
on backends large requests may take too much time. To avoid that new
configuration options allow to hint initiator maximal sizes it should not
exceed.

Fix HA mode configuration on FreeBSD 10.x.

This is direct commit, compensating CTLFLAG_RDTUN difference between
FreeBSD 10.x and 11.x branches.

MFC r300293:
Pass task management response information from CTL through CAM to isp(4),
utilizing previously unused arg field of struct ccb_notify_acknowledge.

This makes new QUERY TASK, QUERY TASK SET and QUERY ASYNC EVENT requests
really functional for CAM target mode drivers.

MFC r292384:

Fix style issues around existing SDT probes.

** Changes to sys/netinet/in_kdtrace.c and sys/netinet/in_kdtrace.h skipped.

MFC r299347, r299348: Validate XCOPY range offsets and lengths.

MFC r299346: More XCOPY parameters validation.

MFC r299329: Improve validation of some POPULATE TOKEN parameters.

MFC r298703:
cam: unsign some types to match their definitions and avoid overflows.

numpatterns is u_int.

ctl:
CTL_NUM_MODE_PAGES comes from sizeof().
In struct:ctl_scsiio, kern_sg_entries is uint32_t.

MFC r298279

Plug memory leak in ctl(4) when ctl_copyin_args() is called with a non-
null terminated ASCII string.

PR: 207626
Submitted by: cturt@hardenedbsd.org

MFC r296392: Set bhsdi_target_transfer_tag to reserved value, which is
0xffffffff.

This should be a purely cosmetic change.

MFC r293350:
Convert sys/cam to use make_dev_s().

MFC r292290: Set DS flag, required for LPB log page by spec.

MFC r291383: Fix panic when trying to sort unsupported command in OOA queue.

Handle unsupported commands as not conflicting/blocking.

MFC r290820: Add NULL check to make Coverity happy.

MFC r290670: Modify target port groups logic in CTL.

- Introduce "ha_shared" port option, which being set to "on" moves the
port into separate port group, shared between HA nodes. This allows to
better handle cases when iSCSI portals are bound to CARP address that can
dynamically move between nodes. Some initiators (at least VMware) don't
detect that after iSCSI reconnect they've attached to different SCSI port
from different port group, that totally breakes ALUA status parsing.
In theory, I believe, it should be enough to have different iSCSI portal
group tags on different nodes to make initiators detect this condition,
but it seems like VMware ignores those values, and even full LUN retaste
forced by UA does not help.
- Make CTL report up to three port groups: 1 -- non-HA mode or ports
with "ha_shared" option set, 2 -- HA node 1, 3 -- HA node 2.
- Report Transitioning state for all port groups when HA interlink is
connected, but neither of nodes is primary for the LUN.

MFC r290458: Add two more KASSERTs.

MFC r289881: Give CTL support for PIM_EXTLUNS when talking to CAM.

CTL itself still lives in flat LUN space, but it can generate extended
numbers if CAM SIM reports such capability.

MFC r289843: Add partial support for QUERY TMF to CAM and isp(4).

This change allows to decode respective functions in isp(4) in target mode
and pass them through CAM to CTL. Unfortunately neither CAM nor isp(4)
support returning response info for those task management functions now.

On the other side I just have no initiator to test this functionality.

MFC r289702: Make some panic strings mode informative.

MFC r289819: Fix LUN disable in CAM broken at r285155.

MFC r290004: Don't try to replicate mode pages not present on this device.

MFC r289136: Remove lock upgrade attempt from ctl_be_block_open_file().

I am not sure what for it was done. Now open routine should automatically
fall back to read-only if open for writing is impossible. In such case
attempt to upgrade to write sounds strange.

MFC r289017: Add missing vnode lock in case of file modify request.

Submitted by: Richard Kojedzinszky

MFC r288458: More aggressively fill WUT read pipeline.

On some tests I've measured 5% copy speedup from this.

MFC r288450: Make zero WUT use WRITE SAME with recently allowed NDOB flag.

MFC r288449:
Implement SPC-3 exceptions to SPC-2 RESERVE and RELEASE behavior.

MFC r288448: Unify PR variable names to reduce confusion.

MFC r288427: Use proper STAILQ_* macros where possible.

MFC r288369: Really implement PREVENT ALLOW MEDIUM REMOVAL command.

MFC r288368: Don't report SYNC_NV bit set in SYNCHRONIZE CACHE as error.

While this bit is obsolete in SBC-3, behavior controlled by it is allowed
on device discretion.

MFC r288367: Fix arguments order.

MFC r288359: Report that we can read all flavours of DVD. Why not?

MFC r288358: Add CD/DVD Capabilities and Mechanical Status Page.

This page is obsolete since MMC-4, but still used by some software.

approved by:

MFC r288348: Umplement media load/eject support for removable devices.

In case of block backend eject really closes the backing store, while
load tries to open it back. Failed store open is reported as no media.

MFC r288310: Add to CTL initial support for CDROMs and removable devices.

Relnotes: yes

MFC r288264: Allow LOG SENSE command on non-disk devices.

MFC r288262: Remove concept of control device.

MFC r288261: Move ioctl frontend defines where they belong.

MFC r288260: Remove few more unused variables.

MFC r288259: Remove some duplicate, legacy, dead and questionable code.

MFC r288247: Pass to sobind() copy of the address, not the original.

MFC r288239: Properly lock LUN in ctl_failover_lun().

MFC r288224: Constify ctl_serialize_table.

MFC r288229: Fix bit masks changed for the wrong commands in r288221.

MFC r288221: Remove some dead code found by Clang analyzer.

MFC r288220: Remove stale comments and some excessive empty lines.

MFC r288215: Switch I/O time accounting from system time to uptime.

While there, make num_dmas accounted independently of CTL_TIME_IO.

MFC r288214: Collect DMA statistics on secondary HA node.

MFC r288213:
Make HA handle datamove and done in a universal way, using port methods.

Now from primary node point of view requests transferred from secondary
node should look almost normal and always have valid port.

MFC r288211: Remove some control_softc references.

MFC r288175: Allow WRITE SAME with NDOB bit set but without UNMAP.

This combination was originally forbidden, but allowed at spc4r3.

MFC r288170: Add new report types to REPORT LUNS command.

This is only for completeness, since we have nothing new to report there.

MFC r288166: Update WRITE ATOMIC(16) support to sbc4r8 draft.

This is only a cosmetic change. We still don't support atomic boundary
field in the CDB, but at least now we do it formally.

MFC r288165: Add support for READ BUFFER(16) command.

MFC r288148: Synchronize mode pages between HA peers.

We allow to modify only few fields in mode pages now, but still it is
not good if they unexpectedly change during failover. Also this fixes
reporting of "Mode parameters changed" UAs on secondary node.

MFC r288146: Make HA peers announce their parameters on connect.

HA protocol requires strict version, parameters and configuration match.
Differences there may cause full set of problems up to kernel panic.
To avoid that, validate peer parameters on connect, and abort connection
immediately if some mismatch detected.

MFC r288110: Add support for Control extension mode page.

MFC r288104: Remove duplicate and incomplete code handling LUN modify.

Instead reuse code from LUN creation. This allows most of LUN media
options to be changed live with modify request without full restart.

MFC r288067: Log iSCSI session reinstatements.

False session reinstatements can be result of misconfiguration, when
several initiators use the same initiator name and ISID.

MFC r288021: Mark with DMA flag I/Os waiting for iSCSI write data after R2T.

Reads and immediate writes are not blocking, so don't bother.

MFC r288020: Remove couple excess SGLIST I/O flags.

Those flags duplicated respective (sg_entries > 0) values.

MFC r287994: Always execute REPORT LUNS and REQUEST SENSE commands locally.

REPORT LUNS command is more related to target rather then specific LUN.
This node may be primary for LUNs for some reason unknown to another,
and command forwarded to another node won't be able to report them.

REQUEST SENSE is related to LUN, but in our implementation it reports
only UAs and CAs, that are stored locally rather then on primary node.

MFC r287993: Split two command flags with different meaning.

This is only a cosmetical change.

MFC r287991: Pack struct ctl_ha_msg_hdr by 8 bytes.

MFC r287986: Fix memory corruption when >128K transferred through HA link.

While there, do some cleaning.

MFC r287968: Mark I/Os with DMA flag while moving data through the HA link.

MFC r287967: Relax serseq option operation for reads.

Previously, with serseq enabled, next command was unblocked only after
previous completed. With this change, for read operations, next command
is unblocked as soon as last media read completed. This is important
for frontends that actually wait for data move completion (like camtgt),
or when data are moved through the HA link, or especially when both.

MFC r287957: Kill HA link and shutdown the threads on shutdown.

MFC r287940: Replicate initiators WWPNs and names between HA peers.

MFC r287933: Replicate port->init_devid to HA peer.

MFC r287921: When reporting TPT UA, report which of thresholds was reached.

MFC r287913: Report number of failed XCOPY segment.

MFC r287912: Report proper medium error code for VERIFY commands.

MFC r287875: Fix reading after end of file for file-backed LUNs.

If backing file is smaller then the LUN size, we have to explicitly clear
the rest of the buffer to not leak some random data from previous I/Os.

MFC r287868: Make COMPARE AND WRITE report offset of difference.

MFC r287860: Frontends don't need to set errors themselves.

MFC r287855: Don't flap the HA link if sysctl is reset to the same value.

MFC r287818: Fix completion/error status reporting.

MFC r287816: Close potential race between datamove and HA failover.

MFC r287784: Implement data/status aggregation for secondary HA node.

For short read requests this reduces latency by 30%, reporting command
completion after two interlink roundtrips instead of full three.

MFC r287783: Implement data/status aggregation for camsim backend.

This is almost pointless for primary HA node, but useful for secondary,
where between fe_datamove and fe_done calls goes another link roundtrip.

MFC r287778: Remove CTL_PRIV_LBA_LEN from HA messages.

Previously it was used for statistics, but now just a 16 extra bytes.

MFC r287774: Implement QUERY TASK, QUERY TASK SET and QUERY ASYNC EVENT.

Now we support most of SAM-5 task management.

MFC r287765: Map CLEAR TASK SET and I_T NEXUS RESET for iSCSI.

The last should not be called without iSCSIProtocolLevel negotiation.

MFC r287764: Implement iSCSI TARGET COLD RESET task management function.

Implement it as CTL_TASK_TARGET_RESET plus termination of all sessions.

MFC r287760: Improve read-only support.

MFC r287758: Reannounce port to HA peer if LUN map changed after online.

MFC r287756:
Report INQUIRY DATA HAS CHANGED for related LUNs on port on-/offline.

MFC r287754: Report CTL_UA_LUN_CHANGE on LUN map change.

MFC r287748: Make TARGET RESET respect LUN mapping.

MFC r287724: Check for obsolete NUL bin in CSCD descriptor.

MFC r287721: Add HA support for CTL_TASK_I_T_NEXUS_RESET.

MFC r287720: Fix false CTL_UA_RES_RELEASE on secondary HA node.

MFC r287715: Improve XCOPY error reporting.

MFC r287714: Report that we have no limit on POPULATE TOKEN segment size.

MFC r287712: Correct RTPG bitmask.

MFC r287711: Some HA polishing.

MFC r287707: CTL documentation update, mostly for HA.

MFC r287671:
Make ctld restart on primary HA node less invasive for secondary.

MFC r287670: Close races between device close and request processing.

All requests arriving for processing after OFFLINE flag set are rejected
with BUSY status. Races around OFFLINE flag setting are closed by calling
taskqueue_drain_all().

MFC r287664: Reference/release devices on every I/O, rather on open/close.

While this may be slower, it allows device destruction to complete,
rather then block waiting for indefinitely long time.

MFC r287621: Reimplement CTL High Availability.

CTL HA functionality was originally implemented by Copan many years ago,
but large part of the sources was never published. This change includes
clean room implementation of the missing code and fixes for many bugs.

This code supports dual-node HA with ALUA in four modes:
- Active/Unavailable without interlink between nodes;
- Active/Standby with second node handling only basic LUN discovery and
reservation, synchronizing with the first node through the interlink;
- Active/Active with both nodes processing commands and accessing the
backing storage, synchronizing with the first node through the interlink;
- Active/Active with second node working as proxy, transfering all
commands to the first node for execution through the interlink.

Unlike original Copan's implementation, depending on specific hardware,
this code uses simple custom TCP-based protocol for interlink. It has
no authentication, so it should never be enabled on public interfaces.

The code may still need some polishing, but generally it is functional.

Relnotes: yes
Sponsored by: iXsystems, Inc.

MFC r287620: Remove unused target and initiator IDs.

MFC r287618: Disable CTL_IO_DELAY feature.

It is too developer-oriented to be enabled by default.

MFC r287500: Allow LUN options modification via CTL_LUNREQ_MODIFY.

Not all changes take effect, but that is a different question.

MFC r287499: Move setting of media parameters inside open routines.

This is preparation for possibility to open/close media several times
per LUN life cycle. While there, rename variables to reduce confusion.
As additional bonus this allows to open read-only media, such as ZFS
snapshots.

MFC r287455, r287458, r287459: Remove some dead code.

MFC r287433: Small UA cleanup.

MFC r287372:
Make most of port methods optional and remove bunch of dummies.

MFC r287293: Remove 600 bytes of port_priv from struct ctl_io_hdr.

This field used only for camtgt frontend, and once it any way preallocates
all requests, let it preallocate this memory too, not bothering core code.

MFC r287221: Fix type bug introduced at r286811.

MFC r286811: Polish sizes processing.

MFC r286807: Move "ioctl" CAM frontend into separate file.

It has nothing to share with too huge ctl.c other then device descriptor,
but even that may be counted as design error that may be fixed later.
At some point we may even want to have several ioctl ports.

MFC r286806: Drop "internal" CTL frontend.

Its idea was to be a simple initiator and execute several commands from
kernel level, but FreeBSD never had consumer for that functionality,
while its implementation polluted many unrelated places.

MFC r286406, r286414:
Wrap some unused functions with notyet, it is necessary to be able to
build the modules/ctl directly.
Remove a dead MALLOC_DEFINE.

MFC r285155:
Make first step toward supporting target and initiator roles same time.

To avoid conflicts between target and initiator devices in CAM, make
CTL use target ID reported by HBA as its initiator_id in XPT_PATH_INQ.
That target ID is known to never be used for initiator role, so it won't
conflict. For Fibre Channel and FireWire HBAs this specific ID choice
is irrelevant since all target IDs there are virtual. Same time for SPI
HBAs it seems could be even requirement to use same target ID for both
initiator and target roles.

While there are some more things to polish in isp(4) driver, first tests
of using both roles same time on the same port appeared successfull:

# camcontrol devlist -v
scbus0 on isp0 bus 0:
<FREEBSD CTLDISK 0001> at scbus0 target 1 lun 0 (da20,pass21)
<> at scbus0 target 256 lun 0 (ctl0)
<> at scbus0 target -1 lun ffffffff (ctl1)

MFC r285030: Fix couple panics on forced unmount of backing file.

MFC r278398 (by trasz):
Fix ordering of "*logout" and "*terminate"; no functional changes.

MFC r287432: Fix copy-paste bug introduced in r275458.

MFC r286353: Pass SYNCHRONIZE CACHE command parameters to backends.

At this point IMMED flag is translated to MNT_NOWAIT flag of VOP_FSYNC(),
hoping that file system implements that (ZFS seems doesn't).

MFC r286345: Relax serialization of SYNCHRONIZE CACHE commands.

Before this change SYNCHRONIZE CACHE commands were executed exclusively,
as if they had ORDERED tag. But looking through SCSI specs I've found
no any reason to be so strict. For reads this ordering seems pointless.
For writes it looks less obvious, so I left ordering against preceeding
write commands, while following ones are no longer required to wait.

MFC r286320: Issue all reads of single XCOPY segment simultaneously.

During vMotion and Clone VMware by default runs multiple sequential 4MB
XCOPY requests same time. If CTL issues reads sequentially in 1MB chunks
for each XCOPY command, reads from different commands are not detected
as sequential by serseq option code and allowed to execute simultaneously.
Such read pattern confused ZFS prefetcher, causing suboptimal disk access.
Issuing all reads same time make serseq code work properly, serializing
reads both within each XCOPY command and between them.

My tests with ZFS pool of 14 disks in RAID10 shows prefetcher efficiency
improved from 37% to 99.7%, copying speed improved by 10-60%, average
read latency reduced twice on HDD layer and by five times on zvol layer.

MFC r286514: Remove verbose CTL messages.

Reporting SCSI errors to console is often useless, pollutes logs and may
affect performance. For debugging there is kern.cam.ctl.debug sysctl

MFC r284640: Bring per-port LUN enable/disable code up to date:
- remove last remnants of never implemented multiple targets support;
- implement missing support for LUN mapping in this area.

Due to existing locking constraints LUN mapping code is practically
unlocked at this point. Hopefully it is not racy enough to live until
somebody get idea how to call sleeping fronend methods under lock also
taken by the same frontend in non-sleepable context. :(

MFC r284627: Remove some dead and duplicate LUN enabling code.

MFC r284639: Introduce separate lock for tokens to reduce ctl_lock scope.

MFC r284641:
Fix REPORT LUNS command output for the case when same LUN mapped to same
port several times. While it is unusual configuration, it is not illegal.

MFC r284622: Remove device queue freeze handling and replace it with dummy.

At this point CTL has no known use case for device queue freezes.
Same time existing (considered to be broken) code was found to cause
modify-after-free issues.

MFC r284541:
Change ATIO/INOT counting to prevent periph destruction while requests
are still running inside CTL.

MFC r284013: Allow setting only WWNN or only WWPN.

MFC r284044: Fix wrong function name in error message.

MFC r282565: Handle EDQUOT backend storage errors same as ENOSPC.

MFC revision 280463

------------------------------------------------------------------------
r280463 | ken | 2015-03-24 15:42:28 -0600 (Tue, 24 Mar 2015) | 5 lines

Remove some #if 0'ed code that apparently confuses cscope.

Requested by: Peter Xu <xzpeter@gmail.com>
MFC after: 3 days

------------------------------------------------------------------------

Sponsored by: Spectra Logic

Merge r263233 from HEAD to stable/10:

Update kernel inclusions of capability.h to use capsicum.h instead; some
further refinement is required as some device drivers intended to be
portable over FreeBSD versions rely on __FreeBSD_version to decide whether
to include capability.h.

Sponsored by: Google, Inc.

MFC r278584: Add support for General Statistics and Performance log page.

CTL already collects most of statistics reported there, so why not.

MFC r278672: Teach CTL to ask GEOM devices about BIO_DELETE support.

MFC r278625: Make XCOPY and WUT commands respect physical block size/offset.

This change by 2-3 times improves performance of misaligned XCOPY and WUT
commands by avoiding unneeded read-modify-write cycles inside ZFS.

MFC r278161: Bring some more order into iSCSI portal group tags support.

While ctld(8) still does not allow multiple portal groups per target
to be configured, kernel should now be able to handle it.

Sponsored by: iXsystems, Inc.

MFC r278037: CTL LUN mapping rewrite.

Replace iSCSI-specific LUN mapping mechanism with new one, working for any
ports. By default all ports are created without LUN mapping, exposing all
CTL LUNs as before. But, if needed, LUN mapping can be manually set on
per-port basis via ctladm. For its iSCSI ports ctld does it via ioctl(2).
The next step will be to teach ctld to work with FibreChannel ports also.

Respecting additional flexibility of the new mechanism, ctl.conf now allows
alternative syntax for LUN definition. LUNs can now be defined in global
context, and then referenced from targets by unique name, as needed. It
allows same LUN to be exposed several times via multiple targets.

While there, increase limit for LUNs per target in ctld from 256 to 1024.
Some initiators do not support LUNs above 255, but that is not our problem.

Relnotes: yes
Sponsored by: iXsystems, Inc.

MFC r278619: Make WRITE SAME commands respect physical block size.

This change by 2-3 times improves performance of misaligned WRITE SAME
commands by avoiding unneeded read-modify-write cycles inside ZFS.

MFC r278500: Do not abort already aborted tasks.

This fixes abort of new tasks with the same tags as previously aborted,
but still remaining on the queue.

MFC r277917 (by ken), r278598:
Improve SCSI Extended Inquiry VPD page (0x86) support.

sys/cam/scsi/scsi_all.h:
In struct scsi_extended_inquiry_data:
- Increase the length field to 2 bytes, as it is 2 bytes in SPC-4.
- Add bit definitions for the various Activiate Microcode actions.
- Add the Sequential Access Logical Block Protection support bit,
since we need that in the sa(4) driver. (For modifications
that will come later.)
- Add definitions for the various Multi I_T Nexus Microcode
Download modes.

sys/cam/ctl/ctl.c:
As of SPC-4, a single report of "REPORTED LUNS DATA HAS CHANGED"
is to be given per I_T nexus. Once it is reported, the unit
attention condition should be cleared for all LUNS attached to
an I_T nexus.

Previously that only happened when a REPORT LUNS command was
processed.

This behavior may be different (according to SAM-5) when the
UA_INTLCK_CTRL bits are non-zero in the control mode page but
CTL does not currently support that.

So, in view of the spec, whenever we report a LUN inventory
change unit attention, clear it on all LUNs for that
particular I_T nexus.

Add a new function, ctl_clear_ua() that will clear a unit
attention on all LUNs for the given I_T nexus.

One field in the extended inquiry data that we could potentially
report at some point is the maximum supported sense data length.
To do that, we would the SIM to report (via path inquiry
perhaps) how much sense data it is able to send.

Add comments to explain some of the bits that are set in the
Extended Inquiry VPD page.

Add a few comments to make it more clear which functions handle
various VPD pages.

MFC r277247: Don't count status as sent until CTIO completes successfully.

If we aggregated status sending with data move and got error, allow status
to be updated and resent again separately. Without this command may stuck
without status sent at all.

MFC r277529: Don't count requests with status sent as overlapping.

While those requests are still in target OOA queue, for initiator they are
already completed, so tags can be reused.

MFC r277647: Fix wrong LUN reference in XCOPY block-to-block operation.

This could cause data corruption due to accessing wrong LUN in case of
retries on write errors. Failed writes were retried to read LUN.

MFC r274036 (by trasz):
s/icl_pdu_new_bhs/icl_pdu_new/; no functional changes, just a little
nicer code.

MFC r276141: Hide block device VPD pages for non-block devices.

MFC r275953: Replace ctl_min() macro with MIN().

MFC r275943: Constify some static data.

MFC r275942: Reduce number of places where global control_softc is used.

At some point we may want to have several CTL instances, and that is not
really impossible.

MFC r275864: Make sequence numbers checks more strict.

While we don't support MCS, hole in received sequence numbers may mean
only PDU loss. While we don't support lost PDU recovery, terminate the
connection to avoid stuck commands.

While there, improve handling of sequence numbers wrap after 2^32 PDUs.

MFC r275920, r276127: Pass real optimal transfer size supported by backend.

For files and ZVOLs that is 1MB now, not 128K.

MFC r275865:
Add configuration options to override physical and UNMAP blocks geometry.

While in most cases CTL should correctly fetch those values from backing
storages, there are some initiators (like MS SQL), that may not like large
physical block sizes, even if they are true. For such cases allow override
fetched values with supported ones (like 4K).

MFC r275959: Report initiator id in portlist XML in more formalized way.

MFC r275842: Do not count RCTD bit set as an error.

We can not really implement it, but specification tells that it "shall"
work, so it can be safely ignored.

MFC r275568:
Count consecutive read requests as blocking in CTL for files and ZVOLs.

Technically read requests can be executed in any order or simultaneously
since they are not changing any data. But ZFS prefetcher goes crasy when
it receives consecutive requests from different threads. Since prefetcher
works on level of separate blocks, instead of two consecutive 128K requests
it may receive 32 8K requests in mixed order.

This patch is more workaround then a real fix, and it does not fix all of
prefetcher problems, but it improves sequential read speed by 3-4x times
in some configurations. On the other side it may hurt performance if
some backing store has no prefetch, that is why it is disabled by default
for raw devices.

MFC r275512:
In addition to r275481 allow threshold notifications work without UNMAP.

While without UNMAP support there is not much initiator can do about it,
the administrator still better be notified about the storage overflow.

Sponsored by: iXsystems, Inc.

MFC r275481:
Add to CTL support for threshold notifications for file-backed LUNs.

Previously it was supported only for ZVOL-backed LUNs, but now should work
for file-backed LUNs too. Used value in this case is a space occupied by
the backing file, while available value is an available space on file
system. Pool thresholds are still not implemented in this case.

Sponsored by: iXsystems, Inc.

MFC r275474: Add GET LBA STATUS command support to CTL.

It is implemented for LUNs backed by ZVOLs in "dev" mode and files.
GEOM has no such API, so for LUNs backed by raw devices all LBAs will
be reported as mapped/unknown.

Sponsored by: iXsystems, Inc.

MFC r275461:
Increase CTL ports limit from 128 to 256 and LUNs limit from 256 to 1024.

After recent optimizations this change is no longer blocked by CTL memory
consumption. Those limits are still not free, but much cheaper now.

Relnotes: yes
Sponsored by: iXsystems, Inc.

MFC r275459: Unify function names after r275458.

MFC r275458:
Do not pre-allocate UNIT ATTENTIONs storage for every possible initiator.

Abusing ability of major UAs cover minor ones we may not account UAs for
inactive ports. Allocate UAs storage for port and start accounting only
after some initiator from that port fetched its first POWER ON OCCURRED.

This reduces per-LUN CTL memory usage from >1MB to less then 100K.

MFC r275455: Remove some unused code.

MFC r275447:
Do not pre-allocate reservation keys memory for every possible initiator.

In configurations with many ports, like iSCSI, each LUN is typically
accessed only by limited subset of ports. Allocating that memory on
demand allows to reduce CTL memory usage from 5.3MB/LUN to 1.3MB/LUN.

MFC r275405: Convert persis_offset from global variable to softc field.

MFC r275404: Reduce code duplication by creating ctl_set_res_ua() helper.

MFC r275403: Removed unused variable and unify some names.

MFC r275365: Move ctlfe_onoffline() out of lock to let it sleep when needed.

Do some more other polishing while there.

MFC r275058: Coalesce last data move and command status for read commands.

Make CTL core and block backend set success status before initiating last
data move for read commands. Make CAM target and iSCSI frontends detect
such condition and send command status together with data. New I/O flag
allows to skip duplicate status sending on later fe_done() call.

For Fibre Channel this change saves one of three interrupts per read command,
increasing performance from 126K to 160K IOPS. For iSCSI this change saves
one of three PDUs per read command, increasing performance from 1M to 1.2M
IOPS.

Sponsored by: iXsystems, Inc.

MFC r275032: Decouple datamove/done logic from CTL status set.

MFC r275009: Use ctl_set_success() instead of direct inlining.

MFC r274962: Replace home-grown CTL IO allocator with UMA.

Old allocator created significant lock congestion protecting its lists
of preallocated I/Os, while UMA provides much better SMP scalability.
The downside of UMA is lack of reliable preallocation, that could guarantee
successful allocation in non-sleepable environments. But careful code
review shown, that only CAM target frontend really has that requirement.
Fix that making that frontend preallocate and statically bind CTL I/O for
every ATIO/INOT it preallocates any way. That allows to avoid allocations
in hot I/O path. Other frontends either may sleep in allocation context
or can properly handle allocation errors.

On 40-core server with 6 ZVOL-backed LUNs and 7 iSCSI client connections
this change increases peak performance from ~700K to >1M IOPS! Yay! :)

Sponsored by: iXsystems, Inc.

MFC r275478: Swap resource count scopes for used/available space.

Used count should be reported as per-LUN, while available should not.

MFC r275446: Plug memory leaks on UNMAP and XCOPY with invalid parameters.

MFC r274805:
Make cfiscsi_offline() synchronous, waiting for connections termination
before return. This should make ctld restart more clean and predictable.

MFC r274795:
Close race between cfiscsi_offline() and new connection arrival.

Incoming connection should be either rejected or accepted and terminated.

MFC r274785: Partially reconstruct Active/Standby clusting.

In this mode one head is in Active state, supporting all commands, while
another is in Standby state, supporting only minimal LUN discovery subset.

It is still incomplete since Standby state requires reservation support,
which is impossible to do right without having interlink between heads.
But it allows to run some basic experiments.

MFC r274703:

Fix typo.

Sponsored by: The FreeBSD Foundation

MFC r273918:

Change the default log level for iSCSI target from 3 to 1. It should
have been 1 from the beginning; not sure how it ended up at 3.

Sponsored by: The FreeBSD Foundation

MFC r274840, r274940:
Make iSCSI frontend less chatty while waiting for tasks termination.

MFC r274790: Remove bunch of unused lun variables.

MFC r274789: Reduce race between LUN destruction and request arrival.

MFC r274786: Log errors for absent LUNs too.

MFC r274154, r274163:
Add to CTL support for logical block provisioning threshold notifications.

For ZVOL-backed LUNs this allows to inform initiators if storage's used or
available spaces get above/below the configured thresholds.

Sponsored by: iXsystems, Inc.

MFC r274333: Handle PREEMPT AND ABORT service action equal to PREEMPT.

With command serialization used in CTL, there are no other commands to abort
when PREEMPT AND ABORT gets to run, so it is practically equal to PREEMPT.

MFC r274206:
Synchronize medium rotation rate in legacy Rigid Disk Drive Geometry mode
page with modern Block Device Characteristics VPD page.

MFC r274081: Fix residual copy/paste in r274080.

MFC r274080: Improve error handling around duplicate lun and port enable.

This fixes kernel panic if port enabled twice and then disabled.

MFC r274253: Fix LUN resize broken by r272911 commit.

MFC r273809:
Implement better handling for ENOSPC error for both CTL and CAM.

This makes VMWare VAAI Thin Provisioning Stun primitive activate, pausing
the virtual machine, when backing storage (ZFS pool) is getting overflowed.

MFC r273730, r273731:
Reduce code duplication around Write Exclusive persistent reservation.

While there, allow some more commands to pass persistent reservation.

MFC r273711:
Allocate buffer for READ BUFFER/WRITE BUFFER commands on demand.

These commands are rare, but consume additional 256KB RAM per LUN.

MFC r273708: Fix support for LUN flat space addressing.

MFC r273693: Fix printing non-terminated strings in devlist XML.

MFC r273687:
Add "rpm" and "formfactor" LUN options to match istgt functionality.

MFC r273640: Add support for 12/16-byte EUI and 16-byte NAA IDs.

MFC r273075: Remove couple Copan's vendor-specific mode pages.

Those pages are highly system-/hardware-specific, the code is incomplete,
and so they hardly can be useful for anybody else.

MFC r273073: Some groundwork for later Informational Exceptions support.

This includes support for:
- Read-Write Error Recovery mode page;
- Informational Exceptions Control mode page;
- Logical Block Provisioning mode page;
- LOG SENSE command.

No real Informational Exceptions features yet. This is only a placeholder.

MFC r273259:
Make VPD 80h (Serial Number) transfer length match serial number length.

MFC r273163: Implement more functional CTL debug logging.

Setting bits in kern.cam.ctl.debug allows to log errors, commands and some
commands data respectively.

MFC r273046:
Don't confuse frontend with zero length data moves, just return immediately.

MFC r273038: Add support for READ DEFECT DATA (10/12) commands.

SPC-4 r2 allows to return empty defect list if the list is not supported.
We don't reallu support defect data lists, but this suppresses some errors.

MFC r273029:
Report physical block size for file-backed LUNs, using vattr.va_blocksize.

MFC r273008: Remove stale comments.

MFC r272978: Improve and document `ctladm portlist` subcommand.

Make this subcommand less FC-specific, reporting target and port addresses
in more generic way. Also make it report list of connected initiators in
unified way, working for both FC and iSCSI, and potentially others.

MFC r272947: Give physical and virtual ports numbers some more meaning.

MFC r272939: Shorten frontend name.

MFC r272938: Filter out duplicate AC_PATH_REGISTERED async events.

Queued async events handling in CAM opened race, that may lead to duplicate
AC_PATH_REGISTERED events delivery during boot. That was not happening
before r272935 because the driver was initialized later. After that change
it started create duplicate ports in CTL.

MFC r272935: Mark CTL frontend's CAM driver as CAM_PERIPH_DRV_EARLY.

Target mode operation does not depend on the initiator mode scan process.
This change allows the target driver to attach earlier and receive some
async events (like AC_CONTRACT) that could be lost otherwise.

MFC r272911:
Make ctld start even if some LUNs are unable to open backing storage.

Such LUNs will be visible to initiators, but return "not ready" status
on media access commands. If backing storage become available later,
`ctladm modify ...` or `service ctld reload` can trigger its reopen.

MFC r272893:
Store persistent reservation keys as uint64_t instead of uint8_t[8].

This allows to simplify the code and save 512KB of RAM per LUN (8%)
by removing no longer needed "registered" keys flags.

MFC r272812: Make iSCSI connection close somewhat less aggressive.

It allows to push out some final data from the send queue to the socket
before its close. In particular, it increases chances for logout response
to be delivered to the initiator.

MFC r272748:
Implement software (mode page) and hardware (config) write protection.

MFC r272734:
Add support for WRITE ATOMIC (16) command and report SBC-4 compliance.

Atomic writes are only supported for ZVOLs in "dev" mode. In other cases
atomicity can not be guarantied and so the command is blocked.

MFC r272613:
Add support for MaxBurstLength and Expected Data transfer Length parameters.

Before this change target could send R2T request for write transfer of any
size, that could violate iSCSI RFC, which allows initiator to limit maximum
R2T size by negotiating MaxBurstLength connection parameter.

Also report an error in case of write underflow, when initiator provides
less data than initiator expects. Previously in such case our target
sent R2T request for non-existing data, violating the RFC, and confusing
some initiators. SCSI specs don't explicitly define how write underflows
should be handled and there are different oppinions, but reporting error
is hopefully better then violating iSCSI RFC with unpredictable results.

MFC r272597: Fix length of Extended INQUIRY Data VPD page.

MFC r271395 (by trasz):
Make sure we handle less than zero timeouts in iSCSI initiator and target
in a reasonable way.

Sponsored by: The FreeBSD Foundation

MFC r272650: Set CAM_SIM_QUEUED flag before calling ctl_queue() to avoid race.

PR: 194128
Submitted by: Scott M. Ferris <smferris@gmail.com>
Sponsored by: EMC/Isilon Storage Division

MFC r272355: Fix couple issues with ROD tokens content.

MFC r272247:
Do not transfer unneeded training zero bytes in INQUIRY response.

It is an addition to r269631.

MFC r272224: Fix page length reported for Block Limits VPD page.

MFC r272033:
Fix ASCQ for "Logical unit not ready, manual intervention required" error.

MFC r271959: Pretend that we support BYTCHK=1 in WRITE AND VERIFY command.

Technically that is not true, but since we don't implement VERIFY there
at all, doing only WRITE part, this is a minor sin.

MFC r271954:
Deny ANCHOR flag set without UNMAP flag set in WRITE SAME commands.

MFC r271951: Don't try to continue aborted commands if status was not set.

MFC r271945:
Simplify legacy reservation handling. Drop it on I_T nexus loss.

MFC r271942:
Don't report unsupported FUA_NV bit set in READ/WRITE commands as error.

While this bit is obsolete in SBC-3, SBC-2 allowed to silently ignore it.

MFC r271941:
Report proper errors codes for unsupported SERVICE ACTION values.

MFC r271940: Polish INQUIRY command fields validation.

MFC r271930: Allow SUBPAGE CODE field in MODE SENSE commands.

MFC r271869:
Fix inverted expression to report block size in mode page block descriptor.

MFC r271845:
Allow more commands to pass persistent reservation according to SPC-4 r37.

MFC r271839:
Add support for "no Data-Out Buffer" (NDOB) flag of WRITE SAME (16) command.

MFC r271606:
Always report that we support REPORT TARGET PORT GROUPS command.

Without clustering support we any way have only one group of permanently
active ports, but that gives us one more supported VMWare feature. ;)

Solaris' Comstar also reports it even when only one port is present.

MFC r271507:
Implement control over command reordering via options and control mode page.

It allows to bypass range checks between UNMAP and READ/WRITE commands,
which may introduce additional delays while waiting for UNMAP parameters.
READ and WRITE commands are always processed in safe order since their
range checks are almost free.

MFC r271505:
Add "readcache" and "writecache" LUN options to control default behavior.

Default values are "on". Disabling requires backend to support IO_DIRECT
and IO_SYNC flags respectively, or some alternatives.

MFC r271503: Implement range checks between UNMAP and READ/WRITE commands.

Before this change UNMAP completely blocked other I/Os while running.
Now it blocks only colliding ones, slowing down others only due to ZFS
locks collisions.

Sponsored by: iXsystems, Inc.

MFC r271443: Add support for Extended INQUIRY Data (0x86) VPD page.

MFC r271360: Remove uninitialized and unused variable, reported by Coverity.

MFC r271358: Fix array overrun, reported by Coverity.

MFC r271354: Fix couple off-by-one range check errors, reported by Coverity.

MFC r271353: Fix memory leak on error, reported by Coverity.

MFC r271352: Fix minor buffer overflow reported by Coverity.

MFC r271316: Report that DPO and FUA bits are supported after r271311.

MFC r271311:
Add support for Mode Page Policy (0x87) VPD page.

MFC r271309:
Improve cache control support, including DPO/FUA flags and the mode page.

At this moment it works only for files and ZVOLs in device mode since BIOs
have no respective respective cache control flags (DPO/FUA).

MFC r272040:
When reporting some major UNIT ATTENTION condition, like POWER ON OCCURRED
or I_T NEXUS LOSS, clear all minor UAs for the LUN, redundant in this case.

All SAM specifications tell that target MAY do it, but libiscsi initiator
seems require it to be done, terminating connection with error if some more
UAs happen to be reported during iSCSI connection.

Approved by: re (gjb)

MFC r271957: Fix read overrun handling, broken by using wrong variable.

Approved by: re (marius)

MFC r271949:
Fix UNMAP stuck if the last block descriptor in the list is empty.

Approved by: re (glebius)

MFC r271794: When updating device media size use cached cdevsw pointer.

Using pointer from the cdev directly is dangerous since we have no
reference on it, and it may change any time. That caused panic if
device has gone.

While there, report capacity change only if it really changed.

Approved by: re (dephij)

MFC r271702:
Fix tpc_create_token() introduced in r269497 to encode CREATOR LOGICAL
UNIT DESCRIPTOR field as Identification Descriptor CSCD descriptor, not
just as Identification Descriptor.

Approved by: re (gjb)

MFC r271393:

Make it possible to disable NOP-In PDUs by the iSCSI initiator by setting
kern.cam.ctl.iscsi.ping_timeout to 0. This fixes interoperability with
some initiators that don't properly support NOP-Ins, namely iPXE/gPXE.

Approved by: re (kib)

MFC r271362:
Make ctl_port_mask an array to support more then 32 ports.

Overflow reported by Coverity.

CID: 1229894

Approved by: re (marius)

MFC r270282:

Use proper include paths in kernel iSCSI code.

Sponsored by: The FreeBSD Foundation

MFC r270176:
Fix lock recursion on LUN shutdown, introduced on r269497.

MFC r269631:
Reduce reported additional INQUIRY data length.

sizeof(struct scsi_inquiry_data) of 256 bytes combined with off-by-one
error in the changed code gave total INQUIRY data length above 255 bytes,
that was maximal INQUIRY length in SPC-2. While SPC-3 increased the
maximal length to 64K, at least sg3_utils are still confused by that.

MFC r269622:
Fix several issues and inconsistencies in UNMAP capabilities reporting.

This makes Windows 2012 to start using UNMAP on our disks.

MFC r269587:
Reimplement WRITE USING TOKEN with Block Zero token using WRITE SAME.

On my ZVOL of SSDs that increases speed of zero writing in that way from
1 to 2.5GB/s by reducing CPU overhead.

MFC r269497:
Add support for Windows dialect of EXTENDED COPY command, aka Microsoft ODX.

This allows to avoid extra network traffic when copying files on NTFS iSCSI
disks within one storage host by drag'n'dropping them in Windows Explorer
of Windows 8/2012. It should also accelerate Hyper-V VM operations, etc.

MFC r269444, r269450:
Plug EXTENDED COPY request data memory leak.

MFC r269442:
Fix some bugs in RECEIVE COPY STATUS data.

MFC r269441:
Add missing comparisons to make list IDs in EXTENDED COPY per-initiator,
as they should be. Wrap it into a function to not duplicate the code.

MFC r269123:
Implement separate I/O dispatch method for ZVOLs in "dev" mode.

Unlike disk devices ZVOLs process all requests synchronously. That makes
impossible sending multiple requests to them from single thread. From the
other side ZVOLs have real d_read/d_write methods, which unlike d_strategy
can handle uio scatter/gather and have no strict I/O size limitations.

So, if ZVOL in "dev" mode is detected, use of d_read/d_write methods instead
of d_strategy allows to avoid pointless splitting of large requests into
MAXPHYS (128K) sized chunks.

MFC r268808:
Increase maximal number of SCSI ports in CTL from 32 to 128.

After I gave each iSCSI target its own port, the old limit appeared to be
not so big. This change almost proportionally increases per-LUN memory
use, but it is still three times better then it was before r268807.

MFC r268807:
Reduce per-LUN memory usage from 18MB to 1.8MB.

CTL never had use for CA support code since SPI has gone, and there is no
even frontends supporting that. But it still was reserving 256 bytes of
memory per LUN per every possible initiator on every possible port.

Wrap unused code with ifdef's in case somebody ever need it.

MFC r268767:
Add support for VMWare dialect of EXTENDED COPY command, aka VAAI Clone.

This allows to clone VMs and move them between LUNs inside one storage
host without generating extra network traffic to the initiator and back,
and without being limited by network bandwidth.

LUNs participating in copy operation should have UNIQUE NAA or EUI IDs set.
For LUNs without these IDs VMWare will use traditional copy operations.

Beware: the above LUN IDs explicitly set to values non-unique from the VM
cluster point of view may cause data corruption if wrong LUN is addressed!

Sponsored by: iXsystems, Inc.

MFC r268581:
Merge several equal serialization indexes.

MFC r269149:
Fix several cases of NULL dereference when INQUIRY sent to absent LUN.

MFC r269122:
Fix infinite loop, when doing WRITE SAME on file-backed LUN.

MFC r269088:

Fix ctl(4) kldload failure that manifested like this:

link_elf_obj: symbol icl_pdu_new_bhs undefined

PR: 192031
Submitted by: Nils Beyer (earlier version)
Sponsored by: FreeBSD Foundation

MFC r269058:
Fix build with QUEUE_MACRO_DEBUG.

MFC r268447:
Add LUN options to specify 64-bit EUI and NAA identifiers.

MFC r268421:
Remove status setting from datamove() path. Leave that to other places.

MFC r268418:
Enable TAS feature: notify initiator if its command was aborted by other.

That should make operation more kind to multi-initiator environment.
Without this, other initiators may find out that something bad happened
to their commands only via command timeout.

MFC r268387:
Fix task management functions status: task not found is not an error,
while not implemented function is.

MFC r268364:
Fix "use after free" on port creation error in r268291.

MFC r268363:
Add support for READ FULL STATUS action of PERSISTENT RESERVE IN command.

MFC r268362:
Teach ctl_add_initiator() to dynamically allocate IIDs from pool.

If port passed negative IID value, the function will try to allocate IID
from the pool of unused, based on passed wwpn or name arguments. It does
all its best to make IID unique and persistent across reconnects.

This makes persistent reservation properly work for iSCSI. Previously,
in case of reconnects, reservation could be unexpectedly lost, or even
migrate between intiators.

MFC r268356, r268357:
When new connection comes in, check whether we already have session from
the same intiator (Name+ISID). If so -- terminate the old session and let
the new one take its place, as required by iSCSI RFC.

MFC r268353:
Implement ABORT TASK SET and I_T NEXUS RESET task management functions.

Use the last one to terminate active commands on iSCSI session termination.
Previous code was aborting only commands doing some data moves.

MFC r268330:
Make gcc happy, init idlen2.

MFC r268328:
Close race in r268291 between port destruction, delayed by sessions
teardown, and new port creation during `service ctld restart`.

Close it by returning iSCSI port internal state, that allows to identify
dying ports, which should not be counted as existing, from really alive.

MFC r268309:
Add support for SCSI Ports (88h) VPD page.

MFC r268308:
Make REPORT TARGET PORT GROUPS command report realistic data instead of
hardcoded garbage.

MFC r268307:
Move lun_map() method from command nexus to port.

Previous implementation made impossible to do some things, such as calling
it for ports other then one through which command arrived.

MFC r268302:
Pass through iSCSI session ISID from LOGIN request to the CTL frontend.

ISID is an important part of initiator transport ID for iSCSI. It is not
used now, but should be to properly implement persistent reservation.

MFC r268293:
Burry devid port method, which was a gross hack.

Instead make ports provide wanted port and target IDs, and LUNs provide
wanted LUN IDs. After that core Device ID VPD code only had to link all
of them together and add relative port and port group numbers.

LUN ID for iSCSI LUNs no longer created by CTL, but by ctld, and passed
to CTL as "scsiname" LUN option. This makes LUNs to report the same set
of IDs, independently from the port through which it is accessed, as
required by SCSI specifications.

MFC r268291:
Create separate CTL port for every iSCSI target (and maybe portal group).

Having single port for all iSCSI connections makes problematic implementing
some more advanced SCSI functionality in CTL, that require proper ports
enumeration and identification.

This change extends CTL iSCSI API, making ctld daemon to control list of
iSCSI ports in CTL. When new target is defined in config fine, ctld will
create respective port in CTL. When target is removed -- port will be
also removed after all active commands through that port properly aborted.
This change require ctld to be rebuilt to match the kernel.

As a minor side effect, this allows to have iSCSI targets without LUNs.
While that may look odd and not very useful, that is not incorrect.

MFC r268288:
Improve CTL_BEARG_* flags support, including optional values copyout.

MFC r268287:
Implement and use ctl_frontend_find().

MFC r268284:
Introduce new IOCTL CTL_PORT_LIST reporting in more flexible XML format.

Leave old CTL_GET_PORT_LIST in place so far. Garbage-collect it later.

MFC r268280:
Make options KPI more generic to allow it to be used for ports too,
not only for LUNs.

MFC r268266, r268275:
Separate concepts of frontend and port.

Before iSCSI implementation CTL had no knowledge about frontend drivers,
it had only frontends, which really were ports (alike to LUNs, if comparing
to backends). But iSCSI added there ioctl() method, which does not belong
to frontend as a port, but belongs to a frontend driver.

MFC r268265:
Remove targ_enable()/targ_disable() frontend methods.

Those methods were never implemented, and I believe that their concept is
wrong, since single frontend (SCSI port) can not handle several targets.

MFC r268103:
Add support for REPORT TIMESTAMP command.

MFC r268096, r268306, r268361:
Add more formal and strict command parsing and validation.

For every supported command define CDB length and mask of bits that are
allowed to be set. This allows to remove bunch of checks through the code
and still make the validation more strict. To properly do it for commands
supporting multiple service actions, formalize their parsing by adding
subtables for each of such commands.

As visible effect, this change allows to add support for REPORT SUPPORTED
OPERATION CODES command, reporting to client all the data about supported
SCSI commands, except timeouts.

MFC r267639:
Increase CTL_DEVID_LEN from 16 to 64 bytes.

SPC-4 recommends T10 vendor ID based LUN ID was created by concatenating
product name and serial number (and istgt follows that). But product name
is 16 bytes long by itself, so 16 bytes total length is clearly not enough
to fit both.

To keep compatibility with existing configurations, pad short device IDs
to old length of 16, same as before.

This change probably breaks CTL user-level ABI, so control tools should
be rebuilt after this change.

MFC r267643, r267873, r268391, r268398:
Introduce fine-grained CTL locking to improve SMP scalability.

Split global ctl_lock, historically protecting most of CTL context:
- remaining ctl_lock now protects lists of fronends and backends;
- per-LUN lun_lock(s) protect LUN-specific information;
- per-thread queue_lock(s) protect request queues.
This allows to radically reduce congestion on ctl_lock.

Create multiple worker threads, depending on number of CPUs, and assign
each LUN to one of them. This allows to spread load between multiple CPUs,
still avoiging congestion on queues and LUNs locks.

On 40-core server, exporting 5 LUNs, each backed by gstripe of SATA SSDs,
accessed via 6 iSCSI connections, this change improves peak request rate
from 250K to 680K IOPS.

Sponsored by: iXsystems, Inc.

MFC r268283:
Improve readability of XML generated by CTL_LUN_LIST.

MFC r268392:
Do not return statuses for aborted iSCSI commands.

MFC r268204:
Use separate memory type M_CTLIO for I/Os.

CTL allocate large amount of RAM. This change give some more stats.

MFC r267933:
Simplify statistics calculation.

Instead of trying to guess size of disk I/O operations (it just won't work
that way for newly added commands, and is equal to data move size for old
ones), account data move traffic. If disk I/Os are that interesting, then
backends have to account and provide that information.

Block backend already exports the information about disk I/Os via devstat,
so having it here too is excessive.

MFC r267906:
Allow MODE SENSE commands through Write Exclusive persistent reservation,
as required by SPC-4.

Report that fact in persistent reservation capabilities.

MFC r267905:
Add READ BUFFER and improve WRITE BUFFER SCSI commands support.

This gives some use to 512KB per-LUN buffers, allocated for Copan-specific
processor code and not used. It allows, for example, to test transport
performance and/or correctness without accessing the media, as supported
by Linux version of sg3_utils.

MFC r267877:
Lock devstat updates in block backend to make it usable. Polish lock names.

MFC r268419:
Fix use-after-free on XPT_RESET_BUS.

That command is not queued, so does not use later status update.

MFC r267986:
Remove odd practice of inverting error codes.

-EPERM is equal to ERESTART, returning which from ioctl() handler causes
infinite syscall restart.

MFC r267641:
Add some more CTL_FLAG_ABORT check points.

This should allow to abort commands doing mostly disk I/O, such as VERIFY
or WRITE SAME. Before this change CTL_FLAG_ABORT was only checked around
data moves, which for these commands may not happen for a very long time.

MFC r267637:
Add iSCSI Target Name ID descriptor to VPD 83h.

It shall/should be there according to SPC-4, and istgt also provides it.

MFC r267610 (by trasz):
Rework session termination in iSCSI target to actually wait
for any outstanding commands to be properly aborted by CTL.
Without it, in some cases (such as files backing the LUNs
stored on failing disk drives), terminating a busy session
would result in panic.

MFC r267574 (by trasz):
Make cs_terminating a bool; no functional changes.

MFC r267547, r267551 (by trasz):
Add LUN-associated name to VPD, to make Hyper-V Failover Cluster happy.

MFC r267537:
Add support for VERIFY(10/12/16) and COMPARE AND WRITE SCSI commands.

Make data_submit backends method support not only read and write requests,
but also two new ones: verify and compare. Verify just checks readability
of the data in specified location without transferring them outside.
Compare reads the specified data and compares them to received data,
returning error if they are different.

VERIFY(10/12/16) commands request either verify or compare from backend,
depending on BYTCHK CDB field. COMPARE AND WRITE command executed in two
stages: first it requests compare, and then, if succeesed, requests write.
Atomicity of operation is guarantied by CTL request ordering code.

Sponsored by: iXsystems, Inc.

MFC r267519:
Make backends track completion by processed number of sectors instead of
total transfer size.

Commands such as VERIFY or COMPARE AND WRITE may have transfer size not
matching directly to number of sectors.

MFC r267515:
Remove memcpy() from ctl_private[] accesses.

That union is aligned enough to access data directly.

MFC r267514:
Move kern_total_len setting from backend to core code.

MFC r267500:
Format Portal Group Tag same as istgt does -- %4.4x instead of %x.

SPC-4 spec tells it should be "two or more hexadecimal digits".
RFC3720 tells it is 16-bit value.

MFC r267499:
Remove custom processing for "file" option.

MFC r267496, r267498:
Add "vendor", "product" and "revision" options to control inquiry data.

MFC r267485:
Remove non-functional remnants of control LUN -- 18MB of RAM for nothing.

MFC r267481, r267952:
Implement small KPI to access LUN options instead doing it by hands.

MFC r265323 (by trasz):
Provide better descriptions for 'struct ctl_scsiio' fields; based mostly
on emails from ken@.

MFC r267872:
Allow to use iSCSI immediate data by several ctl_datamove() calls.

While for FreeBSD client that is only a minor optimization, VMWare client
doesn't support additional data requests after all data being sent once as
immediate.

MFC r267429:
Fix some leaks on LUN creation error.

MFC r264886:
Remove limits on size of READ/WRITE operations.

Instead of allocating up to 16MB or RAM at once to handle whole I/O,
allocate up to 1MB at a time, but do multiple ctl_datamove() and storage
I/Os if needed.

MFC r264884:
Make CAM target CTL frontend respect SIM I/O size limitations.

If datamove size is bigger then SIM can handle, or it has more segments
then this code can handle -- split it into several CTIO requests.

MFC r264880 (by trasz):
Modify CTL iSCSI frontend to properly handle situations where datamove
routine is called multiple times per SCSI task.

MFC r264407:
Join CTL worker threads into one process for convenience.
Report their idle state as "-".

MFC r264274, r264279, r264283, r264296, r264297:
Add support for SCSI UNMAP commands to CTL.

This patch adds support for three new SCSI commands: UNMAP, WRITE SAME(10)
and WRITE SAME(16). WRITE SAME commands support both normal write mode
and UNMAP flag. To properly report UNMAP capabilities this patch also adds
support for reporting two new VPD pages: Block limits and Logical Block
Provisioning.

UNMAP support can be enabled per-LUN by adding "-o unmap=on" to `ctladm
create` command line or "option unmap on" to lun sections of /etc/ctl.conf.

At this moment UNMAP supported for ramdisks and device-backed block LUNs.
It was tested to work great with ZFS ZVOLs. For file-backed LUNs UNMAP
support is unfortunately missing due to absence of respective VFS KPI.

Sponsored by: iXsystems, Inc

MFC r264530:

Properly pass the initiator address when running in proxy mode.

Sponsored by: The FreeBSD Foundation

MFC r264527:

Make it possible to interrupt login when running in proxy mode.

Sponsored by: The FreeBSD Foundation

MFC r264526:

Properly identify target portal when running in proxy mode. While here,
remove CTL_ISCSI_CLOSE, it wasn't used or implemented anyway.

Sponsored by: The FreeBSD Foundation

MFC r264525:

Add some stuff to make it easier to figure out for the system administrator
whether the ICL_KERNEL_PROXY stuff got compiled in correctly.

Sponsored by: The FreeBSD Foundation

MFC r264524:

Make it possible for the iSCSI target side to operate in both normal
and ICL_KERNEL_PROXY mode, and fix some bit rot so the latter actually
works again.

Sponsored by: The FreeBSD Foundation

MFC r263740:

Use a less unusual syntax in debug printfs.

Sponsored by: The FreeBSD Foundation

MFC r264264 by mav@:

Wakeup only one thread of added in r263978i at a time.

This slightly reduces lock congestion between threads.

MFC r264110:

All the iSCSI sysctls are also tunables; advertise that.

Sponsored by: The FreeBSD Foundation

MFC r264024:

Use atomic ops instead of mutexes where appropriate.

Submitted by: mav@
Sponsored by: The FreeBSD Foundation

MFC r264023:

Instead of "icltx" and "iclrx", use thread names with prefix from upper
layer, so that one can see which side of the stack the threads are for.

Sponsored by: The FreeBSD Foundation

MFC r264022:

Get rid of ICL lock; use upper-layer (initiator or target) lock instead.
This avoids extra locking in icl_pdu_queue(); the upper layer needs to call
it while holding its own lock anyway, to avoid sending PDUs out of order.

Sponsored by: The FreeBSD Foundation

MFC r264020:

Remove the homegrown ctl_be_block_io allocator, replacing it with UMA.
There is no performance difference.

Reviewed by: mav@
Sponsored by: The FreeBSD Foundation

MFC r263979:

Hide CTL messages about SCSI error responses. Too many users take
them for actual target errors. They can be enabled back by setting
kern.cam.ctl.verbose=1, or booting with bootverbose.

Sponsored by: The FreeBSD Foundation

MFC r263978:

Make it possible to have multiple CTL worker threads. Leave the default
of 1 for now.

Sponsored by: The FreeBSD Foundation

MFC r263811:

Remove ctl_mem_pool.{c,h}.

Sponsored by: The FreeBSD Foundation