zfsd: Use vdev prop values for fault/degrade thresholds

ZED uses vdev props for setting disk fault/degrade thresholds, this
patch enables zfsd to use the same vdev props for these same tasks.

OpenZFS on Linux is using vdev props for ZED disk fault/degrade
thresholds. Originally the thresholds supported were for io and checksum
events and recently this was updated to process slow io events as
well, see
https://github.com/openzfs/zfs/commit/cbe882298e4ddc3917dfaf239eca475fe06d62d4

This patch enables us to use the same vdev props in zfsd as ZED uses.
After this patch is merged both OSs will use the same vdev props to set
retirement thresholds.

It's probably important to note that the threshold defaults are
different between OS. I've kept the existing defaults inside zfsd and
DID NOT match them to what ZED does.

Differential Revision: https://reviews.freebsd.org/D44043
MFC after: 2 weeks
Relnotes: yes
Reviewed by: asomers, allanjude
Sponsored by: Axcient
Submitted by: Alek Pinchuk <apinchuk@axcient.com>

zfsd: fault disks that generate too many I/O delay events

If ZFS reports that a disk had at least 8 I/O operations over 60s that
were each delayed by at least 30s (implying a queue depth > 4 or I/O
aggregation, obviously), fault that disk. Disks that respond this
slowly can degrade the entire system's performance.

MFC after: 2 weeks
Sponsored by: Axcient
Reviewed by: delphij
Differential Revision: https://reviews.freebsd.org/D42825

Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

zfsd: listen for sysevent.fs.zfs instead of misc.fs.zfs

At some point the names of these devd events changed. Probably it
happened when importing OpenZFS. Before that, FreeBSD's sysevent_alloc
method didn't create a "class" nvpair in the event, which led to
log_sysevent using the event's ev_subclass field as its type.

MFC after: 2 weeks
Sponsored by: Axcient
Differential Revision: https://reviews.freebsd.org/D39437

zfsd: add support for hotplugging spares

If you remove an unused spare and then reinsert it, zfsd will now online
it in all pools.

Do not MFC without 2a58b312b62 (but it's ok to MFC that one without this
one).

Submitted by: Ameer Hamza <ahamza@ixsystems.com> (zfsd), Me (tests)
MFC after: 2 weeks
MFC with: 2a58b312b62f908ec92311d1bd8536dbaeb8e55b
Sponsored by: iX Systems, Axcient
Pull Request: https://github.com/freebsd/freebsd-src/pull/697

zfsd: Check for error from zpool_vdev_online

Onlining a vdev can fail. Log the error if it does.

Reviewed by: mav, asomers
MFC after: 1 week
Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D30882

Merge OpenZFS support in to HEAD.

The primary benefit is maintaining a completely shared
code base with the community allowing FreeBSD to receive
new features sooner and with less effort.

I would advise against doing 'zpool upgrade'
or creating indispensable pools using new
features until this change has had a month+
to soak.

Work on merging FreeBSD support in to what was
at the time "ZFS on Linux" began in August 2018.
I first publicly proposed transitioning FreeBSD
to (new) OpenZFS on December 18th, 2018. FreeBSD
support in OpenZFS was finally completed in December
2019. A CFT for downstreaming OpenZFS support in
to FreeBSD was first issued on July 8th. All issues
that were reported have been addressed or, for
a couple of less critical matters there are
pull requests in progress with OpenZFS. iXsystems
has tested and dogfooded extensively internally.
The TrueNAS 12 release is based on OpenZFS with
some additional features that have not yet made
it upstream.

Improvements include:
project quotas, encrypted datasets,
allocation classes, vectorized raidz,
vectorized checksums, various command line
improvements, zstd compression.

Thanks to those who have helped along the way:
Ryan Moeller, Allan Jude, Zack Welch, and many
others.

Sponsored by: iXsystems, Inc.
Differential Revision: https://reviews.freebsd.org/D25872

Optimize zfsd for the happy case

If there are no damaged pools, then ignore all GEOM events. We only use
them to fix damaged pools. However, still pay attention to ZFS events.

MFC after: 20 days
X-MFC-With: 329284
Sponsored by: Spectra Logic Corp

zfsd: Don't spare a vdev that's being replaced

If a zfs pool contains a replacing vdev (either created manually by "zpool
replace" or by zfsd(8) via autoreplace by physical path) and then new spares
get added to the pool, zfsd shouldn't use one to replace the drive that is
already being replaced. That's a waste of resources that just slows down
the rebuild.

PR: 225547
MFC after: 3 weeks
Sponsored by: Spectra Logic Corp

zfsd should be able to online an L2ARC that disappears and returns

Previously, this didn't work because L2ARC devices' labels don't contain
pool GUIDs. Modify zfsd so that the pool GUID won't be required:

lib/libdevdctl/guid.h
Change INVALID_GUID from a uint64_t constant to a function that
returns an invalid Guid object. Remove the void constructor.
Nothing uses it, and it violates RAII.

cddl/usr.sbin/zfsd/case_file.h
cddl/usr.sbin/zfsd/case_file.cc
Allow CaseFile::Find to match a CaseFile based on Vdev GUID alone.
In CaseFile::ReEvaluate, attempt to online devices even if the newly
arrived device has no pool GUID.

cddl/usr.sbin/zfsd/vdev_iterator.cc
Iterate through a pool's cache devices as well as its regular
devices.

Reported by: avg
Reviewed by: avg
MFC after: 3 weeks
Sponsored by: Spectra Logic Corp
Differential Revision: https://reviews.freebsd.org/D12791

Fix an unchecked return value in zfsd

It's pretty unlikely to actually hit this, but good to check it anyway

Reported by: Coverity
CID: 1362018
MFC after: 4 weeks
Sponsored by: Spectra Logic Corp

Coverity fixes for r300906

lib/libdevdctl/consumer.cc
In Consumer::DisconnectFromDevd, don't close the socket if it's
already closed.

cddl/usr.sbin/zfsd/case_file.cc
lib/libdevdctl/consumer.h
Delete dead code leftover from before devd(8) gained SOCK_SEQPACKET
support

Reported by: Coverity
CID: 1356155, 1356169
Sponsored by: Spectra Logic Corp

Avoid more literal-suffix errors with C++11

zfsd(8), the ZFS fault management daemon

Add zfsd, which deals with hard drive faults in ZFS pools. It manages
hotspares and replements in drive slots that publish physical paths.

cddl/usr.sbin/zfsd
Add zfsd(8) and its unit tests

cddl/usr.sbin/Makefile
Add zfsd to the build

lib/libdevdctl
A C++ library that helps devd clients process events

lib/Makefile
share/mk/bsd.libnames.mk
share/mk/src.libnames.mk
Add libdevdctl to the build. It's a private library, unusable by
out-of-tree software.

etc/defaults/rc.conf
By default, set zfsd_enable to NO

etc/mtree/BSD.include.dist
Add a directory for libdevdctl's include files

etc/mtree/BSD.tests.dist
Add a directory for zfsd's unit tests

etc/mtree/BSD.var.dist
Add /var/db/zfsd/cases, where zfsd stores case files while it's shut
down.

etc/rc.d/Makefile
etc/rc.d/zfsd
Add zfsd's rc script

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c
Fix the resource.fs.zfs.statechange message. It had a number of
problems:

It was only being emitted on a transition to the HEALTHY state.
That made it impossible for zfsd to take actions based on drives
getting sicker.

It compared the new state to vdev_prevstate, which is the state that
the vdev had the last time it was opened. That doesn't make sense,
because a vdev can change state multiple times without being
reopened.

vdev_set_state contains logic that will change the device's new
state based on various conditions. However, the statechange event
was being posted _before_ that logic took effect. Now it's being
posted after.

Submitted by: gibbs, asomers, mav, allanjude
Reviewed by: mav, delphij
Relnotes: yes
Sponsored by: Spectra Logic Corp, iX Systems
Differential Revision: https://reviews.freebsd.org/D6564