Copy stable/9 to releng/9.3 as part of the 9.3-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

MFC r246922 (pjd):

- Add support for 'memsync' mode. This is the fastest replication mode that's
why it will now be the default.
- Bump protocol version to 2 and add backward compatibility for version 1.
- Allow to specify hosts by kern.hostid as well (in addition to hostname and
kern.hostuuid) in configuration file.

Sponsored by: Panzura

MFC r236507, r237931, r238120, r238538:

r236507 (pjd):

Simplify the code by using snprlcat().

r237931 (pjd):

Check if there is cmsg at all.

r238120 (pjd):

Make use of GEOM Gate direct reads feature. This allows HAST to serve
reads with native speed of the underlying provider.
There are three situations when direct reads are not used:
1. Data is being synchronized and synchronization source is the secondary
node, which means secondary node has more recent data and we should read
from it.
2. Local read failed and we have to try to read from the secondary node.
3. Local component is unavailable and all I/O requests are served from the
secondary node.

Sponsored by: Panzura, http://www.panzura.com

r238538:

Metaflush on/off values don't need quotes.

Reviewed by: pjd

MFC r235337:

General mdoc(7) and typo fixes.

PR: 167804

MFC r229699, r229744, r229778, r229944, r229945, r229946, r230092, r230395,
r230396, r230436, r230457, r230515, r230976:

r229744 (pjd):

fork(2) returns -1 on failure, not some random negative number.

r229699 (pjd):

Constify argument.

r229778 (uqs):

Spelling fixes for sbin/

r229944 (pjd):

Don't touch pidfiles when running in foreground. Before that change we
would create an empty pidfile on start and check if it changed on SIGHUP.

r229945 (pjd):

For functions that return -1 on failure check exactly for -1 and not for
any negative number.

r229946 (pjd):

- Fix a bug where pidfile was removed in SIGHUP when it hasn't changed in
configuration file.
- Log the fact that pidfile has changed.

r230092 (pjd):

Style cleanups.

r230395 (pjd):

Remove unused token 'port'.

r230396 (pjd):

Remove another unused token.

r230436 (pjd):

Fix minor memory leak.

r230457 (pjd):

Free memory that won't be used in child.

r230515 (pjd):

- Fix documentation to note that /etc/hast.conf is the default configuration
file for hastd(8) and hastctl(8) and not hast.conf.
- In copyright statement correct that this file is documentation, not software.
- Bump date.

r230976 (pjd):

Fix typo in comment.

MFC r225773, r225781, r225782, r225783, r225784, 225785, r225786, r225787,
r225830, r225831, r225832, r225835, r226461, r226462, r226463, r226842,
r226851, r226852, r226854, r226855, r226856, r226857, r226859, r226861,
r228542, r228542, r228543, r228544, r228695, r228696:

r225773 (pjd):

Ensure that pjdlog functions don't modify errno.

r225781 (pjd):

No need to use KEEP_ERRNO() macro around pjdlog functions, as they don't
modify errno.

r225782 (pjd):

Prefer PJDLOG_ASSERT() and PJDLOG_ABORT() over assert() and abort().
pjdlog versions will log problem to syslog when application is running in
background.

r225783 (pjd):

Correct two mistakes when converting asserts to PJDLOG_ASSERT()/PJDLOG_ABORT().

r225784 (pjd):

- Convert some impossible conditions into assertions.
- Add missing 'if' in comment.

r225785 (pjd):

Prefer PJDLOG_ASSERT()/PJDLOG_ABORT() over assert().

r225786 (pjd):

No need to wrap pjdlog functions around with KEEP_ERRNO() macro.

r225787 (pjd):

Use PJDLOG_ASSERT() and PJDLOG_ABORT() everywhere instead of assert().

r225830 (pjd):

After every activemap change flush disk's write cache, so that write
reordering won't make the actual write to be committed before marking
the coresponding extent as dirty.

It can be disabled in configuration file.

If BIO_FLUSH is not supported by the underlying file system we log a warning
and never send BIO_FLUSH again to that GEOM provider.

r225831 (pjd):

Break a bit earlier.

r225832 (pjd):

If the underlying provider doesn't support BIO_FLUSH, log it only once
and don't bother trying in the future.

r225835 (pjd):

Correct typo.

r226461 (pjd):

When path to the configuration file is relative, obtain full path,
so we can always find the file, even after daemonizing and changing
working directory to /.

r226462 (pjd):

Remove redundant space.

r226463 (pjd):

Allow to specify pidfile in HAST configuration file.

r226842 (pjd):

Correct comments.

r226851 (pjd):

Delay resuid generation until first connection to secondary, not until first
write. This way on first connection we will synchronize only the extents that
were modified during the lifetime of primary node, not entire GEOM provider.

r226852 (pjd):

Minor cleanups.

r226854 (pjd):

- Eliminate the need for hio_nv.
- Introduce hio_clear() function for clearing hio before returning it
onto free queue.

r226855 (pjd):

Improve comment so it doesn't suggest race is possible, but that we handle
the race.

r226856 (pjd):

Reduce indentation.

r226857 (pjd):

Minor cleanups.

r226859 (pjd):

Implement 'async' mode for HAST.

r226861 (pjd):

Remove redundant space.

r228542 (pjd):

Remove redundant setting of the error variable.

Found by: Clang Static Analyzer

r228543 (pjd):

Simplify code by changing functions types from int to avoid, as the functions
always return 0.

Found by: Clang Static Analyzer

r228544 (pjd):

Remove redundant assignment.

Found by: Clang Static Analyzer

r228695 (pjd):

Don't use function name as format string.

Detected by: clang

r228696 (pjd):

Use lex's standard way of not generating unused function.

Inspired by: r228555

Copy head to stable/9 as part of 9.0-RELEASE release cycle.

Approved by: re (implicit)

Document IPv6 support.

MFC after: 3 weeks

In preparation for IPv6 support allow to specify multiple addresses to
listen on.

MFC after: 3 weeks

Increase default timeout from 5 seconds to 20 seconds. 5 seconds is definitely
to short under heavy load and I was experiencing those timeouts in my recent
tests.

MFC after: 1 week

In hast.conf we define the other node's address in 'remote' variable.
This way we know how to connect to secondary node when we are primary.
The same variable is used by the secondary node - it only accepts
connections from the address stored in 'remote' variable.
In cluster configurations it is common that each node has its individual
IP address and there is one addtional shared IP address which is assigned
to primary node. It seems it is possible that if the shared IP address is
from the same network as the individual IP address it might be choosen by
the kernel as a source address for connection with the secondary node.
Such connection will be rejected by secondary, as it doesn't come from
primary node individual IP.

Add 'source' variable that allows to specify source IP address we want to
bind to before connecting to the secondary node.

MFC after: 1 week

Allow to compress on-the-wire data using two algorithms:
- HOLE - it simply turns all-zero blocks into few bytes header;
it is extremely fast, so it is turned on by default;
it is mostly intended to speed up initial synchronization
where we expect many zeros;
- LZF - very fast algorithm by Marc Alexander Lehmann, which shows
very decent compression ratio and has BSD license.

MFC after: 2 weeks

Allow to checksum on-the-wire data using either CRC32 or SHA256.

MFC after: 2 weeks

Execute hook when connection between the nodes is established or lost.

MFC after: 2 weeks
Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com

Execute hook when split-brain is detected.

MFC after: 2 weeks
Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com

- Call hook on role change.
- Document new event.

MFC after: 2 weeks
Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com

Document new 'exec' parameter.

MFC after: 2 weeks
Obtained from: Wheel Systems Sp. z o.o. http://www.wheelsystems.com

Fix typos, spelling, formatting and mdoc mistakes found by Nobuyuki while
translating these manual pages. Minor corrections by me.

Submitted by: Nobuyuki Koganemaru <n-kogane@syd.odn.ne.jp>

Document 'none' value for remote.

Reviewed by: dougb
MFC after: 1 month

Spelling fixes.

Actually, only the fullsync mode is implemented, not memsync mode.
Correct manual page.

MFC after: 3 days

mdoc: move remaining sections into consistent order

This pertains mostly to FILES, HISTORY, EXIT STATUS and AUTHORS sections.

Found by: mdocml lint run
Reviewed by: ru

Fix a problem where hastd will stuck in recv(2) after sending request to
secondary, which died between send(2) and recv(2). Do it by adding timeout
to recv(2) for primary incoming and outgoing sockets and secondary outgoing
socket.

Reported by: Mikolaj Golub <to.my.trociny@gmail.com>
Tested by: Mikolaj Golub <to.my.trociny@gmail.com>
MFC after: 3 days

Please welcome HAST - Highly Avalable Storage.

HAST allows to transparently store data on two physically separated machines
connected over the TCP/IP network. HAST works in Primary-Secondary
(Master-Backup, Master-Slave) configuration, which means that only one of the
cluster nodes can be active at any given time. Only Primary node is able to
handle I/O requests to HAST-managed devices. Currently HAST is limited to two
cluster nodes in total.

HAST operates on block level - it provides disk-like devices in /dev/hast/
directory for use by file systems and/or applications. Working on block level
makes it transparent for file systems and applications. There in no difference
between using HAST-provided device and raw disk, partition, etc. All of them
are just regular GEOM providers in FreeBSD.

For more information please consult hastd(8), hastctl(8) and hast.conf(5)
manual pages, as well as http://wiki.FreeBSD.org/HAST.

Sponsored by: FreeBSD Foundation
Sponsored by: OMCnet Internet Service GmbH
Sponsored by: TransIP BV