sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

crypto: Remove unused devclass arguments to DRIVER_MODULE.

cryptosoft: Avoid referencing end-of-buffer cursors

Once a crypto cursor has reached the end of its buffer, it is invalid to
call crypto_cursor_segment() for at least some crypto buffer types.
Reorganize loops to avoid this.

Fixes: cfb7b942bed7 ("cryptosoft: Use multi-block encrypt/decrypt for non-AEAD ciphers.")
Fixes: a221a8f4a0de ("cryptosoft: Use multi-block encrypt/decrypt for AES-GCM.")
Fixes: f8580fcaa1e1 ("cryptosoft: Use multi-block encrypt/decrypt for AES-CCM.")
Fixes: 5022c68732e6 ("cryptosoft: Use multi-block encrypt/decrypt for ChaCha20-Poly1305.")
Reported and tested by: madpilot
Discussed with: jhb
Sponsored by: The FreeBSD Foundation

cryptosoft: Use multi-block encrypt/decrypt for ChaCha20-Poly1305.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33758

cryptosoft: Use multi-block encrypt/decrypt for AES-CCM.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33757

cryptosoft: Use multi-block encrypt/decrypt for AES-GCM.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33756

swcr_encdec: Rename blks to blksz.

This better reflects the variables purpose and matches other functions
in this file.

Requested by: markj
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33755

cryptosoft: Use multi-block encrypt/decrypt for non-AEAD ciphers.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33531

crypto: Add support for the XChaCha20-Poly1305 AEAD cipher.

This cipher is a wrapper around the ChaCha20-Poly1305 AEAD cipher
which accepts a larger nonce. Part of the nonce is used along with
the key as an input to HChaCha20 to generate a derived key used for
ChaCha20-Poly1305.

This cipher is used by WireGuard.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33523

OCF: Hook up plain RIPEMD160 in cryptosoft and /dev/crypto.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33612

cryptosoft: Use POLY1305_BLOCK_LEN instead of a magic number.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33528

cryptosoft: Consolidate calls to explicit_bzero.

Group sensitive on-stack variables into anonymous structs so that they
can be cleared with a single call to explicit_bzero rather than
multiple calls.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33527

cryptosoft: Reuse 'blk' to hold the initial nonce in swcr_gmac.

Reusing the storage removes the need for an additional explicit_bzero.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33526

cryptosoft: Remove unused 'iv' in swcr_ccm_cbc_mac.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33525

cryptosoft: Remove CBC handling from swcr_encdec.

All of the CBC ciphers now handle block chaining in the enc_xform.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33489

cryptosoft: Don't treat CRYPTO_NULL_HMAC as an hmac algorithm.

Despite it's name, CRYPTO_NULL_HMAC is actually an unkeyed digest
similar to plain SHA-1 rather than a keyed HMAC algorithm.

Reported by: syzbot+17d7c656f1a797b0faae@syzkaller.appspotmail.com
Discussed with: markj
Fixes: 6113a08b98e4 cryptosoft: Fully support per-operation keys for auth algorithms.
Sponsored by: The FreeBSD Foundation

cryptosoft: Stop single-threading requests within a session.

All of the request handlers no longer modify session state, so remove
the mutex limiting operations to one per session. In addition, change
the pointer to the session state passed to process callbacks to const.

Suggested by: mjg
Reviewed by: mjg, markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33317

cryptosoft: Fully support per-operation keys for auth algorithms.

Only pre-allocate auth contexts when a session-wide key is provided or
for sessions without keys. For sessions with per-operation keys,
always initialize the on-stack context directly rather than
initializing the session context in swcr_authprepare (now removed) and
then copying that session context into the on-stack context.

This approach permits parallel auth operations without needing a
serializing lock. In addition, the previous code assumed that auth
sessions always provided an initial key unlike cipher sessions which
assume either an initial key or per-op keys.

While here, fix the Blake2 auth transforms to function like other auth
transforms where Setkey is invoked after Init rather than before.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33316

cryptosoft: Allocate cipher contexts on the stack during operations.

As is done with authentication contexts, allocate cipher contexts on
the stack while completing requests. This permits safely dispatching
concurrent requests on a single session. The cipher context in the
session is now only allocated when a session key is provided during
session setup to serve as a template to initialize the on-stack
context similar to auth operations.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33198

cryptosoft: Rename sw_kschedule member to sw_ctx.

The cipher context isn't always a key schedule, so use a more generic
name.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33197

crypto: Refactor software support for AEAD ciphers.

Extend struct enc_xform to add new members to handle auth operations
for AEAD ciphers. In particular, AEAD operations in cryptosoft no
longer use a struct auth_hash. Instead, the setkey and reinit methods
of struct enc_xform are responsible for initializing both the cipher
and auth state.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33196

cryptosoft: Reject AES-CCM/GCM sessions with invalid key lengths.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33195

crypto: Validate AES-GCM IV length in check_csp().

This centralizes the check for valid nonce lengths for AES-GCM.

While here, remove some duplicate checks for valid AES-GCM tag lengths
from ccp(4) and ccr(4).

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33194

Fix "set but not used" in opencrypto

Sponsored by: Rubicon Communications, LLC ("Netgate")

crypto: Support Chacha20-Poly1305 with a nonce size of 8 bytes.

This is useful for WireGuard which uses a nonce of 8 bytes rather
than the 12 bytes used for IPsec and TLS.

Note that this also fixes a (should be) harmless bug in ossl(4) where
the counter was incorrectly treated as a 64-bit counter instead of a
32-bit counter in terms of wrapping when using a 12 byte nonce.
However, this required a single message (TLS record) longer than 64 *
(2^32 - 1) bytes (about 256 GB) to trigger.

Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32122

cryptosoft: Fix support for variable tag lengths in AES-CCM.

The tag length is included as one of the values in the flags byte of
block 0 passed to CBC_MAC, so merely copying the first N bytes is
insufficient.

To avoid adding more sideband data to the CBC MAC software context,
pull the generation of block 0, the AAD length, and AAD padding out of
cbc_mac.c and into cryptosoft.c. This matches how GCM/GMAC are
handled where the length block is constructed in cryptosoft.c and
passed as an input to the Update callback. As a result, the CBC MAC
Update() routine is now much simpler and simply performs the
XOR-and-encrypt step on each input block.

While here, avoid a copy to the staging block in the Update routine
when one or more full blocks are passed as input to the Update
callback.

Reviewed by: sef
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32120

crypto: Support multiple nonce lengths for AES-CCM.

Permit nonces of lengths 7 through 13 in the OCF framework and the
cryptosoft driver. A helper function (ccm_max_payload_length) can be
used in OCF drivers to reject CCM requests which are too large for the
specified nonce length.

Reviewed by: sef
Sponsored by: Chelsio Communications, The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32111

cryptosoft, ccr: Use crp_iv directly for AES-CCM and AES-GCM.

Rather than copying crp_iv to a local array on the stack that is then
passed to xform reinit routines, pass crp_iv directly and remove the
local copy.

Reviewed by: markj
Sponsored by: Chelsio Communications, The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32106

crypto: Permit variable-sized IVs for ciphers with a reinit hook.

Add a 'len' argument to the reinit hook in 'struct enc_xform' to
permit support for AEAD ciphers such as AES-CCM and Chacha20-Poly1305
which support different nonce lengths.

Reviewed by: markj
Sponsored by: Chelsio Communications, The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32105

crypto: Constify all transform descriptors

No functional change intended.

Reviewed by: ae, jhb
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31196

cryptosoft: Use crypto_cursor_segment().

Reviewed by: markj
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D30446

crypto: Add a new type of crypto buffer for a single mbuf.

This is intended for use in KTLS transmit where each TLS record is
described by a single mbuf that is itself queued in the socket buffer.
Using the existing CRYPTO_BUF_MBUF would result in
bus_dmamap_load_crp() walking additional mbufs in the socket buffer
that are not relevant, but generating a S/G list that potentially
exceeds the limit of the tag (while also wasting CPU cycles).

Reviewed by: markj
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D30136

opencrypto: Make cryptosoft attach silently

cryptosoft is always present and doesn't print any useful information
when it attaches.

Reviewed by: jhb
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D29098

cryptosoft: Support per-op keys for AES-GCM and AES-CCM.

Reviewed by: cem
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D28752

Add an implementation of CHACHA20_POLY1305 to cryptosoft.

This uses the chacha20 IETF and poly1305 implementations from
libsodium. A seperate auth_hash is created for the auth side whose
Setkey method derives the poly1305 key from the AEAD key and nonce as
described in RFC 8439.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D27837

Consistently use C99 fixed-width types in the in-kernel crypto code.

Reviewed by: markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D27061

Add support for ESN in cryptosoft

This patch adds support for IPsec ESN (Extended Sequence Numbers) in
encrypt and authenticate mode (eg. AES-CBC and SHA256) and combined mode
(eg. AES-GCM).

For encrypt and authenticate mode the ESN is stored in separate crp_esn
buffer because the high-order 32 bits of the sequence number are
appended after the Next Header (RFC 4303).

For combined modes the high-order 32 bits of the sequence number [e.g.
RFC 4106, Chapter 5 AAD Construction] are part of crp_aad (prepared by
netipsec layer in case of ESN support enabled), therefore non visible
diff around combined modes.

Submitted by: Grzegorz Jaszczyk <jaz@semihalf.com>
Patryk Duda <pdk@semihalf.com>
Reviewed by: jhb
Differential revision: https://reviews.freebsd.org/D22364
Obtained from: Semihalf
Sponsored by: Stormshield

Simplify swcr_authcompute() after removal of deprecated algorithms.

- Just use sw->octx != NULL to handle the HMAC case when finalizing
the MAC.

- Explicitly zero the on-stack auth context.

Reviewed by: markj
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D26688

crypto(9): add CRYPTO_BUF_VMPAGE

crypto(9) functions can now be used on buffers composed of an array of
vm_page_t structures, such as those stored in an unmapped struct bio. It
requires the running to kernel to support the direct memory map, so not all
architectures can use it.

Reviewed by: markj, kib, jhb, mjg, mat, bcr (manpages)
MFC after: 1 week
Sponsored by: Axcient
Differential Revision: https://reviews.freebsd.org/D25671

Use zfree() instead of explicit_bzero() and free().

In addition to reducing lines of code, this also ensures that the full
allocation is always zeroed avoiding possible bugs with incorrect
lengths passed to explicit_bzero().

Suggested by: cem
Reviewed by: cem, delphij
Approved by: csprng (cem)
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D25435

Add support to the crypto framework for separate AAD buffers.

This permits requests to provide the AAD in a separate side buffer
instead of as a region in the crypto request input buffer. This is
useful when the main data buffer might not contain the full AAD
(e.g. for TLS or IPsec with ESN).

Unlike separate IVs which are constrained in size and stored in an
array in struct cryptop, separate AAD is provided by the caller
setting a new crp_aad pointer to the buffer. The caller must ensure
the pointer remains valid and the buffer contents static until the
request is completed (e.g. when the callback routine is invoked).

As with separate output buffers, not all drivers support this feature.
Consumers must request use of this feature via a new session flag.

To aid in driver testing, kern.crypto.cryptodev_separate_aad can be
set to force /dev/crypto requests to use a separate AAD buffer.

Discussed with: cem
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D25288

Various optimizations to software AES-CCM and AES-GCM.

- Make use of cursors to avoid data copies for AES-CCM and AES-GCM.

Pass pointers into the request's input and/or output buffers
directly to the Update, encrypt, and decrypt hooks rather than
always copying all data into a temporary block buffer on the stack.

- Move handling for partial final blocks out of the main loop.

This removes branches from the main loop and permits using
encrypt/decrypt_last which avoids a memset to clear the rest of the
block on the stack.

- Shrink the on-stack buffers to assume AES block sizes and CCM/GCM
tag lengths.

- For AAD data, pass larger chunks to axf->Update. CCM can take each
AAD segment in a single call. GMAC can take multiple blocks at a
time.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D25058

Adjust crypto_apply function callbacks for OCF.

- crypto_apply() is only used for reading a buffer to compute a
digest, so change the data pointer to a const pointer.

- To better match m_apply(), change the data pointer type to void *
and the length from uint16_t to u_int. The length field in
particular matters as none of the apply logic was splitting requests
larger than UINT16_MAX.

- Adjust the auth_xform Update callback to match the function
prototype passed to crypto_apply() and crypto_apply_buf(). This
removes the needs for casts when using the Update callback.

- Change the Reinit and Setkey callbacks to also use a u_int length
instead of uint16_t.

- Update auth transforms for the changes. While here, use C99
initializers for auth_hash structures and avoid casts on callbacks.

Reviewed by: cem
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D25171

Add explicit bzero's of sensitive data in software crypto consumers.

Explicitly zero IVs, block buffers, and hashes/digests.

Reviewed by: delphij
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D25057

Add support for optional separate output buffers to in-kernel crypto.

Some crypto consumers such as GELI and KTLS for file-backed sendfile
need to store their output in a separate buffer from the input.
Currently these consumers copy the contents of the input buffer into
the output buffer and queue an in-place crypto operation on the output
buffer. Using a separate output buffer avoids this copy.

- Create a new 'struct crypto_buffer' describing a crypto buffer
containing a type and type-specific fields. crp_ilen is gone,
instead buffers that use a flat kernel buffer have a cb_buf_len
field for their length. The length of other buffer types is
inferred from the backing store (e.g. uio_resid for a uio).
Requests now have two such structures: crp_buf for the input buffer,
and crp_obuf for the output buffer.

- Consumers now use helper functions (crypto_use_*,
e.g. crypto_use_mbuf()) to configure the input buffer. If an output
buffer is not configured, the request still modifies the input
buffer in-place. A consumer uses a second set of helper functions
(crypto_use_output_*) to configure an output buffer.

- Consumers must request support for separate output buffers when
creating a crypto session via the CSP_F_SEPARATE_OUTPUT flag and are
only permitted to queue a request with a separate output buffer on
sessions with this flag set. Existing drivers already reject
sessions with unknown flags, so this permits drivers to be modified
to support this extension without requiring all drivers to change.

- Several data-related functions now have matching versions that
operate on an explicit buffer (e.g. crypto_apply_buf,
crypto_contiguous_subsegment_buf, bus_dma_load_crp_buf).

- Most of the existing data-related functions operate on the input
buffer. However crypto_copyback always writes to the output buffer
if a request uses a separate output buffer.

- For the regions in input/output buffers, the following conventions
are followed:
- AAD and IV are always present in input only and their
fields are offsets into the input buffer.
- payload is always present in both buffers. If a request uses a
separate output buffer, it must set a new crp_payload_start_output
field to the offset of the payload in the output buffer.
- digest is in the input buffer for verify operations, and in the
output buffer for compute operations. crp_digest_start is relative
to the appropriate buffer.

- Add a crypto buffer cursor abstraction. This is a more general form
of some bits in the cryptosoft driver that tried to always use uio's.
However, compared to the original code, this avoids rewalking the uio
iovec array for requests with multiple vectors. It also avoids
allocate an iovec array for mbufs and populating it by instead walking
the mbuf chain directly.

- Update the cryptosoft(4) driver to support separate output buffers
making use of the cursor abstraction.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D24545

Improve support for stream ciphers in the software encryption interface.

Add a 'native_blocksize' member to 'struct enc_xform' that ciphers can
use if they support a partial final block. This is particular useful
for stream ciphers, but can also apply to other ciphers. cryptosoft
will only pass in native blocks to the encrypt and decrypt hooks. For
the final partial block, 'struct enc_xform' now has new
encrypt_last/decrypt_last hooks which accept the length of the final
block. The multi_block methods are also retired.

Mark AES-ICM (AES-CTR) as a stream cipher. This has some interesting
effects on IPsec in that FreeBSD can now properly receive all packets
sent by Linux when using AES-CTR, but FreeBSD can no longer
interoperate with OpenBSD and older verisons of FreeBSD which assume
AES-CTR packets have a payload padded to a 16-byte boundary. Kornel
has offered to work on a patch to add a compatiblity sysctl to enforce
additional padding for AES-CTR in esp_output to permit compatibility
with OpenBSD and older versions of FreeBSD.

AES-XTS continues to use a block size of a single AES block length.
It is possible to adjust it to support partial final blocks by
implementing cipher text stealing via encrypt_last/decrypt_last hooks,
but I have not done so.

Reviewed by: cem (earlier version)
Tested by: Kornel Dulęba <mindal@semihalf.com> (AES-CTR with IPsec)
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D24906

Various cleanups to the software encryption transform interface.

- Consistently use 'void *' for key schedules / key contexts instead
of a mix of 'caddr_t', 'uint8_t *', and 'void *'.

- Add a ctxsize member to enc_xform similar to what auth transforms use
and require callers to malloc/zfree the context. The setkey callback
now supplies the caller-allocated context pointer and the zerokey
callback is removed. Callers now always use zfree() to ensure
key contexts are zeroed.

- Consistently use C99 initializers for all statically-initialized
instances of 'struct enc_xform'.

- Change the encrypt and decrypt functions to accept separate in and
out buffer pointers. Almost all of the backend crypto functions
already supported separate input and output buffers and this makes
it simpler to support separate buffers in OCF.

- Remove xform_userland.h shim to permit transforms to be compiled in
userland. Transforms no longer call malloc/free directly.

Reviewed by: cem (earlier version)
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D24855

Remove MD5 HMAC from OCF.

There are no in-kernel consumers.

Reviewed by: cem
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24775

Remove support for the Blowfish algorithm from OCF.

It no longer has any in-kernel consumers.

Reviewed by: cem
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24772

Remove support for keyed MD5 and SHA1 authentication hashes.

They no longer have any in-tree consumers. Note that these are a
different from MD5-HMAC and SHA1-HMAC and were only used with IPsec.

Reviewed by: cem
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24770

Remove support for the skipjack encryption algorithm.

This was removed from IPsec in r286100 and no longer has any in-tree
consumers.

Reviewed by: cem
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24769

Remove support for the cast128 encryption algorithm.

It no longer has any in-tree consumers.

Reviewed by: cem
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24768

Remove incomplete support for plain MD5 from OCF.

Although a few drivers supported this algorithm, there were never any
in-kernel consumers. cryptosoft and cryptodev never supported it,
and there was not a software xform auth_hash for it.

Reviewed by: cem
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24767

Retire the CRYPTO_F_IV_GENERATE flag.

The sole in-tree user of this flag has been retired, so remove this
complexity from all drivers. While here, add a helper routine drivers
can use to read the current request's IV into a local buffer. Use
this routine to replace duplicated code in nearly all drivers.

Reviewed by: cem
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D24450

Refactor driver and consumer interfaces for OCF (in-kernel crypto).

- The linked list of cryptoini structures used in session
initialization is replaced with a new flat structure: struct
crypto_session_params. This session includes a new mode to define
how the other fields should be interpreted. Available modes
include:

- COMPRESS (for compression/decompression)
- CIPHER (for simply encryption/decryption)
- DIGEST (computing and verifying digests)
- AEAD (combined auth and encryption such as AES-GCM and AES-CCM)
- ETA (combined auth and encryption using encrypt-then-authenticate)

Additional modes could be added in the future (e.g. if we wanted to
support TLS MtE for AES-CBC in the kernel we could add a new mode
for that. TLS modes might also affect how AAD is interpreted, etc.)

The flat structure also includes the key lengths and algorithms as
before. However, code doesn't have to walk the linked list and
switch on the algorithm to determine which key is the auth key vs
encryption key. The 'csp_auth_*' fields are always used for auth
keys and settings and 'csp_cipher_*' for cipher. (Compression
algorithms are stored in csp_cipher_alg.)

- Drivers no longer register a list of supported algorithms. This
doesn't quite work when you factor in modes (e.g. a driver might
support both AES-CBC and SHA2-256-HMAC separately but not combined
for ETA). Instead, a new 'crypto_probesession' method has been
added to the kobj interface for symmteric crypto drivers. This
method returns a negative value on success (similar to how
device_probe works) and the crypto framework uses this value to pick
the "best" driver. There are three constants for hardware
(e.g. ccr), accelerated software (e.g. aesni), and plain software
(cryptosoft) that give preference in that order. One effect of this
is that if you request only hardware when creating a new session,
you will no longer get a session using accelerated software.
Another effect is that the default setting to disallow software
crypto via /dev/crypto now disables accelerated software.

Once a driver is chosen, 'crypto_newsession' is invoked as before.

- Crypto operations are now solely described by the flat 'cryptop'
structure. The linked list of descriptors has been removed.

A separate enum has been added to describe the type of data buffer
in use instead of using CRYPTO_F_* flags to make it easier to add
more types in the future if needed (e.g. wired userspace buffers for
zero-copy). It will also make it easier to re-introduce separate
input and output buffers (in-kernel TLS would benefit from this).

Try to make the flags related to IV handling less insane:

- CRYPTO_F_IV_SEPARATE means that the IV is stored in the 'crp_iv'
member of the operation structure. If this flag is not set, the
IV is stored in the data buffer at the 'crp_iv_start' offset.

- CRYPTO_F_IV_GENERATE means that a random IV should be generated
and stored into the data buffer. This cannot be used with
CRYPTO_F_IV_SEPARATE.

If a consumer wants to deal with explicit vs implicit IVs, etc. it
can always generate the IV however it needs and store partial IVs in
the buffer and the full IV/nonce in crp_iv and set
CRYPTO_F_IV_SEPARATE.

The layout of the buffer is now described via fields in cryptop.
crp_aad_start and crp_aad_length define the boundaries of any AAD.
Previously with GCM and CCM you defined an auth crd with this range,
but for ETA your auth crd had to span both the AAD and plaintext
(and they had to be adjacent).

crp_payload_start and crp_payload_length define the boundaries of
the plaintext/ciphertext. Modes that only do a single operation
(COMPRESS, CIPHER, DIGEST) should only use this region and leave the
AAD region empty.

If a digest is present (or should be generated), it's starting
location is marked by crp_digest_start.

Instead of using the CRD_F_ENCRYPT flag to determine the direction
of the operation, cryptop now includes an 'op' field defining the
operation to perform. For digests I've added a new VERIFY digest
mode which assumes a digest is present in the input and fails the
request with EBADMSG if it doesn't match the internally-computed
digest. GCM and CCM already assumed this, and the new AEAD mode
requires this for decryption. The new ETA mode now also requires
this for decryption, so IPsec and GELI no longer do their own
authentication verification. Simple DIGEST operations can also do
this, though there are no in-tree consumers.

To eventually support some refcounting to close races, the session
cookie is now passed to crypto_getop() and clients should no longer
set crp_sesssion directly.

- Assymteric crypto operation structures should be allocated via
crypto_getkreq() and freed via crypto_freekreq(). This permits the
crypto layer to track open asym requests and close races with a
driver trying to unregister while asym requests are in flight.

- crypto_copyback, crypto_copydata, crypto_apply, and
crypto_contiguous_subsegment now accept the 'crp' object as the
first parameter instead of individual members. This makes it easier
to deal with different buffer types in the future as well as
separate input and output buffers. It's also simpler for driver
writers to use.

- bus_dmamap_load_crp() loads a DMA mapping for a crypto buffer.
This understands the various types of buffers so that drivers that
use DMA do not have to be aware of different buffer types.

- Helper routines now exist to build an auth context for HMAC IPAD
and OPAD. This reduces some duplicated work among drivers.

- Key buffers are now treated as const throughout the framework and in
device drivers. However, session key buffers provided when a session
is created are expected to remain alive for the duration of the
session.

- GCM and CCM sessions now only specify a cipher algorithm and a cipher
key. The redundant auth information is not needed or used.

- For cryptosoft, split up the code a bit such that the 'process'
callback now invokes a function pointer in the session. This
function pointer is set based on the mode (in effect) though it
simplifies a few edge cases that would otherwise be in the switch in
'process'.

It does split up GCM vs CCM which I think is more readable even if there
is some duplication.

- I changed /dev/crypto to support GMAC requests using CRYPTO_AES_NIST_GMAC
as an auth algorithm and updated cryptocheck to work with it.

- Combined cipher and auth sessions via /dev/crypto now always use ETA
mode. The COP_F_CIPHER_FIRST flag is now a no-op that is ignored.
This was actually documented as being true in crypto(4) before, but
the code had not implemented this before I added the CIPHER_FIRST
flag.

- I have not yet updated /dev/crypto to be aware of explicit modes for
sessions. I will probably do that at some point in the future as well
as teach it about IV/nonce and tag lengths for AEAD so we can support
all of the NIST KAT tests for GCM and CCM.

- I've split up the exising crypto.9 manpage into several pages
of which many are written from scratch.

- I have converted all drivers and consumers in the tree and verified
that they compile, but I have not tested all of them. I have tested
the following drivers:

- cryptosoft
- aesni (AES only)
- blake2
- ccr

and the following consumers:

- cryptodev
- IPsec
- ktls_ocf
- GELI (lightly)

I have not tested the following:

- ccp
- aesni with sha
- hifn
- kgssapi_krb5
- ubsec
- padlock
- safe
- armv8_crypto (aarch64)
- glxsb (i386)
- sec (ppc)
- cesa (armv7)
- cryptocteon (mips64)
- nlmsec (mips64)

Discussed with: cem
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D23677

Add AES-CCM encryption, and plumb into OCF.

This commit essentially has three parts:

* Add the AES-CCM encryption hooks. This is in and of itself fairly small,
as there is only a small difference between CCM and the other ICM-based
algorithms.
* Hook the code into the OpenCrypto framework. This is the bulk of the
changes, as the algorithm type has to be checked for, and the differences
between it and GCM dealt with.
* Update the cryptocheck tool to be aware of it. This is invaluable for
confirming that the code works.

This is a software-only implementation, meaning that the performance is very
low.

Sponsored by: iXsystems Inc.
Differential Revision: https://reviews.freebsd.org/D19090

Plug memory leak for AES_*_NIST_GMAC algorithms.

swcr_newsession() allocates sw_ictx for these algorithms, thus we need
to free() it in swcr_freesession().

PR: 233907
MFC after: 1 week

Add per-session locking to cryptosoft (swcr).

As part of ZFS Crypto, I started getting a series of panics when I did not
have AESNI loaded. Adding locking fixed it, and I concluded that the
Reinit function altered the AES key schedule. This locking is not as
fine-grained as it could be (AESNI uses per-cpu locking), but
it's minimally invasive.

Sponsored by: iXsystems Inc
Reviewed by: cem, mav
Approved by: re (gjb), mav (mentor)
Differential Revision: https://reviews.freebsd.org/D17307

cryptosoft: Reduce generality of supported algorithm composition

Fix a regression introduced in r336439.

Rather than allowing any linked list of algorithms, allow at most two
(typically, some combination of encrypt and/or MAC). Removes a WAITOK
malloc in an unsleepable context (classic LOR) by placing both software
algorithm contexts within the OCF-managed session object.

Tested with 'cryptocheck -a all -d cryptosoft0', which includes some
encrypt-and-MAC modes.

PR: 230304
Reported by: sef@

crypto(4): Add cryptosoft, cryptodev support for Poly-1305

OpenCrypto: Convert sessions to opaque handles instead of integers

Track session objects in the framework, and pass handles between the
framework (OCF), consumers, and drivers. Avoid redundancy and complexity in
individual drivers by allocating session memory in the framework and
providing it to drivers in ::newsession().

Session handles are no longer integers with information encoded in various
high bits. Use of the CRYPTO_SESID2FOO() macros should be replaced with the
appropriate crypto_ses2foo() function on the opaque session handle.

Convert OCF drivers (in particular, cryptosoft, as well as myriad others) to
the opaque handle interface. Discard existing session tracking as much as
possible (quick pass). There may be additional code ripe for deletion.

Convert OCF consumers (ipsec, geom_eli, krb5, cryptodev) to handle-style
interface. The conversion is largely mechnical.

The change is documented in crypto.9.

Inspired by
https://lists.freebsd.org/pipermail/freebsd-arch/2018-January/018835.html .

No objection from: ae (ipsec portion)
Reported by: jhb

OCF: Add plain hash modes

In part, to support OpenSSL's use of cryptodev, which puts the HMAC pieces
in software and only offloads the raw hash primitive.

The following cryptodev identifiers are added:

* CRYPTO_RIPEMD160 (not hooked up)
* CRYPTO_SHA2_224
* CRYPTO_SHA2_256
* CRYPTO_SHA2_384
* CRYPTO_SHA2_512

The plain SHA1 and 2 hashes are plumbed through cryptodev (feels like there
is a lot of redundancy here...) and cryptosoft.

This adds new auth_hash implementations for the plain hashes, as well as
SHA1 (which had a cryptodev.h identifier, but no implementation).

Add plain SHA 1 and 2 hash tests to the cryptocheck tool.

Motivation stems from John Baldwin's earlier OCF email,
https://lists.freebsd.org/pipermail/freebsd-arch/2018-January/018835.html .

OCF: Add CRYPTO_SHA2_224_HMAC mode

Round out the complete set of basic SHA2 HMAC modes with SHA2-224.

Support is added to the cryptocheck test tool.

cryptosoft: Do not exceed crd_len around *crypt_multi

When a caller passes in a uio or mbuf chain that is longer than crd_len, in
tandem with a transform that supports the multi-block interface,
swcr_encdec() would process the entire mbuf or uio instead of just the
portion indicated by crd_len (+ crd_skip).

De/encryption are performed in-place, so this would trash subsequent uio or
mbuf contents.

This was introduced in r331639 (mea culpa). It only affects the
{de,en}crypt_multi() family of interfaces. That interface only has one
consumer transform in-tree (for now): Chacha20.

PR: 227605
Submitted by: Valentin Vergez <valentin.vergez AT stormshield.eu>

cryptosoft: Remove a dead store

Introduced in r331639 by removing an instance of undefined behavior.

While we're here, the variable scope can be entirely moved inside the loop.

Reported by: Coverity
CID: 1387985
Sponsored by: Dell EMC Isilon

opencrypto: Integrate Chacha20 algorithm into OCF

Mostly this is a thin shim around existing code to integrate with enc_xform
and cryptosoft (+ cryptodev).

Expand the cryptodev buffer used to match that of Chacha20's native block
size as a performance enhancement for chacha20_xform_crypt_multi.

opencrypto: Add mechanism to pass multiple crypto blocks to some ciphers

xforms that support processing of multiple blocks at a time (to support more
efficient modes, for example) can define the encrypt_ and decrypt_multi
interfaces. If these interfaces are not present, the generic cryptosoft
code falls back on the block-at-a-time encrypt/decrypt interfaces.

Stream ciphers may support arbitrarily sized inputs (equivalent to an input
block size of 1 byte) but may be more efficient if a larger block is passed.

Sponsored by: Dell EMC Isilon

Import Blake2 algorithms (blake2b, blake2s) from libb2

The upstream repository is on github BLAKE2/libb2. Files landed in
sys/contrib/libb2 are the unmodified upstream files, except for one
difference: secure_zero_memory's contents have been replaced with
explicit_bzero() only because the previous implementation broke powerpc
link. Preferential use of explicit_bzero() is in progress upstream, so
it is anticipated we will be able to drop this diff in the future.

sys/crypto/blake2 contains the source files needed to port libb2 to our
build system, a wrapped (limited) variant of the algorithm to match the API
of our auth_transform softcrypto abstraction, incorporation into the Open
Crypto Framework (OCF) cryptosoft(4) driver, as well as an x86 SSE/AVX
accelerated OCF driver, blake2(4).

Optimized variants of blake2 are compiled for a number of x86 machines
(anything from SSE2 to AVX + XOP). On those machines, FPU context will need
to be explicitly saved before using blake2(4)-provided algorithms directly.
Use via cryptodev / OCF saves FPU state automatically, and use via the
auth_transform softcrypto abstraction does not use FPU.

The intent of the OCF driver is mostly to enable testing in userspace via
/dev/crypto. ATF tests are added with published KAT test vectors to
validate correctness.

Reviewed by: jhb, markj
Obtained from: github BLAKE2/libb2
Differential Revision: https://reviews.freebsd.org/D14662

cryptosoft(4): Zero plain hash contexts, too

An OCF-naive user program could use these primitives to implement HMAC, for
example. This would make the freed context sensitive data.

Probably other bzeros in this file should be explicit_bzeros as well.
Future work.

Reviewed by: jhb, markj
Differential Revision: https://reviews.freebsd.org/D14662 (minor part of a larger work)

Remove a redunant check.

Do pass removing some write-only variables from the kernel.

This reduces noise when kernel is compiled by newer GCC versions,
such as one used by external toolchain ports.

Reviewed by: kib, andrew(sys/arm and sys/arm64), emaste(partial), erj(partial)
Reviewed by: jhb (sys/dev/pci/* sys/kern/vfs_aio.c and sys/kern/kern_synch.c)
Differential Revision: https://reviews.freebsd.org/D10385

Don't leak a session and lock if a GMAC key has an invalid length.

Reviewed by: delphij (secteam)
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D10273

Make IPsec work with AES-GCM and AES-ICM (aka CTR) in OCF... IPsec
defines the keys differently than NIST does, so we have to muck with
key lengths and nonce/IVs to be standard compliant...

Remove the iv from secasvar as it was unused...

Add a counter protected by a mutex to ensure that the counter for GCM
and ICM will never be repeated.. This is a requirement for security..
I would use atomics, but we don't have a 64bit one on all platforms..

Fix a bug where IPsec was depending upon the OCF to ensure that the
blocksize was always at least 4 bytes to maintain alignment... Move
this logic into IPsec so changes to OCF won't break IPsec...

In one place, espx was always non-NULL, so don't test that it's
non-NULL before doing work..

minor style cleanups...

drop setting key and klen as they were not used...

Enforce that OCF won't pass invalid key lengths to AES that would
panic the machine...

This was has been tested by others too... I tested this against
NetBSD 6.1.5 using mini-test suite in
https://github.com/jmgurney/ipseccfgs and the only things that don't
pass are keyed md5 and sha1, and 3des-deriv (setkey syntax error),
all other modes listed in setkey's man page... The nice thing is
that NetBSD uses setkey, so same config files were used on both...

Reviewed by: gnn

we may get here w/ non-sleepable locks held, so switch to _NOWAIT when
doing this memory allocation...

Reviewed by: ae

Add some new modes to OpenCrypto. These modes are AES-ICM (can be used
for counter mode), and AES-GCM. Both of these modes have been added to
the aesni module.

Included is a set of tests to validate that the software and aesni
module calculate the correct values. These use the NIST KAT test
vectors. To run the test, you will need to install a soon to be
committed port, nist-kat that will install the vectors. Using a port
is necessary as the test vectors are around 25MB.

All the man pages were updated. I have added a new man page, crypto.7,
which includes a description of how to use each mode. All the new modes
and some other AES modes are present. It would be good for someone
else to go through and document the other modes.

A new ioctl was added to support AEAD modes which AES-GCM is one of them.
Without this ioctl, it is not possible to test AEAD modes from userland.

Add a timing safe bcmp for use to compare MACs. Previously we were using
bcmp which could leak timing info and result in the ability to forge
messages.

Add a minor optimization to the aesni module so that single segment
mbufs don't get copied and instead are updated in place. The aesni
module needs to be updated to support blocked IO so segmented mbufs
don't have to be copied.

We require that the IV be specified for all calls for both GCM and ICM.
This is to ensure proper use of these functions.

Obtained from: p4: //depot/projects/opencrypto
Relnotes: yes
Sponsored by: FreeBSD Foundation
Sponsored by: NetGate

some minor clean up.. Always _ZERO memory so mtx_init won't panic...

use the proper macro instead of hand rolling it...

Reviewed by: jhb (only the malloc change)
MFC after: 1 week

Prevent races in accesses of the software crypto session array.

swcr_newsession can change the pointer for swcr_sessions which races with
swcr_process which is looking up entries in this array.

Add a rwlock that protects changes to the array pointer so that
swcr_newsession and swcr_process no longer race.

Original patch by: Steve O'Hara-Smith <Steve.OHaraSmith@isilon.com>
Reviewed by: jmg
Sponsored by: EMC / Isilon Storage Division

Let cryptosoft(4) add its pseudo-device with a specific unit number and its
probe method return BUS_PROBE_NOWILDCARD so it doesn't get attached to real
devices hanging off of nexus(4) with no specific devclass set. Actually, the
more desirable fix for this would be to get rid of the newbus interface of
cryptosoft(4) altogether but apparently crypto(9) was written with support
for cryptographic hardware in mind so that approach would require some KPI
breaking changes which don't seem worth it.

MFC after: 1 week

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Add support for AES-XTS.

Obtained from: OpenBSD
MFC after: 1 week

MFC r201898:
Add comments trying to explain what bad things happen here, i.e.
how hashed MD5/SHA are implemented, abusing Final() for padding and
sw_octx to transport the key from the beginning to the end.

Enlightened about what was going on here by: cperciva
Reviewed by: cperciva

Add comments trying to explain what bad things happen here, i.e.
how hashed MD5/SHA are implemented, abusing Final() for padding and
sw_octx to transport the key from the beginning to the end.

Enlightened about what was going on here by: cperciva
Reviewed by: cperciva
MFC After: 3 days
X-MFC with: r187826
PR: kern/126468

MFC r199906:

In case the compression result is the same size as the orignal version,
the compression was useless as well. Make sure to not update the data
and return, else we would waste resources when decompressing.

This also avoids the copyback() changing data other consumers like
xform_ipcomp.c would have ignored because of no win and sent out without
noting that compression was used, resulting in invalid packets at the
receiver.

In case the compression result is the same size as the orignal version,
the compression was useless as well. Make sure to not update the data
and return, else we would waste resources when decompressing.

This also avoids the copyback() changing data other consumers like
xform_ipcomp.c would have ignored because of no win and sent out without
noting that compression was used, resulting in invalid packets at the
receiver.

MFC after: 5 days

Fix return type for detach routine (should be int)
Fix first parameter for identify routine (should be driver_t *)

While OpenBSD's crypto/ framework has sha1 and md5 implementations that
can cope with a result buffer of NULL in the "Final" function, we cannot.
Thus pass in a temporary buffer long enough for either md5 or sha1 results
so that we do not panic.

PR: bin/126468
MFC after: 1 week

Don't hang if encrypting/decrypting using struct iovecs where one of the
iovecs ends on a crypto block boundary.

Retire the MALLOC and FREE macros. They are an abomination unto style(9).

MFC after: 3 months

- Import the HEAD csup code which is the basis for the cvsmode work.

Integrate the Camellia Block Cipher. For more information see RFC 4132
and its bibliography.

Submitted by: Tomoyuki Okazaki <okazaki at kick dot gr dot jp>
MFC after: 1 month

Overhaul driver/subsystem api's:
o make all crypto drivers have a device_t; pseudo drivers like the s/w
crypto driver synthesize one
o change the api between the crypto subsystem and drivers to use kobj;
cryptodev_if.m defines this api
o use the fact that all crypto drivers now have a device_t to add support
for specifying which of several potential devices to use when doing
crypto operations
o add new ioctls that allow user apps to select a specific crypto device
to use (previous ioctls maintained for compatibility)
o overhaul crypto subsystem code to eliminate lots of cruft and hide
implementation details from drivers
o bring in numerous fixes from Michale Richardson/hifn; mostly for
795x parts
o add an optional mechanism for mmap'ing the hifn 795x public key h/w
to user space for use by openssl (not enabled by default)
o update crypto test tools to use new ioctl's and add cmd line options
to specify a device to use for tests

These changes will also enable much future work on improving the core
crypto subsystem; including proper load balancing and interposing code
between the core and drivers to dispatch small operations to the s/w
driver as appropriate.

These changes were instigated by the work of Michael Richardson.

Reviewed by: pjd
Approved by: re

Use newly added functions to simplify the code.

Move COPYDATA() and COPYBACK() macros to cryptodev.h, they will be used
in padlock(4) as well.

Rename HMAC_BLOCK_MAXLEN to HMAC_MAX_BLOCK_LEN to be consistent with
EALG_MAX_BLOCK_LEN.

Rename AALG_MAX_RESULT_LEN to HASH_MAX_LEN to look more constent with
other defines.

Kill an unused argument.

- Fix a very old bug in HMAC/SHA{384,512}. When HMAC is using SHA384
or SHA512, the blocksize is 128 bytes, not 64 bytes as anywhere else.
The bug also exists in NetBSD, OpenBSD and various other independed
implementations I look at.
- We cannot decide which hash function to use for HMAC based on the key
length, because any HMAC function can use any key length.
To fix it split CRYPTO_SHA2_HMAC into three algorithm:
CRYPTO_SHA2_256_HMAC, CRYPTO_SHA2_384_HMAC and CRYPTO_SHA2_512_HMAC.
Those names are consistent with OpenBSD's naming.
- Remove authsize field from auth_hash structure.
- Allow consumer to define size of hash he wants to receive.
This allows to use HMAC not only for IPsec, where 96 bits MAC is requested.
The size of requested MAC is defined at newsession time in the cri_mlen
field - when 0, entire MAC will be returned.
- Add swcr_authprepare() function which prepares authentication key.
- Allow to provide key for every authentication operation, not only at
newsession time by honoring CRD_F_KEY_EXPLICIT flag.
- Make giving key at newsession time optional - don't try to operate on it
if its NULL.
- Extend COPYBACK()/COPYDATA() macros to handle CRYPTO_BUF_CONTIG buffer
type as well.
- Accept CRYPTO_BUF_IOV buffer type in swcr_authcompute() as we have
cuio_apply() now.
- 16 bits for key length (SW_klen) is more than enough.

Reviewed by: sam

- Simplify the code by using arc4rand(9) instead of arc4random(9) in a loop.
- Correct a comment.

MFC after: 2 weeks

Fix memory leak which occurs when crypto.ko module is unloaded.

Discussed with: sam
MFC after 3 days

refer opencrypto/cast.h directly.

/* -> /*- for license, minor formatting changes

Add CRD_F_KEY_EXPLICIT which allows the key to be changed per
operation, just like it was possible to change the IV.

Currently supported on Hifn and software engines only.

Approved by: sam@

style(9) pass and type fixups.

Submitted by: bde

Push m_apply() and m_getptr() up into the colleciton of standard mbuf
routines, and purge them from opencrypto.

Reviewed by: sam
Obtained from: NetBSD
Sponsored by: spc.org

Add support to eliminate a context switch per crypto op when using the
software crypto device:

o record crypto device capabilities in each session id
o add a capability that indicates if the crypto driver operates synchronously
o tag the software crypto driver as operating synchronously

This commit also introduces crypto session id macros that cleanup their
construction and querying.

Use __FBSDID().

Change iov_base's type from `char *' to the standard `void *'. All
uses of iov_base which assume its type is `char *' (in order to do
pointer arithmetic) have been updated to cast iov_base to `char *'.

In-kernel crypto framework derived from openbsd. This facility provides
a consistent interface to h/w and s/w crypto algorithms for use by the
kernel and (for h/w at least) by user-mode apps. Access for user-level
code is through a /dev/crypto device that'll eventually be used by openssl
to (potentially) accelerate many applications. Coming soon is an IPsec
that makes use of this service to accelerate ESP, AH, and IPCOMP protocols.

Included here is the "core" crypto support, /dev/crypto driver, various
crypto algorithms that are not already present in the KAME crypto area,
and support routines used by crypto device drivers.

Obtained from: openbsd