sys: Automated cleanup of cdefs and other formatting

Apply the following automated changes to try to eliminate
no-longer-needed sys/cdefs.h includes as well as now-empty
blank lines in a row.

Remove /^#if.*\n#endif.*\n#include\s+<sys/cdefs.h>.*\n/
Remove /\n+#include\s+<sys/cdefs.h>.*\n+#if.*\n#endif.*\n+/
Remove /\n+#if.*\n#endif.*\n+/
Remove /^#if.*\n#endif.*\n/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/types.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/param.h>/
Remove /\n+#include\s+<sys/cdefs.h>\n#include\s+<sys/capsicum.h>/

Sponsored by: Netflix

sys: Remove $FreeBSD$: one-line .c pattern

Remove /^[\s*]*__FBSDID\("\$FreeBSD\$"\);?\s*\n/

cryptodev: Use a private malloc type (M_CRYPTODEV) instead of M_XDATA.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33991

/dev/crypto: Store blocksize in cse rather than txform pointer.

Reviewed by: markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D33614

/dev/crypto: Minimize cipher-specific logic.

Rather than duplicating the switches in crypto_auth_hash() and
crypto_cipher(), copy the algorithm constants from the new session
ioctl into a csp directly which permits using the functions in
crypto.c.

Reviewed by: markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D33613

OCF: Hook up plain RIPEMD160 in cryptosoft and /dev/crypto.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33612

crypto: Consistently use AES instead of Rijndael128 for the AES-CBC cipher.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33486

crypto: Refactor software support for AEAD ciphers.

Extend struct enc_xform to add new members to handle auth operations
for AEAD ciphers. In particular, AEAD operations in cryptosoft no
longer use a struct auth_hash. Instead, the setkey and reinit methods
of struct enc_xform are responsible for initializing both the cipher
and auth state.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D33196

cryptodev: Allow some CIOCCRYPT operations with an empty payload.

If an operation would generate a MAC output (e.g. for digest operation
or for an AEAD or EtA operation), then an empty payload buffer is
valid. Only reject requests with an empty buffer for "plain" cipher
sessions.

Some of the AES-CCM NIST KAT vectors use an empty payload.

While here, don't advance crp_payload_start for requests that use an
empty payload with an inline IV. (*)

Reported by: syzbot+d4b94fbd9a44b032f428@syzkaller.appspotmail.com (*)
Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32109

cryptodev: Permit CIOCCRYPT for AEAD ciphers.

A request without AAD for an AEAD cipher can be submitted via
CIOCCRYPT rather than CIOCCRYPTAEAD.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32108

cryptodev: Permit explicit IV/nonce and MAC/tag lengths.

Add 'ivlen' and 'maclen' fields to the structure used for CIOGSESSION2
to specify the explicit IV/nonce and MAC/tag lengths for crypto
sessions. If these fields are zero, the default lengths are used.

This permits selecting an alternate nonce length for AEAD ciphers such
as AES-CCM which support multiple nonce leengths. It also supports
truncated MACs as input to AEAD or ETA requests.

Reviewed by: markj
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32107

cryptodev: Use 'csp' in the handlers for requests.

- Retire cse->mode and use csp->csp_mode instead.
- Use csp->csp_cipher_algorithm instead of the ivsize when checking
for the fixup for the IV length for AES-XTS.

Reviewed by: markj
Sponsored by: Chelsio Communications, The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D32103

opencrypto: Disallow requests which pass VERIFY_DIGEST without a MAC

Otherwise we can end up comparing the computed digest with an
uninitialized kernel buffer.

In cryptoaead_op() we already unconditionally fail the request if a
pointer to a digest buffer is not specified.

Based on a patch by Simran Kathpalia.

Reported by: syzkaller
Reviewed by: jhb
MFC after: 1 week
Pull Request: https://github.com/freebsd/freebsd-src/pull/529
Differential Revision: https://reviews.freebsd.org/D32124

crypto: Constify all transform descriptors

No functional change intended.

Reviewed by: ae, jhb
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D31196

cryptodev: Fix some input validation bugs

- When we do not have a separate IV, make sure that the IV length
specified by the session is not larger than the payload size.
- Disallow AEAD requests without a separate IV. crp_sanity() asserts
that CRYPTO_F_IV_SEPARATE is set for AEAD requests, and some (but not
all) drivers require it.
- Return EINVAL for AEAD requests if an IV is specified but the
transform does not expect one.

Reported by: syzbot+c9e8f6ff5cb7fa6a1250@syzkaller.appspotmail.com
Reported by: syzbot+007341439ae295cee74f@syzkaller.appspotmail.com
Reported by: syzbot+46e0cc42a428b3b0a40d@syzkaller.appspotmail.com
Reported by: syzbot+2c4d670173b8bdb947df@syzkaller.appspotmail.com
Reported by: syzbot+220faa5eeb4d47b23877@syzkaller.appspotmail.com
Reported by: syzbot+e83434b40f05843722f7@syzkaller.appspotmail.com
Reviewed by: jhb
MFC after: 3 days
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D30154

OCF: Remove support for asymmetric cryptographic operations.

There haven't been any non-obscure drivers that supported this
functionality and it has been impossible to test to ensure that it
still works. The only known consumer of this interface was the engine
in OpenSSL < 1.1. Modern OpenSSL versions do not include support for
this interface as it was not well-documented.

Reviewed by: cem
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D29736

Add an OCF algorithm for ChaCha20-Poly1305 AEAD.

Note that this algorithm implements the mode defined in RFC 8439.

Reviewed by: cem
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D27836

Remove the cloned file descriptors for /dev/crypto.

Crypto file descriptors were added in the original OCF import as a way
to provide per-open data (specifically the list of symmetric
sessions). However, this gives a bit of a confusing API where one has
to open /dev/crypto and then invoke an ioctl to obtain a second file
descriptor. This also does not match the API used with /dev/crypto on
other BSDs or with Linux's /dev/crypto driver.

Character devices have gained support for per-open data via cdevpriv
since OCF was imported, so use cdevpriv to simplify the userland API
by permitting ioctls directly on /dev/crypto descriptors.

To provide backwards compatibility, CRIOGET now opens another
/dev/crypto descriptor via kern_openat() rather than dup'ing the
existing file descriptor. This preserves prior semantics in case
CRIOGET is invoked multiple times on a single file descriptor.

Reviewed by: markj
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D27302

Group session management routines together before first use.

- Rename cse*() to cse_*() to more closely match other local APIs in
this file.

- Merge the old csecreate() into cryptodev_create_session() and rename
the new function to cse_create().

Reviewed by: markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D27070

Move cryptof_ioctl() below the routines it calls.

Reviewed by: markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D27069

Split logic to create new sessions into a separate function.

This simplifies cryptof_ioctl as it now a wrapper around functions that
contain the bulk of the per-ioctl logic.

Reviewed by: markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D27068

Move cryptodev_cb earlier before it is used.

This is consistent with cryptodevkey_cb being defined before it is used
and removes a prototype in the middle of the file.

Reviewed by: markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D27067

Style fixes for function prototypes and definitions.

Reviewed by: markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D27066

Don't modify the destination pointer in ioctl requests.

This breaks the case where the original pointer was NULL but an
in-line IV was used.

Reviewed by: markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D27064

Consistently use C99 fixed-width types in the in-kernel crypto code.

Reviewed by: markj
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D27061

Mark asymmetric cryptography via OCF deprecated for 14.0.

Only one MIPS-specific driver implements support for one of the
asymmetric operations. There are no in-kernel users besides
/dev/crypto. The only known user of the /dev/crypto interface was the
engine in OpenSSL releases before 1.1.0. 1.1.0 includes a rewritten
engine that does not use the asymmetric operations due to lack of
documentation.

Reviewed by: cem, markj
MFC after: 1 week
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D26810

Name the on-stack union of compat thunks.

C does not permit an anonymous union at a top-level scope.

Pointy hat to: jhb

Add freebsd32 compat support for CIOCCRYPTAEAD.

Reviewed by: markj (earlier version)
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D26179

Simplify compat shims for /dev/crypto.

- Make session handling always use the CIOGSESSION2 structure.
CIOGSESSION requests use a thunk similar to COMPAT_FREEBSD32 session
requests. This permits the ioctl handler to use the 'crid' field
unconditionally.

- Move COMPAT_FREEBSD32 handling out of the main ioctl handler body
and instead do conversions in/out of thunk structures in dedicated
blocks at the start and end of the ioctl function.

Reviewed by: markj (earlier version)
Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D26178

crypto(9): Stop checking for failures from malloc(M_WAITOK).

PR: 240545
Submitted by: Andrew Reiter <arr@watson.org>
Reviewed by: cem, delphij, jhb
MFC after: 1 week
Event: July 2020 Bugathon

Remove unused 32-bit compatibility structures from cryptodev.

The counters are exported by a sysctl and have the same width on all
platforms anyway.

Reviewed by: cem, delphij, jhb
Sponsored by: Rubicon Communications, LLC (Netgate)
Differential Revision: https://reviews.freebsd.org/D25465

Add support to the crypto framework for separate AAD buffers.

This permits requests to provide the AAD in a separate side buffer
instead of as a region in the crypto request input buffer. This is
useful when the main data buffer might not contain the full AAD
(e.g. for TLS or IPsec with ESN).

Unlike separate IVs which are constrained in size and stored in an
array in struct cryptop, separate AAD is provided by the caller
setting a new crp_aad pointer to the buffer. The caller must ensure
the pointer remains valid and the buffer contents static until the
request is completed (e.g. when the callback routine is invoked).

As with separate output buffers, not all drivers support this feature.
Consumers must request use of this feature via a new session flag.

To aid in driver testing, kern.crypto.cryptodev_separate_aad can be
set to force /dev/crypto requests to use a separate AAD buffer.

Discussed with: cem
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D25288

Add a sysctl knob to use separate output buffers for /dev/crypto.

This is a testing aid to permit using testing a driver's support of
separate output buffers via cryptocheck.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D24545

Add support for optional separate output buffers to in-kernel crypto.

Some crypto consumers such as GELI and KTLS for file-backed sendfile
need to store their output in a separate buffer from the input.
Currently these consumers copy the contents of the input buffer into
the output buffer and queue an in-place crypto operation on the output
buffer. Using a separate output buffer avoids this copy.

- Create a new 'struct crypto_buffer' describing a crypto buffer
containing a type and type-specific fields. crp_ilen is gone,
instead buffers that use a flat kernel buffer have a cb_buf_len
field for their length. The length of other buffer types is
inferred from the backing store (e.g. uio_resid for a uio).
Requests now have two such structures: crp_buf for the input buffer,
and crp_obuf for the output buffer.

- Consumers now use helper functions (crypto_use_*,
e.g. crypto_use_mbuf()) to configure the input buffer. If an output
buffer is not configured, the request still modifies the input
buffer in-place. A consumer uses a second set of helper functions
(crypto_use_output_*) to configure an output buffer.

- Consumers must request support for separate output buffers when
creating a crypto session via the CSP_F_SEPARATE_OUTPUT flag and are
only permitted to queue a request with a separate output buffer on
sessions with this flag set. Existing drivers already reject
sessions with unknown flags, so this permits drivers to be modified
to support this extension without requiring all drivers to change.

- Several data-related functions now have matching versions that
operate on an explicit buffer (e.g. crypto_apply_buf,
crypto_contiguous_subsegment_buf, bus_dma_load_crp_buf).

- Most of the existing data-related functions operate on the input
buffer. However crypto_copyback always writes to the output buffer
if a request uses a separate output buffer.

- For the regions in input/output buffers, the following conventions
are followed:
- AAD and IV are always present in input only and their
fields are offsets into the input buffer.
- payload is always present in both buffers. If a request uses a
separate output buffer, it must set a new crp_payload_start_output
field to the offset of the payload in the output buffer.
- digest is in the input buffer for verify operations, and in the
output buffer for compute operations. crp_digest_start is relative
to the appropriate buffer.

- Add a crypto buffer cursor abstraction. This is a more general form
of some bits in the cryptosoft driver that tried to always use uio's.
However, compared to the original code, this avoids rewalking the uio
iovec array for requests with multiple vectors. It also avoids
allocate an iovec array for mbufs and populating it by instead walking
the mbuf chain directly.

- Update the cryptosoft(4) driver to support separate output buffers
making use of the cursor abstraction.

Sponsored by: Netflix
Differential Revision: https://reviews.freebsd.org/D24545

opencrypto: Add missing ioctl exit SDTs

The opencrypto ioctl code has very useful probe points at the various exit
points. These allow us to figure out exactly why a request failed. However, a
few paths did not have these probe points. Add them here.

Reviewed by: jhb

Remove incomplete support for plain MD5 from OCF.

Although a few drivers supported this algorithm, there were never any
in-kernel consumers. cryptosoft and cryptodev never supported it,
and there was not a software xform auth_hash for it.

Reviewed by: cem
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24767

Actually remove support for Triple DES, not just the warning.

Missed in r360562.

Remove support for the algorithms deprecated in r348876.

This removes support for the following algorithms:
- ARC4
- Blowfish
- CAST128
- DES
- 3DES
- MD5-HMAC
- Skipjack

Since /dev/crypto no longer supports 3DES, stop testing the 3DES KAT
vectors in cryptotest.py.

Reviewed by: cem (previous version)
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D24346

Avoid checking pointers that are never NULL.

Coverity noted that cod pointer is always non-NULL at the end of
cryptodev_aead(). While here, fix cryptodev_op() to match by making
one earlier failure case before cod and crp are allocated just return
directly.

CID: 1422185
Reported by: Coverity

Refactor driver and consumer interfaces for OCF (in-kernel crypto).

- The linked list of cryptoini structures used in session
initialization is replaced with a new flat structure: struct
crypto_session_params. This session includes a new mode to define
how the other fields should be interpreted. Available modes
include:

- COMPRESS (for compression/decompression)
- CIPHER (for simply encryption/decryption)
- DIGEST (computing and verifying digests)
- AEAD (combined auth and encryption such as AES-GCM and AES-CCM)
- ETA (combined auth and encryption using encrypt-then-authenticate)

Additional modes could be added in the future (e.g. if we wanted to
support TLS MtE for AES-CBC in the kernel we could add a new mode
for that. TLS modes might also affect how AAD is interpreted, etc.)

The flat structure also includes the key lengths and algorithms as
before. However, code doesn't have to walk the linked list and
switch on the algorithm to determine which key is the auth key vs
encryption key. The 'csp_auth_*' fields are always used for auth
keys and settings and 'csp_cipher_*' for cipher. (Compression
algorithms are stored in csp_cipher_alg.)

- Drivers no longer register a list of supported algorithms. This
doesn't quite work when you factor in modes (e.g. a driver might
support both AES-CBC and SHA2-256-HMAC separately but not combined
for ETA). Instead, a new 'crypto_probesession' method has been
added to the kobj interface for symmteric crypto drivers. This
method returns a negative value on success (similar to how
device_probe works) and the crypto framework uses this value to pick
the "best" driver. There are three constants for hardware
(e.g. ccr), accelerated software (e.g. aesni), and plain software
(cryptosoft) that give preference in that order. One effect of this
is that if you request only hardware when creating a new session,
you will no longer get a session using accelerated software.
Another effect is that the default setting to disallow software
crypto via /dev/crypto now disables accelerated software.

Once a driver is chosen, 'crypto_newsession' is invoked as before.

- Crypto operations are now solely described by the flat 'cryptop'
structure. The linked list of descriptors has been removed.

A separate enum has been added to describe the type of data buffer
in use instead of using CRYPTO_F_* flags to make it easier to add
more types in the future if needed (e.g. wired userspace buffers for
zero-copy). It will also make it easier to re-introduce separate
input and output buffers (in-kernel TLS would benefit from this).

Try to make the flags related to IV handling less insane:

- CRYPTO_F_IV_SEPARATE means that the IV is stored in the 'crp_iv'
member of the operation structure. If this flag is not set, the
IV is stored in the data buffer at the 'crp_iv_start' offset.

- CRYPTO_F_IV_GENERATE means that a random IV should be generated
and stored into the data buffer. This cannot be used with
CRYPTO_F_IV_SEPARATE.

If a consumer wants to deal with explicit vs implicit IVs, etc. it
can always generate the IV however it needs and store partial IVs in
the buffer and the full IV/nonce in crp_iv and set
CRYPTO_F_IV_SEPARATE.

The layout of the buffer is now described via fields in cryptop.
crp_aad_start and crp_aad_length define the boundaries of any AAD.
Previously with GCM and CCM you defined an auth crd with this range,
but for ETA your auth crd had to span both the AAD and plaintext
(and they had to be adjacent).

crp_payload_start and crp_payload_length define the boundaries of
the plaintext/ciphertext. Modes that only do a single operation
(COMPRESS, CIPHER, DIGEST) should only use this region and leave the
AAD region empty.

If a digest is present (or should be generated), it's starting
location is marked by crp_digest_start.

Instead of using the CRD_F_ENCRYPT flag to determine the direction
of the operation, cryptop now includes an 'op' field defining the
operation to perform. For digests I've added a new VERIFY digest
mode which assumes a digest is present in the input and fails the
request with EBADMSG if it doesn't match the internally-computed
digest. GCM and CCM already assumed this, and the new AEAD mode
requires this for decryption. The new ETA mode now also requires
this for decryption, so IPsec and GELI no longer do their own
authentication verification. Simple DIGEST operations can also do
this, though there are no in-tree consumers.

To eventually support some refcounting to close races, the session
cookie is now passed to crypto_getop() and clients should no longer
set crp_sesssion directly.

- Assymteric crypto operation structures should be allocated via
crypto_getkreq() and freed via crypto_freekreq(). This permits the
crypto layer to track open asym requests and close races with a
driver trying to unregister while asym requests are in flight.

- crypto_copyback, crypto_copydata, crypto_apply, and
crypto_contiguous_subsegment now accept the 'crp' object as the
first parameter instead of individual members. This makes it easier
to deal with different buffer types in the future as well as
separate input and output buffers. It's also simpler for driver
writers to use.

- bus_dmamap_load_crp() loads a DMA mapping for a crypto buffer.
This understands the various types of buffers so that drivers that
use DMA do not have to be aware of different buffer types.

- Helper routines now exist to build an auth context for HMAC IPAD
and OPAD. This reduces some duplicated work among drivers.

- Key buffers are now treated as const throughout the framework and in
device drivers. However, session key buffers provided when a session
is created are expected to remain alive for the duration of the
session.

- GCM and CCM sessions now only specify a cipher algorithm and a cipher
key. The redundant auth information is not needed or used.

- For cryptosoft, split up the code a bit such that the 'process'
callback now invokes a function pointer in the session. This
function pointer is set based on the mode (in effect) though it
simplifies a few edge cases that would otherwise be in the switch in
'process'.

It does split up GCM vs CCM which I think is more readable even if there
is some duplication.

- I changed /dev/crypto to support GMAC requests using CRYPTO_AES_NIST_GMAC
as an auth algorithm and updated cryptocheck to work with it.

- Combined cipher and auth sessions via /dev/crypto now always use ETA
mode. The COP_F_CIPHER_FIRST flag is now a no-op that is ignored.
This was actually documented as being true in crypto(4) before, but
the code had not implemented this before I added the CIPHER_FIRST
flag.

- I have not yet updated /dev/crypto to be aware of explicit modes for
sessions. I will probably do that at some point in the future as well
as teach it about IV/nonce and tag lengths for AEAD so we can support
all of the NIST KAT tests for GCM and CCM.

- I've split up the exising crypto.9 manpage into several pages
of which many are written from scratch.

- I have converted all drivers and consumers in the tree and verified
that they compile, but I have not tested all of them. I have tested
the following drivers:

- cryptosoft
- aesni (AES only)
- blake2
- ccr

and the following consumers:

- cryptodev
- IPsec
- ktls_ocf
- GELI (lightly)

I have not tested the following:

- ccp
- aesni with sha
- hifn
- kgssapi_krb5
- ubsec
- padlock
- safe
- armv8_crypto (aarch64)
- glxsb (i386)
- sec (ppc)
- cesa (armv7)
- cryptocteon (mips64)
- nlmsec (mips64)

Discussed with: cem
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D23677

Add stricter checking on mac key lengths.

Negative lengths are always invalid. The key length should also
be zero for hash algorithms that do not accept a key.

admbugs: 949
Reported by: Yuval Kanarenstein <yuvalk@ssd-disclosure.com>
Reviewed by: cem
MFC after: 1 week
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D23094

Remove no-longer-used function prototype.

Reported by: amd64-gcc

Remove unneeded cdevsw methods and D_NEEDGIANT.

Reviewed by: kib
MFC after: 2 weeks
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D23079

Use falloc_noinstall + finstall for crypto file descriptors.

Reviewed by: cem, kib
MFC after: 1 week
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D23078

Add a reference count to cryptodev sessions.

This prevents use-after-free races with crypto requests (which may
sleep) and CIOCFSESSION as well as races from current CIOCFSESSION
requests.

admbugs: 949
Reported by: Yuval Kanarenstein <yuvalk@ssd-disclosure.com>
Reviewed by: cem
MFC after: 1 week
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D23077

Adjust the deprecated warnings for /dev/crypto to be less noisy.

Warn when actual operations are performed instead of when sessions are
created. The /dev/crypto engine in OpenSSL 1.0.x tries to create
sessions for all possible algorithms each time it is initialized
resulting in spurious warnings.

Reported by: Mike Tancsa
MFC after: 3 days
Sponsored by: Chelsio Communications

Move declaration of warninterval out from under COMPAT_FREEBSD32.

This fixes builds of kernels without COMPAT_FREEBSD32.

Reported by: tinderbox
MFC after: 1 month

Make the warning intervals for deprecated crypto algorithms tunable.

New sysctl/tunables can now set the interval (in seconds) between
rate-limited crypto warnings. The new sysctls are:
- kern.cryptodev_warn_interval for /dev/crypto
- net.inet.ipsec.crypto_warn_interval for IPsec
- kern.kgssapi_warn_interval for KGSSAPI

Reviewed by: cem
MFC after: 1 month
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D20555

Add warnings to /dev/crypto for deprecated algorithms.

These algorithms are deprecated algorithms that will have no in-kernel
consumers in FreeBSD 13. Specifically, deprecate the following
algorithms:
- ARC4
- Blowfish
- CAST128
- DES
- 3DES
- MD5-HMAC
- Skipjack

MFC after: 1 month
Relnotes: yes
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D20554

Add AES-CCM encryption, and plumb into OCF.

This commit essentially has three parts:

* Add the AES-CCM encryption hooks. This is in and of itself fairly small,
as there is only a small difference between CCM and the other ICM-based
algorithms.
* Hook the code into the OpenCrypto framework. This is the bulk of the
changes, as the algorithm type has to be checked for, and the differences
between it and GCM dealt with.
* Update the cryptocheck tool to be aware of it. This is invaluable for
confirming that the code works.

This is a software-only implementation, meaning that the performance is very
low.

Sponsored by: iXsystems Inc.
Differential Revision: https://reviews.freebsd.org/D19090

crypto(4): Add cryptosoft, cryptodev support for Poly-1305

OpenCrypto: Convert sessions to opaque handles instead of integers

Track session objects in the framework, and pass handles between the
framework (OCF), consumers, and drivers. Avoid redundancy and complexity in
individual drivers by allocating session memory in the framework and
providing it to drivers in ::newsession().

Session handles are no longer integers with information encoded in various
high bits. Use of the CRYPTO_SESID2FOO() macros should be replaced with the
appropriate crypto_ses2foo() function on the opaque session handle.

Convert OCF drivers (in particular, cryptosoft, as well as myriad others) to
the opaque handle interface. Discard existing session tracking as much as
possible (quick pass). There may be additional code ripe for deletion.

Convert OCF consumers (ipsec, geom_eli, krb5, cryptodev) to handle-style
interface. The conversion is largely mechnical.

The change is documented in crypto.9.

Inspired by
https://lists.freebsd.org/pipermail/freebsd-arch/2018-January/018835.html .

No objection from: ae (ipsec portion)
Reported by: jhb

OCF: Add a typedef for session identifiers

No functional change.

This should ease the transition from an integer session identifier model to
an opaque pointer model.

OCF: Add plain hash modes

In part, to support OpenSSL's use of cryptodev, which puts the HMAC pieces
in software and only offloads the raw hash primitive.

The following cryptodev identifiers are added:

* CRYPTO_RIPEMD160 (not hooked up)
* CRYPTO_SHA2_224
* CRYPTO_SHA2_256
* CRYPTO_SHA2_384
* CRYPTO_SHA2_512

The plain SHA1 and 2 hashes are plumbed through cryptodev (feels like there
is a lot of redundancy here...) and cryptosoft.

This adds new auth_hash implementations for the plain hashes, as well as
SHA1 (which had a cryptodev.h identifier, but no implementation).

Add plain SHA 1 and 2 hash tests to the cryptocheck tool.

Motivation stems from John Baldwin's earlier OCF email,
https://lists.freebsd.org/pipermail/freebsd-arch/2018-January/018835.html .

OCF: Add CRYPTO_SHA2_224_HMAC mode

Round out the complete set of basic SHA2 HMAC modes with SHA2-224.

Support is added to the cryptocheck test tool.

Move most of the contents of opt_compat.h to opt_global.h.

opt_compat.h is mentioned in nearly 180 files. In-progress network
driver compabibility improvements may add over 100 more so this is
closer to "just about everywhere" than "only some files" per the
guidance in sys/conf/options.

Keep COMPAT_LINUX32 in opt_compat.h as it is confined to a subset of
sys/compat/linux/*.c. A fake _COMPAT_LINUX option ensure opt_compat.h
is created on all architectures.

Move COMPAT_LINUXKPI to opt_dontuse.h as it is only used to control the
set of compiled files.

Reviewed by: kib, cem, jhb, jtl
Sponsored by: DARPA, AFRL
Differential Revision: https://reviews.freebsd.org/D14941

opencrypto: Integrate Chacha20 algorithm into OCF

Mostly this is a thin shim around existing code to integrate with enc_xform
and cryptosoft (+ cryptodev).

Expand the cryptodev buffer used to match that of Chacha20's native block
size as a performance enhancement for chacha20_xform_crypt_multi.

cryptodev: Match intent for enc_xform ciphers with blocksize != ivsize

No functional change for Skipjack, AES-ICM, Blowfish, CAST-128, Camellia,
DES3, Rijndael128, DES. All of these have identical IV and blocksizes
declared in the associated enc_xform.

Functional changes for:
* AES-GCM: block len of 1, IV len of 12
* AES-XTS: block len of 16, IV len of 8
* NULL: block len of 4, IV len of 0

For these, it seems like the IV specified in the enc_xform is correct (and
the blocksize used before was wrong).

Additionally, the not-yet-OCFed cipher Chacha20 has a logical block length
of 1 byte, and a 16 byte IV + nonce.

Rationalize references to IV lengths to refer to the declared ivsize, rather
than declared blocksize.

Sponsored by: Dell EMC Isilon

Import Blake2 algorithms (blake2b, blake2s) from libb2

The upstream repository is on github BLAKE2/libb2. Files landed in
sys/contrib/libb2 are the unmodified upstream files, except for one
difference: secure_zero_memory's contents have been replaced with
explicit_bzero() only because the previous implementation broke powerpc
link. Preferential use of explicit_bzero() is in progress upstream, so
it is anticipated we will be able to drop this diff in the future.

sys/crypto/blake2 contains the source files needed to port libb2 to our
build system, a wrapped (limited) variant of the algorithm to match the API
of our auth_transform softcrypto abstraction, incorporation into the Open
Crypto Framework (OCF) cryptosoft(4) driver, as well as an x86 SSE/AVX
accelerated OCF driver, blake2(4).

Optimized variants of blake2 are compiled for a number of x86 machines
(anything from SSE2 to AVX + XOP). On those machines, FPU context will need
to be explicitly saved before using blake2(4)-provided algorithms directly.
Use via cryptodev / OCF saves FPU state automatically, and use via the
auth_transform softcrypto abstraction does not use FPU.

The intent of the OCF driver is mostly to enable testing in userspace via
/dev/crypto. ATF tests are added with published KAT test vectors to
validate correctness.

Reviewed by: jhb, markj
Obtained from: github BLAKE2/libb2
Differential Revision: https://reviews.freebsd.org/D14662

Move per-operation data out of the csession structure.

Create a struct cryptop_data which contains state needed for a single
symmetric crypto operation and move that state out of the session. This
closes a race with the CRYPTO_F_DONE flag that can result in use after
free.

While here, remove the 'cse->error' member. It was just a copy of
'crp->crp_etype' and cryptodev_op() and cryptodev_aead() checked both
'crp->crp_etype' and 'cse->error'. Similarly, do not check for an
error from mtx_sleep() since it is not used with PCATCH or a timeout
so cannot fail with an error.

PR: 218597
Reviewed by: kib
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D13928

Axe tmp_iv from the cryptodev session structure.

Just copyin the IV into the crypto descriptor directly. This avoids
copying the IV twice for each operation.

Reviewed by: kib
MFC after: 2 weeks
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D13847

Flesh out static dtrace probes for /dev/crypto ioctl errors.

In particular, no probes were present for AEAD requests, but also for
some other error cases in other ioctl requests.

MFC after: 2 weeks
Sponsored by: Chelsio Communications

opencrypto: Loosen restriction on HMAC key sizes

Theoretically, HMACs do not actually have any limit on key sizes.
Transforms should compact input keys larger than the HMAC block size by
using the transform (hash) on the input key.

(Short input keys are padded out with zeros to the HMAC block size.)

Still, not all FreeBSD crypto drivers that provide HMAC functionality
handle longer-than-blocksize keys appropriately, so enforce a "maximum" key
length in the crypto API for auth_hashes that previously expressed a
requirement. (The "maximum" is the size of a single HMAC block for the
given transform.) Unconstrained auth_hashes are left as-is.

I believe the previous hardcoded sizes were committed in the original
import of opencrypto from OpenBSD and are due to specific protocol
details of IPSec. Note that none of the previous sizes actually matched
the appropriate HMAC block size.

The previous hardcoded sizes made the SHA tests in cryptotest.py
useless for testing FreeBSD crypto drivers; none of the NIST-KAT example
inputs had keys sized to the previous expectations.

The following drivers were audited to check that they handled keys up to
the block size of the HMAC safely:

Software HMAC:
* padlock(4)
* cesa
* glxsb
* safe(4)
* ubsec(4)

Hardware accelerated HMAC:
* ccr(4)
* hifn(4)
* sec(4) (Only supports up to 64 byte keys despite claiming to
support SHA2 HMACs, but validates input key sizes)
* cryptocteon (MIPS)
* nlmsec (MIPS)
* rmisec (MIPS) (Amusingly, does not appear to use key material at
all -- presumed broken)

Reviewed by: jhb (previous version), rlibby (previous version)
Sponsored by: Dell EMC Isilon
Differential Revision: https://reviews.freebsd.org/D12437

Support AEAD requests with non-GCM algorithms.

In particular, support chaining an AES cipher with an HMAC for a request
including AAD. This permits submitting requests from userland to encrypt
objects like IPSec packets using these algorithms.

In the non-GCM case, the authentication crypto descriptor covers both the
AAD and the ciphertext. The GCM case remains unchanged. This matches
the requests created internally in IPSec. For the non-GCM case, the
COP_F_CIPHER_FIRST is also supported since the ordering matters.

Note that while this can be used to simulate IPSec requests from userland,
this ioctl cannot currently be used to perform TLS requests using AES-CBC
and MAC-before-encrypt.

Reviewed by: cem
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D11759

Add a new COP_F_CIPHER_FIRST flag for struct crypt_op.

This requests that the cipher be performed before rather than after
the HMAC when both are specified for a single operation.

Reviewed by: cem
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D11757

Place the AAD before the plaintext/ciphertext for CIOCRYPTAEAD.

Software crypto implementations don't care how the buffer is laid out,
but hardware implementations may assume that the AAD is always before
the plain/cipher text and that the hash/tag is immediately after the end
of the plain/cipher text.

In particular, this arrangement matches the layout of both IPSec packets
and TLS frames. Linux's crypto framework also assumes this layout for
AEAD requests.

Reviewed by: cem
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D11758

Remove unneeded includes of opt_kdtrace.h.

As of r258541, KDTRACE_HOOKS is defined in opt_global.h, so opt_kdtrace.h
is not needed when defining SDT(9) probes.

Add some new modes to OpenCrypto. These modes are AES-ICM (can be used
for counter mode), and AES-GCM. Both of these modes have been added to
the aesni module.

Included is a set of tests to validate that the software and aesni
module calculate the correct values. These use the NIST KAT test
vectors. To run the test, you will need to install a soon to be
committed port, nist-kat that will install the vectors. Using a port
is necessary as the test vectors are around 25MB.

All the man pages were updated. I have added a new man page, crypto.7,
which includes a description of how to use each mode. All the new modes
and some other AES modes are present. It would be good for someone
else to go through and document the other modes.

A new ioctl was added to support AEAD modes which AES-GCM is one of them.
Without this ioctl, it is not possible to test AEAD modes from userland.

Add a timing safe bcmp for use to compare MACs. Previously we were using
bcmp which could leak timing info and result in the ability to forge
messages.

Add a minor optimization to the aesni module so that single segment
mbufs don't get copied and instead are updated in place. The aesni
module needs to be updated to support blocked IO so segmented mbufs
don't have to be copied.

We require that the IV be specified for all calls for both GCM and ICM.
This is to ensure proper use of these functions.

Obtained from: p4: //depot/projects/opencrypto
Relnotes: yes
Sponsored by: FreeBSD Foundation
Sponsored by: NetGate

Fix build for kernels without COMPAT_FREEBSD32.

Add a new fo_fill_kinfo fileops method to add type-specific information to
struct kinfo_file.
- Move the various fill_*_info() methods out of kern_descrip.c and into the
various file type implementations.
- Rework the support for kinfo_ofile to generate a suitable kinfo_file object
for each file and then convert that to a kinfo_ofile structure rather than
keeping a second, different set of code that directly manipulates
type-specific file information.
- Remove the shm_path() and ksem_info() layering violations.

Differential Revision: https://reviews.freebsd.org/D775
Reviewed by: kib, glebius (earlier version)

Fix various issues with invalid file operations:
- Add invfo_rdwr() (for read and write), invfo_ioctl(), invfo_poll(),
and invfo_kqfilter() for use by file types that do not support the
respective operations. Home-grown versions of invfo_poll() were
universally broken (they returned an errno value, invfo_poll()
uses poll_no_poll() to return an appropriate event mask). Home-grown
ioctl routines also tended to return an incorrect errno (invfo_ioctl
returns ENOTTY).
- Use the invfo_*() functions instead of local versions for
unsupported file operations.
- Reorder fileops members to match the order in the structure definition
to make it easier to spot missing members.
- Add several missing methods to linuxfileops used by the OFED shim
layer: fo_write(), fo_truncate(), fo_kqfilter(), and fo_stat(). Most
of these used invfo_*(), but a dummy fo_stat() implementation was
added.

allow the kern.cryptodevallowsoft sysctl to enable symetric/hashes too...
This will allow us to more easily test the software versions of these
routines...

Considering that we've never had an software asymetric implmentation,
it's doubtful anyone has this enabled...

some minor clean up.. Always _ZERO memory so mtx_init won't panic...

use the proper macro instead of hand rolling it...

Reviewed by: jhb (only the malloc change)
MFC after: 1 week

Make sendfile() a method in the struct fileops. Currently only
vnode backed file descriptors have this method implemented.

Reviewed by: kib
Sponsored by: Nginx, Inc.
Sponsored by: Netflix

Revert previous commit...

Pointyhat to: kevlo (myself)

Prefer NULL over 0 for pointers

Add the fo_chown and fo_chmod methods to struct fileops and use them
to implement fchown(2) and fchmod(2) support for several file types
that previously lacked it. Add MAC entries for chown/chmod done on
posix shared memory and (old) in-kernel posix semaphores.

Based on the submission by: glebius
Reviewed by: rwatson
Approved by: re (bz)

After the r219999 is merged to stable/8, rename fallocf(9) to falloc(9)
and remove the falloc() version that lacks flag argument. This is done
to reduce the KPI bloat.

Requested by: jhb
X-MFC-note: do not

A new jail(8) with a configuration file, to replace the work currently done
by /etc/rc.d/jail.

Add support for AES-XTS.

Obtained from: OpenBSD
MFC after: 1 week

Fix typo in comment.

Add compat32 shims for opencrypto(4).

Reviewed by: bz
MFC after: 3 weeks

If crypto operation is finished with EAGAIN, don't repeat operation from
the return context, but from the original context.
Before repeating operation clear DONE flag and error.

Reviewed by: sam
Obtained from: Wheel Sp. z o.o. (http://www.wheel.pl)

Fix cryptodev UIO creation.

Cryptodev uses UIO structure do get data from userspace and pass it to
cryptographic engines. Initially UIO size is equal to size of data passed to
engine, but if UIO is prepared for hash calculation an additional small space
is created to hold result of operation.

While creating space for the result, UIO I/O vector size is correctly
extended, but uio_resid field in UIO structure is not modified.

As bus_dma code uses uio_resid field to determine size of UIO DMA mapping,
resulting mapping hasn't correct size. This leads to a crash if all the
following conditions are met:

1. Hardware cryptographic accelerator writes result of hash operation
using DMA.
2. Size of input data is less or equal than (n * PAGE_SIZE),
3. Size of input data plus size of hash result is grather than
(n * PAGE_SIZE, where n is the same as in point 2.

This patch fixes this problem by adding size of the extenstion to uio_resid
field in UIO structure.

Submitted by: Piotr Ziecik kosmo ! semihalf dot com
Reviewed by: philip
Obtained from: Semihalf

Fix a number of style issues in the MALLOC / FREE commit. I've tried to
be careful not to fix anything that was already broken; the NFSv4 code is
particularly bad in this respect.

Retire the MALLOC and FREE macros. They are an abomination unto style(9).

MFC after: 3 months

- Import the HEAD csup code which is the basis for the cvsmode work.

Make ftruncate a 'struct file' operation rather than a vnode operation.
This makes it possible to support ftruncate() on non-vnode file types in
the future.
- 'struct fileops' grows a 'fo_truncate' method to handle an ftruncate() on
a given file descriptor.
- ftruncate() moves to kern/sys_generic.c and now just fetches a file
object and invokes fo_truncate().
- The vnode-specific portions of ftruncate() move to vn_truncate() in
vfs_vnops.c which implements fo_truncate() for vnode file types.
- Non-vnode file types return EINVAL in their fo_truncate() method.

Submitted by: rwatson

Remove explicit locking of struct file.
- Introduce a finit() which is used to initailize the fields of struct file
in such a way that the ops vector is only valid after the data, type,
and flags are valid.
- Protect f_flag and f_count with atomic operations.
- Remove the global list of all files and associated accounting.
- Rewrite the unp garbage collection such that it no longer requires
the global list of all files and instead uses a list of all unp sockets.
- Mark sockets in the accept queue so we don't incorrectly gc them.

Tested by: kris, pho

Deny attempt to malloc unbounded amount of the memory.
Convert malloc()/bzero() to malloc(M_ZERO).

Obtained from: OpenBSD
MFC after: 3 days
Approved by: re (kensmith)

Integrate the Camellia Block Cipher. For more information see RFC 4132
and its bibliography.

Submitted by: Tomoyuki Okazaki <okazaki at kick dot gr dot jp>
MFC after: 1 month

Replace custom file descriptor array sleep lock constructed using a mutex
and flags with an sxlock. This leads to a significant and measurable
performance improvement as a result of access to shared locking for
frequent lookup operations, reduced general overhead, and reduced overhead
in the event of contention. All of these are imported for threaded
applications where simultaneous access to a shared file descriptor array
occurs frequently. Kris has reported 2x-4x transaction rate improvements
on 8-core MySQL benchmarks; smaller improvements can be expected for many
workloads as a result of reduced overhead.

- Generally eliminate the distinction between "fast" and regular
acquisisition of the filedesc lock; the plan is that they will now all
be fast. Change all locking instances to either shared or exclusive
locks.

- Correct a bug (pointed out by kib) in fdfree() where previously msleep()
was called without the mutex held; sx_sleep() is now always called with
the sxlock held exclusively.

- Universally hold the struct file lock over changes to struct file,
rather than the filedesc lock or no lock. Always update the f_ops
field last. A further memory barrier is required here in the future
(discussed with jhb).

- Improve locking and reference management in linux_at(), which fails to
properly acquire vnode references before using vnode pointers. Annotate
improper use of vn_fullpath(), which will be replaced at a future date.

In fcntl(), we conservatively acquire an exclusive lock, even though in
some cases a shared lock may be sufficient, which should be revisited.
The dropping of the filedesc lock in fdgrowtable() is no longer required
as the sxlock can be held over the sleep operation; we should consider
removing that (pointed out by attilio).

Tested by: kris
Discussed with: jhb, kris, attilio, jeff

Overhaul driver/subsystem api's:
o make all crypto drivers have a device_t; pseudo drivers like the s/w
crypto driver synthesize one
o change the api between the crypto subsystem and drivers to use kobj;
cryptodev_if.m defines this api
o use the fact that all crypto drivers now have a device_t to add support
for specifying which of several potential devices to use when doing
crypto operations
o add new ioctls that allow user apps to select a specific crypto device
to use (previous ioctls maintained for compatibility)
o overhaul crypto subsystem code to eliminate lots of cruft and hide
implementation details from drivers
o bring in numerous fixes from Michale Richardson/hifn; mostly for
795x parts
o add an optional mechanism for mmap'ing the hifn 795x public key h/w
to user space for use by openssl (not enabled by default)
o update crypto test tools to use new ioctl's and add cmd line options
to specify a device to use for tests

These changes will also enable much future work on improving the core
crypto subsystem; including proper load balancing and interposing code
between the core and drivers to dispatch small operations to the s/w
driver as appropriate.

These changes were instigated by the work of Michael Richardson.

Reviewed by: pjd
Approved by: re

Fix usage of HMAC algorithms via /dev/crypto.

- Fix a very old bug in HMAC/SHA{384,512}. When HMAC is using SHA384
or SHA512, the blocksize is 128 bytes, not 64 bytes as anywhere else.
The bug also exists in NetBSD, OpenBSD and various other independed
implementations I look at.
- We cannot decide which hash function to use for HMAC based on the key
length, because any HMAC function can use any key length.
To fix it split CRYPTO_SHA2_HMAC into three algorithm:
CRYPTO_SHA2_256_HMAC, CRYPTO_SHA2_384_HMAC and CRYPTO_SHA2_512_HMAC.
Those names are consistent with OpenBSD's naming.
- Remove authsize field from auth_hash structure.
- Allow consumer to define size of hash he wants to receive.
This allows to use HMAC not only for IPsec, where 96 bits MAC is requested.
The size of requested MAC is defined at newsession time in the cri_mlen
field - when 0, entire MAC will be returned.
- Add swcr_authprepare() function which prepares authentication key.
- Allow to provide key for every authentication operation, not only at
newsession time by honoring CRD_F_KEY_EXPLICIT flag.
- Make giving key at newsession time optional - don't try to operate on it
if its NULL.
- Extend COPYBACK()/COPYDATA() macros to handle CRYPTO_BUF_CONTIG buffer
type as well.
- Accept CRYPTO_BUF_IOV buffer type in swcr_authcompute() as we have
cuio_apply() now.
- 16 bits for key length (SW_klen) is more than enough.

Reviewed by: sam

If kern.cryptodevallowsoft is TRUE allow also for symmetric software crypto
in kernel. Useful for testing.

Reviewed by: sam

This patch fixes a problem, which exists if you have IPSEC in your kernel
and want to have crypto support loaded as KLD. By moving zlib to separate
module and adding MODULE_DEPEND directives, it is possible to use such
configuration without complication. Otherwise, since IPSEC is linked with
zlib (just like crypto.ko) you'll get following error:

interface zlib.1 already present in the KLD 'kernel'!

Approved by: cognet (mentor)

Fix bogus check. It was possible to panic the kernel by giving 0 length.
This is actually a local DoS, as every user can use /dev/crypto if there
is crypto hardware in the system and cryptodev.ko is loaded (or compiled
into the kernel).

Reported by: Mike Tancsa <mike@sentex.net>
MFC after: 1 day

Use dynamic major number allocation.

/* -> /*- for license, minor formatting changes

Push Giant down through ioctl.

Don't grab Giant in the upper syscall/wrapper code

NET_LOCK_GIANT in the socket code (sockets/fifos).

mtx_lock(&Giant) in the vnode code.

mtx_lock(&Giant) in the opencrypto code. (This may actually not be
needed, but better safe than sorry).

Devfs grabs Giant if the driver is marked as needing Giant.

Don't acquire Giant in cryptof_close(), as the code is intended to be
able to run MPsafe (and appears to be MPsafe).

Discussed with (some time ago): sam

Push acquisition of Giant from fdrop_closed() into fo_close() so that
individual file object implementations can optionally acquire Giant if
they require it:

- soo_close(): depends on debug.mpsafenet
- pipe_close(): Giant not acquired
- kqueue_close(): Giant required
- vn_close(): Giant required
- cryptof_close(): Giant required (conservative)

Notes:

Giant is still acquired in close() even when closing MPSAFE objects
due to kqueue requiring Giant in the calling closef() code.
Microbenchmarks indicate that this removal of Giant cuts 3%-3% off
of pipe create/destroy pairs from user space with SMP compiled into
the kernel.

The cryptodev and opencrypto code appears MPSAFE, but I'm unable to
test it extensively and so have left Giant over fo_close(). It can
probably be removed given some testing and review.

Do the dreaded s/dev_t/struct cdev */
Bump __FreeBSD_version accordingly.

add missing #include <sys/module.h>

Device megapatch 4/6:

Introduce d_version field in struct cdevsw, this must always be
initialized to D_VERSION.

Flip sense of D_NOGIANT flag to D_NEEDGIANT, this involves removing
four D_NOGIANT flags and adding 145 D_NEEDGIANT flags.

eliminate an unnecessary 8Kbyte bzero that was being done for each
submitted operation

Submitted by: Thor Lancelot Simon
Reviewed by: jhb
Approved by: re (jhb)

falloc allocates a file structure and adds it to the file descriptor
table, acquiring the necessary locks as it works. It usually returns
two references to the new descriptor: one in the descriptor table
and one via a pointer argument.

As falloc releases the FILEDESC lock before returning, there is a
potential for a process to close the reference in the file descriptor
table before falloc's caller gets to use the file. I don't think this
can happen in practice at the moment, because Giant indirectly protects
closes.

To stop the file being completly closed in this situation, this change
makes falloc set the refcount to two when both references are returned.
This makes life easier for several of falloc's callers, because the
first thing they previously did was grab an extra reference on the
file.

Reviewed by: iedowse
Idea run past: jhb

The present defaults for the open and close for device drivers which
provide no methods does not make any sense, and is not used by any
driver.

It is a pretty hard to come up with even a theoretical concept of
a device driver which would always fail open and close with ENODEV.

Change the defaults to be nullopen() and nullclose() which simply
does nothing.

Remove explicit initializations to these from the drivers which
already used them.

Initialize struct fileops with C99 sparse initialization.

Use __FBSDID().

Flush my local cache of cryto subsystem fixes:

o add a ``done'' flag for crypto operations; this is set when the operation
completes and is intended for callers to check operations that may complete
``prematurely'' because of direct callbacks
o close a race for operations where the crypto driver returns ERESTART: we
need to hold the q lock to insure the blocked state for the driver and any
driver-private state is consistent; otherwise drivers may take an interrupt
and notify the crypto subsystem that it can unblock the driver but operations
will be left queued and never be processed
o close a race in /dev/crypto where operations can complete before the caller
can sleep waiting for the callback: use a per-session mutex and the new done
flag to handle this
o correct crypto_dispatch's handling of operations where the driver returns
ERESTART: the return value must be zero and not ERESTART, otherwise the
caller may free the crypto request despite it being queued for later handling
(this typically results in a later panic)
o change crypto mutex ``names'' so witness printouts and the like are more
meaningful

Gigacommit to improve device-driver source compatibility between
branches:

Initialize struct cdevsw using C99 sparse initializtion and remove
all initializations to default values.

This patch is automatically generated and has been tested by compiling
LINT with all the fields in struct cdevsw in reverse order on alpha,
sparc64 and i386.

Approved by: re(scottl)

o add a CRYPTO_F_CBIMM flag to symmetric ops to indicate the callback
should be done in crypto_done rather than in the callback thread
o use this flag to mark operations from /dev/crypto since the callback
routine just does a wakeup; this eliminates the last unneeded ctx switch
o change CRYPTO_F_NODELAY to CRYPTO_F_BATCH with an inverted meaning
so "0" becomes the default/desired setting (needed for user-mode
compatibility with openbsd)
o change crypto_dispatch to honor CRYPTO_F_BATCH instead of always
dispatching immediately
o remove uses of CRYPTO_F_NODELAY
o define COP_F_BATCH for ops submitted through /dev/crypto and pass
this on to the op that is submitted

Similar changes and more eventually coming for asymmetric ops.

MFC if re gives approval.

Back out M_* changes, per decision of the TRB.

Approved by: trb

Remove M_TRYWAIT/M_WAITOK/M_WAIT. Callers should use 0.
Merge M_NOWAIT/M_DONTWAIT into a single flag M_NOWAIT.

Bow to the whining masses and change a union back into void *. Retain
removal of unnecessary casts and throw in some minor cleanups to see if
anyone complains, just for the hell of it.

Change struct file f_data to un_data, a union of the correct struct
pointer types, and remove a huge number of casts from code using it.

Change struct xfile xf_data to xun_data (ABI is still compatible).

If we need to add a #define for f_data and xf_data we can, but I don't
think it will be necessary. There are no operational changes in this
commit.

MFS: crypto timing support; purge usercrypto sysctl (just don't config
cryptodev or kldunload cryptodev module); crypto statistcs; remove
unused alloctype field from crypto op to offset addition of the
performance time stamp

Supported by: Vernier Networks

correct minor # in make_dev call

Submitted by: Doug Ambrisko" <ambrisko@verniernetworks.com>

Module-ize the 'core' crypto stuff. This may still need to be compiled
into the kernel by default (if required), but other modules can now
depend() on this.

Fix inter-module dependancy.

Earlier version OK'ed by: sam

remove CIOGSSESSION (get software session); it was added only for testing

In-kernel crypto framework derived from openbsd. This facility provides
a consistent interface to h/w and s/w crypto algorithms for use by the
kernel and (for h/w at least) by user-mode apps. Access for user-level
code is through a /dev/crypto device that'll eventually be used by openssl
to (potentially) accelerate many applications. Coming soon is an IPsec
that makes use of this service to accelerate ESP, AH, and IPCOMP protocols.

Included here is the "core" crypto support, /dev/crypto driver, various
crypto algorithms that are not already present in the KAME crypto area,
and support routines used by crypto device drivers.

Obtained from: openbsd