Copy stable/10@r272459 to releng/10.1 as part of
the 10.1-RELEASE process.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Copy head (r256279) to stable/10 as part of the 10.0-RELEASE cycle.

Approved by: re (implicit)
Sponsored by: The FreeBSD Foundation

Use the fpu_kern_enter() interface to properly separate usermode FPU
context from in-kernel execution of padlock instructions and to handle
spurious FPUDNA exceptions that sometime are raised when doing padlock
calculations.

Globally mark crypto(9) kthread as using FPU.

Reviewed by: pjd
Hardware provided by: Sentex Communications
Tested by: pho
PR: amd64/135014
MFC after: 1 month

Define an SDT provider for "opencrypto".

MFC after: 6 days

Rename the kthread_xxx (e.g. kthread_create()) calls
to kproc_xxx as they actually make whole processes.
Thos makes way for us to add REAL kthread_create() and friends
that actually make theads. it turns out that most of these
calls actually end up being moved back to the thread version
when it's added. but we need to make this cosmetic change first.

I'd LOVE to do this rename in 7.0 so that we can eventually MFC the
new kthread_xxx() calls.

Overhaul driver/subsystem api's:
o make all crypto drivers have a device_t; pseudo drivers like the s/w
crypto driver synthesize one
o change the api between the crypto subsystem and drivers to use kobj;
cryptodev_if.m defines this api
o use the fact that all crypto drivers now have a device_t to add support
for specifying which of several potential devices to use when doing
crypto operations
o add new ioctls that allow user apps to select a specific crypto device
to use (previous ioctls maintained for compatibility)
o overhaul crypto subsystem code to eliminate lots of cruft and hide
implementation details from drivers
o bring in numerous fixes from Michale Richardson/hifn; mostly for
795x parts
o add an optional mechanism for mmap'ing the hifn 795x public key h/w
to user space for use by openssl (not enabled by default)
o update crypto test tools to use new ioctl's and add cmd line options
to specify a device to use for tests

These changes will also enable much future work on improving the core
crypto subsystem; including proper load balancing and interposing code
between the core and drivers to dispatch small operations to the s/w
driver as appropriate.

These changes were instigated by the work of Michael Richardson.

Reviewed by: pjd
Approved by: re

When DIAGNOSTIC is defined, verify if we don't free crypto requests from
the crypto queue or from the return queue.

Prefer hardware crypto over software crypto.

Before the change if a hardware crypto driver was loaded after
the software crypto driver, calling crypto_newsession() with
hard=0, will always choose software crypto.

Improve the code responsible for waking up the crypto_proc thread.
Checking if the queues are empty is not enough for the crypto_proc thread
(it is enough for the crypto_ret_thread), because drivers can be marked
as blocked. In a situation where we have operations related to different
crypto drivers in the queue, it is possible that one driver is marked as
blocked. In this case, the queue will not be empty and we won't wakeup
the crypto_proc thread to execute operations for the others drivers.

Simply setting a global variable to 1 when we goes to sleep and setting
it back to 0 when we wake up is sufficient. The variable is protected
with the queue lock.

Don't wakeup the crypto_ret_proc thread if it is running already.
Before the change if the thread was working on symmetric operation, we
would send unnecessary wakeup after adding asymmetric operation (when
asym queue was empty) and vice versa.

Don't set cc_kqblocked twice and don't increment cryptostats.cs_kblocks
twice if we call crypto_kinvoke() from crypto_proc thread.
This change also removes unprotected access to cc_kqblocked field
(CRYPTO_Q_LOCK() should be used for protection).

We must synchronize access to cc_qblocked, because there could be a race
where crypto_invoke() returns ERESTART and before we set cc_qblocked to 1,
crypto_unblock() is called and sets it to 0. This way we mark device as
blocked forever.

Fix it by not setting cc_qblocked in the fast path and by protecting
crypto_invoke() in the crypto_proc thread with CRYPTO_Q_LOCK().
This won't slow things down, because there is no contention - we have
only one crypto thread. Actually it can be slightly faster, because we
save two atomic ops per crypto request.
The fast code path remains lock-less.

Silent Coverity Prevent report by asserting that cap != NULL.

Coverity ID: 1414

- Make opencrypto more SMP friendly by dropping the queue lock around
crypto_invoke(). This allows to serve multiple crypto requests in
parallel and not bached requests are served lock-less.
Drivers should not depend on the queue lock beeing held around
crypto_invoke() and if they do, that's an error in the driver - it
should do its own synchronization.
- Don't forget to wakeup the crypto thread when new requests is
queued and only if both symmetric and asymmetric queues are empty.
- Symmetric requests use sessions and there is no way driver can
disappear when there is an active session, so we don't need to check
this, but assert this. This is also safe to not use the driver lock
in this case.
- Assymetric requests don't use sessions, so don't check the driver
in crypto_kinvoke().
- Protect assymetric operation with the driver lock, because if there
is no symmetric session, driver can disappear.
- Don't send assymetric request to the driver if it is marked as
blocked.
- Add an XXX comment, because I don't think migration to another driver
is safe when there are pending requests using freed session.
- Remove 'hint' argument from crypto_kinvoke(), as it serves no purpose.
- Don't hold the driver lock around kprocess method call, instead use
cc_koperations to track number of in-progress requests.
- Cleanup register/unregister code a bit.
- Other small simplifications and cleanups.

Reviewed by: sam

Be sure to wakeup the crypto thread when new request was queued.
This should fix a hang when starting cryptokeytest (and more).

MFC after: 1 month

crypto.ko depends on zlib.

Submitted by: Ben Kelly <bkelly at vadev.org>
Approved by: rwatson
Point hat to: me
MFC after: 1 day

/* -> /*- for license, minor formatting changes

add missing #include <sys/module.h>

kthread_exit() no longer requires Giant, so don't force callers to acquire
Giant just to call kthread_exit().

Requested by: many

consolidate callback optimization check in one location by adding a flag
for crypto operations that indicates the crypto code should do the check
in crypto_done

MFC after: 1 day

Add support to eliminate a context switch per crypto op when using the
software crypto device:

o record crypto device capabilities in each session id
o add a capability that indicates if the crypto driver operates synchronously
o tag the software crypto driver as operating synchronously

This commit also introduces crypto session id macros that cleanup their
construction and querying.

Use __FBSDID().

Flush my local cache of cryto subsystem fixes:

o add a ``done'' flag for crypto operations; this is set when the operation
completes and is intended for callers to check operations that may complete
``prematurely'' because of direct callbacks
o close a race for operations where the crypto driver returns ERESTART: we
need to hold the q lock to insure the blocked state for the driver and any
driver-private state is consistent; otherwise drivers may take an interrupt
and notify the crypto subsystem that it can unblock the driver but operations
will be left queued and never be processed
o close a race in /dev/crypto where operations can complete before the caller
can sleep waiting for the callback: use a per-session mutex and the new done
flag to handle this
o correct crypto_dispatch's handling of operations where the driver returns
ERESTART: the return value must be zero and not ERESTART, otherwise the
caller may free the crypto request despite it being queued for later handling
(this typically results in a later panic)
o change crypto mutex ``names'' so witness printouts and the like are more
meaningful

o add a CRYPTO_F_CBIMM flag to symmetric ops to indicate the callback
should be done in crypto_done rather than in the callback thread
o use this flag to mark operations from /dev/crypto since the callback
routine just does a wakeup; this eliminates the last unneeded ctx switch
o change CRYPTO_F_NODELAY to CRYPTO_F_BATCH with an inverted meaning
so "0" becomes the default/desired setting (needed for user-mode
compatibility with openbsd)
o change crypto_dispatch to honor CRYPTO_F_BATCH instead of always
dispatching immediately
o remove uses of CRYPTO_F_NODELAY
o define COP_F_BATCH for ops submitted through /dev/crypto and pass
this on to the op that is submitted

Similar changes and more eventually coming for asymmetric ops.

MFC if re gives approval.

Invoke the driver directly when a request is submitted via crypto_dispatch
or crypto_kdispatch unless the driver is currently blocked. This eliminates
the context switch to the dispatch thread for virtually all requests.

Note that this change means that for software crypto drivers the caller
will now block until the request is completed and the callback is dispatched
to the callback thread (h/w drivers will typically just dispatch the op to
the device and return quickly). If this is an issue we can either implement
a non-blocking interface in the s/w crypto driver or use either the
"no delay" flag in the crypto request or the "software driver" capability
flag to control what to do.

Sponsored by: Vernier Networks

correct flags passed when allocation crypto request structures so we
don't block and the returned data is zero'd

manage kernel threads properly; especially shutting them down on module unload

Reviewed by: jhb

MFS: crypto timing support; purge usercrypto sysctl (just don't config
cryptodev or kldunload cryptodev module); crypto statistcs; remove
unused alloctype field from crypto op to offset addition of the
performance time stamp

Supported by: Vernier Networks

change load order so module is present before crypto drivers

Submitted by: Doug Ambrisko" <ambrisko@verniernetworks.com>

Module-ize the 'core' crypto stuff. This may still need to be compiled
into the kernel by default (if required), but other modules can now
depend() on this.

Fix inter-module dependancy.

Earlier version OK'ed by: sam

o split crypto_proc into two threads: one for processing requests and one
for processing callbacks. This closes race conditions caused by locking
too many things with a single mutex.
o reclaim crypto requests under certain (impossible) failure conditions

In-kernel crypto framework derived from openbsd. This facility provides
a consistent interface to h/w and s/w crypto algorithms for use by the
kernel and (for h/w at least) by user-mode apps. Access for user-level
code is through a /dev/crypto device that'll eventually be used by openssl
to (potentially) accelerate many applications. Coming soon is an IPsec
that makes use of this service to accelerate ESP, AH, and IPCOMP protocols.

Included here is the "core" crypto support, /dev/crypto driver, various
crypto algorithms that are not already present in the KAME crypto area,
and support routines used by crypto device drivers.

Obtained from: openbsd