Remove crp_etype and return errors directly from crypto_invoke()

ok patrick@

Stop setting etype in the MD crypto code. So far we have set the etype
and returned the error, which made the MI crypto code set the etype for
a second time. We still have to set etype after calling the MD process
function, as the callers of crypto_invoke() still expect error handling
to be shown through the etype. But at least now all MD crypto code does
not have to worry about that anymore. Once the callers are changed to
not look at etype anymore, we can get rid of it completely.

ok tobhe@

Retire asynchronous crypto API as it is no longer required by any driver and
adds unnecessary complexity. Dedicated crypto offloading devices are not common
anymore. Modern CPU crypto acceleration works synchronously, eliminating the need
for callbacks.

Replace all occurrences of crypto_dispatch() with crypto_invoke(), which is
blocking and only returns after the operation has completed or an error occured.
Invoke callback functions directly from the consumer (e.g. IPsec, softraid)
instead of relying on the crypto driver to call crypto_done().

ok bluhm@ mvs@ patrick@

Remove more dead code related to crypto task queues.

Remove code to run crypto operations in a task queue. The code was
not reachable because all callers had set the CRYPTO_F_NOQUEUE flag.

ok patrick@ mvs@ bluhm@

The function crypto_dispatch() never returns an error. Make it
void and remove error handling in the callers.
OK patrick@ mvs@

The kernel crypto framework sometimes returned an error, sometimes
the callback was called, and sometimes both. So the caller of that
API could not release resources correctly.
A bunch of errors can or should not happen, replace them with an
assert. Remove redundant checks. crypto_invoke() should not return
the error, but pass it via callback.
Some old hardware drivers keep part of their inconsistency as I
cannot test them.
OK mpi@

Do not queue crypto operations for IPsec. The packet entries in
task queues were unlimited and could overflow during havy traffic.
Even if we still use hardware drivers that sleep, softnet task
instead of soft interrupt can handle this now. Without queues net
lock is inherited and kernel lock is only needed once per packet.
This results in less lock contention and faster IPsec.
Also protect tdb drop counters with net lock and avoid a leak in
crypto dispatch error handling.
intense testing Hrvoje Popovski; OK mpi@

Propagate errors from crypto_invoke() and count them in IPsec. They
should not happen, but always check error conditions. tq is never
NULL, remove the check. tdb->tdb_odrops++ is not MP safe, but will
be addressed separately in ipsec_output_cb().
OK mvs@

Remove unused variable cryptodesc_pool. Document global variables
in crypto.c and annotate locking protection. Assert kernel lock
where needed. Remove dead code from crypto_get_driverid(). Move
crypto_init() prototype into header file.
OK mpi@

Break crypto_unregister() sanity check into two expressions, making it
clearer what is being checked.

Original suggestion from kettenis@.

Don't access past end of cc_alg[] when trying to avoid
unregistering an invalid algorithm.

CID 1453298

ok kettenis@ (with suggested improvements to come)

Fix the IPL and flags of the MP-safe crypto taskq. Now a sane IPL
is passed to the mutex implementation, and the queue actually runs
without the kernel lock.

Tested by dhill@
OK mikeb@, dhill@, kettenis@

Reduce the per-packet allocation costs for crypto operations (cryptop)
by pre-allocating two cryptodesc objects and storing them in an array
instead of a linked list. If more than two cryptodesc objects are
required use mallocarray to fetch them. Adapt the drivers to the new
API.

This change results in one pool-get per ESP packet instead of three.
It also simplifies softraid crypto where more cryptodesc objects are
allocated than used.

From, with and ok markus@, ok bluhm@
"looks sane" mpi@

convert bcopy to memcpy. from david hill

all pools have their ipl set via pool_setipl, so fold it into pool_init.

the ioff argument to pool_init() is unused and has been for many
years, so this replaces it with an ipl argument. because the ipl
will be set on init we no longer need pool_setipl.

most of these changes have been done with coccinelle using the spatch
below. cocci sucks at formatting code though, so i fixed that by hand.

the manpage and subr_pool.c bits i did myself.

ok tedu@ jmatthew@

@ipl@
expression pp;
expression ipl;
expression s, a, o, f, m, p;
@@
-pool_init(pp, s, a, o, f, m, p);
-pool_setipl(pp, ipl);
+pool_init(pp, s, a, ipl, f, m, p);

Add a mechanism for dispatching mpsafe crypto operations. This adds a new
CRYPTOCAP_F_MPSAFE flag that crypto implementations can set to indicate that
their cc_process() implementation can safely run without holding the kernel
lock.

ok mikeb@

fairly simple sizes for free(); ok tedu

we want to defer work traditionally (in openbsd) handled in an
interrupt context to a taskq running in a thread. however, there
is a concern that if we do that then we allow accidental use of
sleeping APIs in this work, which will make it harder to move the
work back to interrupts in the future.

guenther and kettenis came up with the idea of marking a proc with
CANTSLEEP which the sleep paths can check and panic on.

this builds on that so you create taskqs that run with CANTSLEEP
set except when they need to sleep for more tasks to run.

the taskq_create api is changed to take a flags argument so users
can specify CANTSLEEP. MPSAFE is also passed via this flags field
now. this means archs that defined IPL_MPSAFE to 0 can now create
mpsafe taskqs too.

lots of discussion at s2k15
ok guenther@ miod@ mpi@ tedu@ pelikan@

remove the second void * argument on tasks.

when workqs were introduced, we provided a second argument so you
could pass a thing and some context to work on it in. there were
very few things that took advantage of the second argument, so when
i introduced pools i suggested removing it. since tasks were meant
to replace workqs, it was requested that we keep the second argument
to make porting from workqs to tasks easier.

now that workqs are gone, i had a look at the use of the second
argument again and found only one good use of it (vdsp(4) on sparc64
if you're interested) and a tiny handful of questionable uses. the
vast majority of tasks only used a single argument. i have since
modified all tasks that used two args to only use one, so now we
can remove the second argument.

so this is a mechanical change. all tasks only passed NULL as their
second argument, so we can just remove it.

ok krw@

pools lock themselves now, we just have to tell them what IPL they
will be used from.

this adds pool_setipl at IPL_VM to the crypto descriptor pools, and
removes all the splvm handling around the use of those pools.

tested by many via tech@
ok kettenis@ deraadt@

apply only the bit of r1.69 that should have been committed:

make the crypto taskq protect things at IPL_VM instead of IPL_HIGH.

everything else in crypto.c uses splvm/IPL_VM. it seems this IPL_HIGH
came about because the hand rolled task list and thread that crypto
used to use was converted to workqs, which unconditionally used
IPL_HIGH internally. when it was converted from workqs to tasks it
blindly ported the protection workqs gave.

tested by many via tech@ and snapshots
ok kettenis@

revert previous. it did more than the commit message said it did.

make the crypto taskq protect things at IPL_VM instead of IPL_HIGH.

everything else in crypto.c uses splvm/IPL_VM. it seems this IPL_HIGH
came about because the hand rolled task list and thread that crypto
used to use was converted to workqs, which unconditionally used
IPL_HIGH internally. when it was converted from workqs to tasks it
blindly ported the protection workqs gave.

tested by many via tech@ and snapshots
ok kettenis@

replace bzeros after allocations with M_ZERO and PR_ZERO as appropriate.

ok deraadt@

remove uneeded proc.h includes
ok mpi@ kspillner@

Bye bye /dev/crypto

The interface has been disabled by default for about 4 years and
currently there's not much value in having it around at all.

ok deraadt

use mallocarray()

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

cc_queued is not used for anything atm, remove it; ok jsing, markus

Respect CRYPTO_F_NOQUEUE flag when dispatching a crypto operation

ok jsing, markus

convert crypto work queue to the task_add(9) api; ok dlg

institute a hard cap on crypto devs instead of a useless wraparound check
ok beck

for key material that is being being discarded, convert bzero() to
explicit_bzero() where required
ok markus mikeb

Reintroduce most crypto/crypto.c r1.55:

Move pool initialization to init_crypto and zap the crypto_pool_initialized
variable. This way we don't have to check if the pool are initialized every
time we do a crypto_getreq().

However, also perform the crypto initialisation earlier in init_main so
that the crypto pools are initialised before they are used.

ok mikeb@ thib@ deraadt@

Backout r1.55 since this breaks anything which does crypto ops prior to
init_crypto() being called from late in init_main(). In particular, this
breaks softraid crypto volumes that are assembled at boot.

No cookies for thib/mikeb!

"Back it out, right now" deraadt@

Revert part of previous.

The splvm protection is needed after all, as we are walking the list
of registered crypto drivers and doing that unprotected is unwise.

Pointed out by kettenis@

Move pool initialization to init_crypto and zap the crypto_pool_initialized
variable. This way we don't have to check if the pool are initialized every
time we do a crypto_getreq().

Move splvm lower as it isnt need all through crypto_newsession().

tiny KNF nit.

From mikeb

OK deraadt@

Remove the CRYPTO_ALGORITHM_ALL define, fixup accordingly
and make the loop invartiants <= CRYPTO_ALGORITHM_MAX
Do this also for the CRK_ALGORITHM_MAX this also fixes
the a bug that caused us to skip CRK_DH_COMPUTE_KEY.

ok deraadt@

crypto hardware (eg, hifn) establishes its interrupt handler at
IPL_NET. when the hardware finishes some work for the crypto subsystem
and therefore something in the kernel that wanted crypto done, it
calls crypto_done from that interrupt handler.

one of the things that uses crypto is ipsec. when crypto is done
for ipsec it then pushes the packet along the network stack. the
problem is that all the structures inside the network stack are
only protected at splsoftnet. we could be in the middle of modifications
to the pf state table or the pfsync queues when we get a hifn
interrupt and then go stomp on the same structures.

the solution is to defer the completions so they can do the right
spl protections.

this basically reverts r1.46 of src/sys/crypto/crypto.c.

found by naddy@

reintroduce mutexes to workqs for locking.

tested by many on many archs including several alpha test.

ok tedu@ go for it deraadt@

finish conversion to workq. remove list remnants, and put spl in the right
places. handle the no workq case here. ok deraadt

convert crypto thread to workq. add WQ_DIRECTOK flag to workq.
combined, this lets us use crypto before the thread is running
and therefore cryptoraid can attach nice and early.
ok/testing deraadt mbalmer marco

do not call crypto_done() on errors, since the drivers already do this.
otherwise we call the callback twice; fixes panics on crypto errors as
seen on reboot; ok hshoexer

remove some silly casts. put spl calls after all declarations.
put one splx in a better spot. make a variable size MALLOC use malloc.
remove null test after malloc(M_WAITOK).
add PR_NOWAIT flag to pool_get instead of 0. change callbacks to correct type.
ok brad deraadt markus mickey

splimp -> splvm

ok miod@

Don't use crypto thread for callbacks.
This primarily improves IPsec performance when using crypto accelerators.
With help from markus@, tested by wvdputte@.

ok deraadt@, markus@

In crypto_thread(), always save return value from splimp(). We were only
storing it once on kernel startup. Scary. "holy crap" --deraadt. art@ ok

Unclear if this was actually a problem in practice, but this doesn't hurt.

Fastpath for userland crypto requests. This change makes userland
crypto requests attempt to call the crypto driver directly to process
crypto layer requests, as opposed to queueing them in the kernel
crypto thread. If we can't use the crypto devices (i.e. they're busy)
we fall back to queueing the request up in the crypto thread as
before. This does allow for faster performance in some cases (smaller
requests, how small seems to be dependent on the card/cpu combination)
where context switching is a major issue in performance.
ok deraadt@ jason@

Copy the ENTIRE table into the supported algorithms (how the hell did this
work before?!)

From Angelos:
- simplistic load balancing across multiple cards
- simplified registration process
- a few style nits.

I don't know why this breaks things for me when sshd starts on sparc64.
But after wasting the whole day trying to just locate the problem I don't care.
Back out since this wasn't tested and showed to anyone else.

Double-pass over drivers, first hardware only, then software (if we
are interested in software).

Fix a typo, cleanup on session migration code in crypto_invoke(), and
add a convention that if the driver returns ERESTART as an error
message of its process method, the crypto framework will unregister
the driver and migrate all its sessions. After discussion with Sam
Leffler and Jason Wright.

kernel changes to make asymmetric crypto work in userland
- modify getfeat to return something more useful to us on devices
(like lofn and everything else until jason fixes it) that can't
do rsa stuff, etc and can only do mod_exp..
- error handling fixes so we correctly fail to software when we can't
deal with a particular key size
- add sysctl kern.userasymcrypto to turn on/off userland asymmetric crypto
via /dev/crypto - 1 == on, 0 == off, default is off

fix ivory tower greek fix. ok angelos@

Don't use an int for the flags, when the structure uses
u_int8_t. Also, make sure the logic is correct (bad theo!)

initial hack at a CIOCSYMFEAT ioctl

driver queueing & callback code for keying operations

crypto_check_alg() is not needed

It looks like there has been one crack smoking and a few cut and pastes.
PR_FREEHEADER should not be set in pool_init by the caller. It shouldn't
be set in pool_init at all. Besides, it's going away soon anyway.

Pool deals fairly well with physical memory shortage, but it doesn't deal
well (not at all) with shortages of the vm_map where the pages are mapped
(usually kmem_map).

Try to deal with it:
- group all information the backend allocator for a pool in a separate
struct. The pool will only have a pointer to that struct.
- change the pool_init API to reflect that.
- link all pools allocating from the same allocator on a linked list.
- Since an allocator is responsible to wait for physical memory it will
only fail (waitok) when it runs out of its backing vm_map, carefully
drain pools using the same allocator so that va space is freed.
(see comments in code for caveats and details).
- change pool_reclaim to return if it actually succeeded to free some
memory, use that information to make draining easier and more efficient.
- get rid of PR_URGENT, noone uses it.

branches: 1.30.2;
and for the case where it allocates a bunch at a time, also make sure the
software flag gets set.

incorrect check

be way more sure that software cannot be used

indent

branches: 1.26.2;
put in tags for ARC4 to please ben, who now has no excuses

KNF

Remove space.

Add crypto_check_alg(), from jgarfiel@seas.upenn.edu

Update copyright; you can use this with or without fee (unless your
name is Theo Deraadt)

New prototype for crypto_register(), to take into account maximum key
length (for PK operations) and various flags.

Structures for public key operations (DH, RSA, DSA). A lot of this
work was done by jgarfiel@seas.upenn.edu

merge crypto/crypto{dev,}.h to crypto/cryptodev.h, to avoid name conflicts inside OpenSSL codebase

KNF

Use pool(9) for some of the structures, and splimp/splx to protect
from ourselves. Should solve PR 1850.

initial cut at /dev/crypto support. takes original mbuf "try, and discard
if we fail" semantics and extends to two varients of data movement: mbuf,
or an iovec style block.

Use the M_CRYPTO_DATA and M_CRYPTO_OPS malloc types.

make the 31-bit code work on 32-bit machines.

branches: 1.14.2;
avoid excessive wakeup(). we think this is safe...

MALLOC/FREE -> malloc/free ok art@ angelos@

Fix tail queue leakage (zzlevo@dd.chalmers.se)

crypto_done(), all it does for now is invoke the callback.

Add Rijndael (128-bit blocksize) in the software crypto driver.

Hacking at OpenBSD Crypto 2000 :-)

Move prototype to include file.

Crypto kernel thread interface; requests are enqueued and processed by
a kernel thread. This allows a much cleaner interface with respect to
spl levels.

OpenBSD tags

crypto_dispatch() only returns an error if the argument it was
provided was NULL or no callback was specified.

avoid using void * when we are talking about pointers

Change the type of freesession to take u_int64_t as argument.

Add a few newlines for readability.

branches: 1.2.2;
split crypto driver front-end from software crypto engine

Cryptographic services framework, and software "device driver". The
idea is to support various cryptographic hardware accelerators (which
may be (detachable) cards, secondary/tertiary/etc processors,
software crypto, etc). Supports session migration between crypto
devices. What it doesn't (yet) support:
- multiple instances of the same algorithm used in the same session
- use of multiple crypto drivers in the same session
- asymmetric crypto

No support for a userland device yet.

IPsec code path modified to allow for asynchronous cryptography
(callbacks used in both input and output processing). Some unrelated
code simplification done in the process (especially for AH).

Development of this code kindly supported by Network Security
Technologies (NSTI). The code was writen mostly in Greece, and is
being committed from Montreal.

Stop setting etype in the MD crypto code. So far we have set the etype
and returned the error, which made the MI crypto code set the etype for
a second time. We still have to set etype after calling the MD process
function, as the callers of crypto_invoke() still expect error handling
to be shown through the etype. But at least now all MD crypto code does
not have to worry about that anymore. Once the callers are changed to
not look at etype anymore, we can get rid of it completely.

ok tobhe@

Retire asynchronous crypto API as it is no longer required by any driver and
adds unnecessary complexity. Dedicated crypto offloading devices are not common
anymore. Modern CPU crypto acceleration works synchronously, eliminating the need
for callbacks.

Replace all occurrences of crypto_dispatch() with crypto_invoke(), which is
blocking and only returns after the operation has completed or an error occured.
Invoke callback functions directly from the consumer (e.g. IPsec, softraid)
instead of relying on the crypto driver to call crypto_done().

ok bluhm@ mvs@ patrick@

Remove more dead code related to crypto task queues.

Remove code to run crypto operations in a task queue. The code was
not reachable because all callers had set the CRYPTO_F_NOQUEUE flag.

ok patrick@ mvs@ bluhm@

The function crypto_dispatch() never returns an error. Make it
void and remove error handling in the callers.
OK patrick@ mvs@

The kernel crypto framework sometimes returned an error, sometimes
the callback was called, and sometimes both. So the caller of that
API could not release resources correctly.
A bunch of errors can or should not happen, replace them with an
assert. Remove redundant checks. crypto_invoke() should not return
the error, but pass it via callback.
Some old hardware drivers keep part of their inconsistency as I
cannot test them.
OK mpi@

Do not queue crypto operations for IPsec. The packet entries in
task queues were unlimited and could overflow during havy traffic.
Even if we still use hardware drivers that sleep, softnet task
instead of soft interrupt can handle this now. Without queues net
lock is inherited and kernel lock is only needed once per packet.
This results in less lock contention and faster IPsec.
Also protect tdb drop counters with net lock and avoid a leak in
crypto dispatch error handling.
intense testing Hrvoje Popovski; OK mpi@

Propagate errors from crypto_invoke() and count them in IPsec. They
should not happen, but always check error conditions. tq is never
NULL, remove the check. tdb->tdb_odrops++ is not MP safe, but will
be addressed separately in ipsec_output_cb().
OK mvs@

Remove unused variable cryptodesc_pool. Document global variables
in crypto.c and annotate locking protection. Assert kernel lock
where needed. Remove dead code from crypto_get_driverid(). Move
crypto_init() prototype into header file.
OK mpi@

Break crypto_unregister() sanity check into two expressions, making it
clearer what is being checked.

Original suggestion from kettenis@.

Don't access past end of cc_alg[] when trying to avoid
unregistering an invalid algorithm.

CID 1453298

ok kettenis@ (with suggested improvements to come)

Fix the IPL and flags of the MP-safe crypto taskq. Now a sane IPL
is passed to the mutex implementation, and the queue actually runs
without the kernel lock.

Tested by dhill@
OK mikeb@, dhill@, kettenis@

Reduce the per-packet allocation costs for crypto operations (cryptop)
by pre-allocating two cryptodesc objects and storing them in an array
instead of a linked list. If more than two cryptodesc objects are
required use mallocarray to fetch them. Adapt the drivers to the new
API.

This change results in one pool-get per ESP packet instead of three.
It also simplifies softraid crypto where more cryptodesc objects are
allocated than used.

From, with and ok markus@, ok bluhm@
"looks sane" mpi@

convert bcopy to memcpy. from david hill

all pools have their ipl set via pool_setipl, so fold it into pool_init.

the ioff argument to pool_init() is unused and has been for many
years, so this replaces it with an ipl argument. because the ipl
will be set on init we no longer need pool_setipl.

most of these changes have been done with coccinelle using the spatch
below. cocci sucks at formatting code though, so i fixed that by hand.

the manpage and subr_pool.c bits i did myself.

ok tedu@ jmatthew@

@ipl@
expression pp;
expression ipl;
expression s, a, o, f, m, p;
@@
-pool_init(pp, s, a, o, f, m, p);
-pool_setipl(pp, ipl);
+pool_init(pp, s, a, ipl, f, m, p);

Add a mechanism for dispatching mpsafe crypto operations. This adds a new
CRYPTOCAP_F_MPSAFE flag that crypto implementations can set to indicate that
their cc_process() implementation can safely run without holding the kernel
lock.

ok mikeb@

fairly simple sizes for free(); ok tedu

we want to defer work traditionally (in openbsd) handled in an
interrupt context to a taskq running in a thread. however, there
is a concern that if we do that then we allow accidental use of
sleeping APIs in this work, which will make it harder to move the
work back to interrupts in the future.

guenther and kettenis came up with the idea of marking a proc with
CANTSLEEP which the sleep paths can check and panic on.

this builds on that so you create taskqs that run with CANTSLEEP
set except when they need to sleep for more tasks to run.

the taskq_create api is changed to take a flags argument so users
can specify CANTSLEEP. MPSAFE is also passed via this flags field
now. this means archs that defined IPL_MPSAFE to 0 can now create
mpsafe taskqs too.

lots of discussion at s2k15
ok guenther@ miod@ mpi@ tedu@ pelikan@

remove the second void * argument on tasks.

when workqs were introduced, we provided a second argument so you
could pass a thing and some context to work on it in. there were
very few things that took advantage of the second argument, so when
i introduced pools i suggested removing it. since tasks were meant
to replace workqs, it was requested that we keep the second argument
to make porting from workqs to tasks easier.

now that workqs are gone, i had a look at the use of the second
argument again and found only one good use of it (vdsp(4) on sparc64
if you're interested) and a tiny handful of questionable uses. the
vast majority of tasks only used a single argument. i have since
modified all tasks that used two args to only use one, so now we
can remove the second argument.

so this is a mechanical change. all tasks only passed NULL as their
second argument, so we can just remove it.

ok krw@

pools lock themselves now, we just have to tell them what IPL they
will be used from.

this adds pool_setipl at IPL_VM to the crypto descriptor pools, and
removes all the splvm handling around the use of those pools.

tested by many via tech@
ok kettenis@ deraadt@

apply only the bit of r1.69 that should have been committed:

make the crypto taskq protect things at IPL_VM instead of IPL_HIGH.

everything else in crypto.c uses splvm/IPL_VM. it seems this IPL_HIGH
came about because the hand rolled task list and thread that crypto
used to use was converted to workqs, which unconditionally used
IPL_HIGH internally. when it was converted from workqs to tasks it
blindly ported the protection workqs gave.

tested by many via tech@ and snapshots
ok kettenis@

revert previous. it did more than the commit message said it did.

make the crypto taskq protect things at IPL_VM instead of IPL_HIGH.

everything else in crypto.c uses splvm/IPL_VM. it seems this IPL_HIGH
came about because the hand rolled task list and thread that crypto
used to use was converted to workqs, which unconditionally used
IPL_HIGH internally. when it was converted from workqs to tasks it
blindly ported the protection workqs gave.

tested by many via tech@ and snapshots
ok kettenis@

replace bzeros after allocations with M_ZERO and PR_ZERO as appropriate.

ok deraadt@

remove uneeded proc.h includes
ok mpi@ kspillner@

Bye bye /dev/crypto

The interface has been disabled by default for about 4 years and
currently there's not much value in having it around at all.

ok deraadt

use mallocarray()

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

cc_queued is not used for anything atm, remove it; ok jsing, markus

Respect CRYPTO_F_NOQUEUE flag when dispatching a crypto operation

ok jsing, markus

convert crypto work queue to the task_add(9) api; ok dlg

institute a hard cap on crypto devs instead of a useless wraparound check
ok beck

for key material that is being being discarded, convert bzero() to
explicit_bzero() where required
ok markus mikeb

Reintroduce most crypto/crypto.c r1.55:

Move pool initialization to init_crypto and zap the crypto_pool_initialized
variable. This way we don't have to check if the pool are initialized every
time we do a crypto_getreq().

However, also perform the crypto initialisation earlier in init_main so
that the crypto pools are initialised before they are used.

ok mikeb@ thib@ deraadt@

Backout r1.55 since this breaks anything which does crypto ops prior to
init_crypto() being called from late in init_main(). In particular, this
breaks softraid crypto volumes that are assembled at boot.

No cookies for thib/mikeb!

"Back it out, right now" deraadt@

Revert part of previous.

The splvm protection is needed after all, as we are walking the list
of registered crypto drivers and doing that unprotected is unwise.

Pointed out by kettenis@

Move pool initialization to init_crypto and zap the crypto_pool_initialized
variable. This way we don't have to check if the pool are initialized every
time we do a crypto_getreq().

Move splvm lower as it isnt need all through crypto_newsession().

tiny KNF nit.

From mikeb

OK deraadt@

Remove the CRYPTO_ALGORITHM_ALL define, fixup accordingly
and make the loop invartiants <= CRYPTO_ALGORITHM_MAX
Do this also for the CRK_ALGORITHM_MAX this also fixes
the a bug that caused us to skip CRK_DH_COMPUTE_KEY.

ok deraadt@

crypto hardware (eg, hifn) establishes its interrupt handler at
IPL_NET. when the hardware finishes some work for the crypto subsystem
and therefore something in the kernel that wanted crypto done, it
calls crypto_done from that interrupt handler.

one of the things that uses crypto is ipsec. when crypto is done
for ipsec it then pushes the packet along the network stack. the
problem is that all the structures inside the network stack are
only protected at splsoftnet. we could be in the middle of modifications
to the pf state table or the pfsync queues when we get a hifn
interrupt and then go stomp on the same structures.

the solution is to defer the completions so they can do the right
spl protections.

this basically reverts r1.46 of src/sys/crypto/crypto.c.

found by naddy@

reintroduce mutexes to workqs for locking.

tested by many on many archs including several alpha test.

ok tedu@ go for it deraadt@

finish conversion to workq. remove list remnants, and put spl in the right
places. handle the no workq case here. ok deraadt

convert crypto thread to workq. add WQ_DIRECTOK flag to workq.
combined, this lets us use crypto before the thread is running
and therefore cryptoraid can attach nice and early.
ok/testing deraadt mbalmer marco

do not call crypto_done() on errors, since the drivers already do this.
otherwise we call the callback twice; fixes panics on crypto errors as
seen on reboot; ok hshoexer

remove some silly casts. put spl calls after all declarations.
put one splx in a better spot. make a variable size MALLOC use malloc.
remove null test after malloc(M_WAITOK).
add PR_NOWAIT flag to pool_get instead of 0. change callbacks to correct type.
ok brad deraadt markus mickey

splimp -> splvm

ok miod@

Don't use crypto thread for callbacks.
This primarily improves IPsec performance when using crypto accelerators.
With help from markus@, tested by wvdputte@.

ok deraadt@, markus@

In crypto_thread(), always save return value from splimp(). We were only
storing it once on kernel startup. Scary. "holy crap" --deraadt. art@ ok

Unclear if this was actually a problem in practice, but this doesn't hurt.

Fastpath for userland crypto requests. This change makes userland
crypto requests attempt to call the crypto driver directly to process
crypto layer requests, as opposed to queueing them in the kernel
crypto thread. If we can't use the crypto devices (i.e. they're busy)
we fall back to queueing the request up in the crypto thread as
before. This does allow for faster performance in some cases (smaller
requests, how small seems to be dependent on the card/cpu combination)
where context switching is a major issue in performance.
ok deraadt@ jason@

Copy the ENTIRE table into the supported algorithms (how the hell did this
work before?!)

From Angelos:
- simplistic load balancing across multiple cards
- simplified registration process
- a few style nits.

I don't know why this breaks things for me when sshd starts on sparc64.
But after wasting the whole day trying to just locate the problem I don't care.
Back out since this wasn't tested and showed to anyone else.

Double-pass over drivers, first hardware only, then software (if we
are interested in software).

Fix a typo, cleanup on session migration code in crypto_invoke(), and
add a convention that if the driver returns ERESTART as an error
message of its process method, the crypto framework will unregister
the driver and migrate all its sessions. After discussion with Sam
Leffler and Jason Wright.

kernel changes to make asymmetric crypto work in userland
- modify getfeat to return something more useful to us on devices
(like lofn and everything else until jason fixes it) that can't
do rsa stuff, etc and can only do mod_exp..
- error handling fixes so we correctly fail to software when we can't
deal with a particular key size
- add sysctl kern.userasymcrypto to turn on/off userland asymmetric crypto
via /dev/crypto - 1 == on, 0 == off, default is off

fix ivory tower greek fix. ok angelos@

Don't use an int for the flags, when the structure uses
u_int8_t. Also, make sure the logic is correct (bad theo!)

initial hack at a CIOCSYMFEAT ioctl

driver queueing & callback code for keying operations

crypto_check_alg() is not needed

It looks like there has been one crack smoking and a few cut and pastes.
PR_FREEHEADER should not be set in pool_init by the caller. It shouldn't
be set in pool_init at all. Besides, it's going away soon anyway.

Pool deals fairly well with physical memory shortage, but it doesn't deal
well (not at all) with shortages of the vm_map where the pages are mapped
(usually kmem_map).

Try to deal with it:
- group all information the backend allocator for a pool in a separate
struct. The pool will only have a pointer to that struct.
- change the pool_init API to reflect that.
- link all pools allocating from the same allocator on a linked list.
- Since an allocator is responsible to wait for physical memory it will
only fail (waitok) when it runs out of its backing vm_map, carefully
drain pools using the same allocator so that va space is freed.
(see comments in code for caveats and details).
- change pool_reclaim to return if it actually succeeded to free some
memory, use that information to make draining easier and more efficient.
- get rid of PR_URGENT, noone uses it.

branches: 1.30.2;
and for the case where it allocates a bunch at a time, also make sure the
software flag gets set.

incorrect check

be way more sure that software cannot be used

indent

branches: 1.26.2;
put in tags for ARC4 to please ben, who now has no excuses

KNF

Remove space.

Add crypto_check_alg(), from jgarfiel@seas.upenn.edu

Update copyright; you can use this with or without fee (unless your
name is Theo Deraadt)

New prototype for crypto_register(), to take into account maximum key
length (for PK operations) and various flags.

Structures for public key operations (DH, RSA, DSA). A lot of this
work was done by jgarfiel@seas.upenn.edu

merge crypto/crypto{dev,}.h to crypto/cryptodev.h, to avoid name conflicts inside OpenSSL codebase

KNF

Use pool(9) for some of the structures, and splimp/splx to protect
from ourselves. Should solve PR 1850.

initial cut at /dev/crypto support. takes original mbuf "try, and discard
if we fail" semantics and extends to two varients of data movement: mbuf,
or an iovec style block.

Use the M_CRYPTO_DATA and M_CRYPTO_OPS malloc types.

make the 31-bit code work on 32-bit machines.

branches: 1.14.2;
avoid excessive wakeup(). we think this is safe...

MALLOC/FREE -> malloc/free ok art@ angelos@

Fix tail queue leakage (zzlevo@dd.chalmers.se)

crypto_done(), all it does for now is invoke the callback.

Add Rijndael (128-bit blocksize) in the software crypto driver.

Hacking at OpenBSD Crypto 2000 :-)

Move prototype to include file.

Crypto kernel thread interface; requests are enqueued and processed by
a kernel thread. This allows a much cleaner interface with respect to
spl levels.

OpenBSD tags

crypto_dispatch() only returns an error if the argument it was
provided was NULL or no callback was specified.

avoid using void * when we are talking about pointers

Change the type of freesession to take u_int64_t as argument.

Add a few newlines for readability.

branches: 1.2.2;
split crypto driver front-end from software crypto engine

Cryptographic services framework, and software "device driver". The
idea is to support various cryptographic hardware accelerators (which
may be (detachable) cards, secondary/tertiary/etc processors,
software crypto, etc). Supports session migration between crypto
devices. What it doesn't (yet) support:
- multiple instances of the same algorithm used in the same session
- use of multiple crypto drivers in the same session
- asymmetric crypto

No support for a userland device yet.

IPsec code path modified to allow for asynchronous cryptography
(callbacks used in both input and output processing). Some unrelated
code simplification done in the process (especially for AH).

Development of this code kindly supported by Network Security
Technologies (NSTI). The code was writen mostly in Greece, and is
being committed from Montreal.

Remove more dead code related to crypto task queues.

Remove code to run crypto operations in a task queue. The code was
not reachable because all callers had set the CRYPTO_F_NOQUEUE flag.

ok patrick@ mvs@ bluhm@

The function crypto_dispatch() never returns an error. Make it
void and remove error handling in the callers.
OK patrick@ mvs@

The kernel crypto framework sometimes returned an error, sometimes
the callback was called, and sometimes both. So the caller of that
API could not release resources correctly.
A bunch of errors can or should not happen, replace them with an
assert. Remove redundant checks. crypto_invoke() should not return
the error, but pass it via callback.
Some old hardware drivers keep part of their inconsistency as I
cannot test them.
OK mpi@

Do not queue crypto operations for IPsec. The packet entries in
task queues were unlimited and could overflow during havy traffic.
Even if we still use hardware drivers that sleep, softnet task
instead of soft interrupt can handle this now. Without queues net
lock is inherited and kernel lock is only needed once per packet.
This results in less lock contention and faster IPsec.
Also protect tdb drop counters with net lock and avoid a leak in
crypto dispatch error handling.
intense testing Hrvoje Popovski; OK mpi@

Propagate errors from crypto_invoke() and count them in IPsec. They
should not happen, but always check error conditions. tq is never
NULL, remove the check. tdb->tdb_odrops++ is not MP safe, but will
be addressed separately in ipsec_output_cb().
OK mvs@

Remove unused variable cryptodesc_pool. Document global variables
in crypto.c and annotate locking protection. Assert kernel lock
where needed. Remove dead code from crypto_get_driverid(). Move
crypto_init() prototype into header file.
OK mpi@

Break crypto_unregister() sanity check into two expressions, making it
clearer what is being checked.

Original suggestion from kettenis@.

Don't access past end of cc_alg[] when trying to avoid
unregistering an invalid algorithm.

CID 1453298

ok kettenis@ (with suggested improvements to come)

Fix the IPL and flags of the MP-safe crypto taskq. Now a sane IPL
is passed to the mutex implementation, and the queue actually runs
without the kernel lock.

Tested by dhill@
OK mikeb@, dhill@, kettenis@

Reduce the per-packet allocation costs for crypto operations (cryptop)
by pre-allocating two cryptodesc objects and storing them in an array
instead of a linked list. If more than two cryptodesc objects are
required use mallocarray to fetch them. Adapt the drivers to the new
API.

This change results in one pool-get per ESP packet instead of three.
It also simplifies softraid crypto where more cryptodesc objects are
allocated than used.

From, with and ok markus@, ok bluhm@
"looks sane" mpi@

convert bcopy to memcpy. from david hill

all pools have their ipl set via pool_setipl, so fold it into pool_init.

the ioff argument to pool_init() is unused and has been for many
years, so this replaces it with an ipl argument. because the ipl
will be set on init we no longer need pool_setipl.

most of these changes have been done with coccinelle using the spatch
below. cocci sucks at formatting code though, so i fixed that by hand.

the manpage and subr_pool.c bits i did myself.

ok tedu@ jmatthew@

@ipl@
expression pp;
expression ipl;
expression s, a, o, f, m, p;
@@
-pool_init(pp, s, a, o, f, m, p);
-pool_setipl(pp, ipl);
+pool_init(pp, s, a, ipl, f, m, p);

Add a mechanism for dispatching mpsafe crypto operations. This adds a new
CRYPTOCAP_F_MPSAFE flag that crypto implementations can set to indicate that
their cc_process() implementation can safely run without holding the kernel
lock.

ok mikeb@

fairly simple sizes for free(); ok tedu

we want to defer work traditionally (in openbsd) handled in an
interrupt context to a taskq running in a thread. however, there
is a concern that if we do that then we allow accidental use of
sleeping APIs in this work, which will make it harder to move the
work back to interrupts in the future.

guenther and kettenis came up with the idea of marking a proc with
CANTSLEEP which the sleep paths can check and panic on.

this builds on that so you create taskqs that run with CANTSLEEP
set except when they need to sleep for more tasks to run.

the taskq_create api is changed to take a flags argument so users
can specify CANTSLEEP. MPSAFE is also passed via this flags field
now. this means archs that defined IPL_MPSAFE to 0 can now create
mpsafe taskqs too.

lots of discussion at s2k15
ok guenther@ miod@ mpi@ tedu@ pelikan@

remove the second void * argument on tasks.

when workqs were introduced, we provided a second argument so you
could pass a thing and some context to work on it in. there were
very few things that took advantage of the second argument, so when
i introduced pools i suggested removing it. since tasks were meant
to replace workqs, it was requested that we keep the second argument
to make porting from workqs to tasks easier.

now that workqs are gone, i had a look at the use of the second
argument again and found only one good use of it (vdsp(4) on sparc64
if you're interested) and a tiny handful of questionable uses. the
vast majority of tasks only used a single argument. i have since
modified all tasks that used two args to only use one, so now we
can remove the second argument.

so this is a mechanical change. all tasks only passed NULL as their
second argument, so we can just remove it.

ok krw@

pools lock themselves now, we just have to tell them what IPL they
will be used from.

this adds pool_setipl at IPL_VM to the crypto descriptor pools, and
removes all the splvm handling around the use of those pools.

tested by many via tech@
ok kettenis@ deraadt@

apply only the bit of r1.69 that should have been committed:

make the crypto taskq protect things at IPL_VM instead of IPL_HIGH.

everything else in crypto.c uses splvm/IPL_VM. it seems this IPL_HIGH
came about because the hand rolled task list and thread that crypto
used to use was converted to workqs, which unconditionally used
IPL_HIGH internally. when it was converted from workqs to tasks it
blindly ported the protection workqs gave.

tested by many via tech@ and snapshots
ok kettenis@

revert previous. it did more than the commit message said it did.

make the crypto taskq protect things at IPL_VM instead of IPL_HIGH.

everything else in crypto.c uses splvm/IPL_VM. it seems this IPL_HIGH
came about because the hand rolled task list and thread that crypto
used to use was converted to workqs, which unconditionally used
IPL_HIGH internally. when it was converted from workqs to tasks it
blindly ported the protection workqs gave.

tested by many via tech@ and snapshots
ok kettenis@

replace bzeros after allocations with M_ZERO and PR_ZERO as appropriate.

ok deraadt@

remove uneeded proc.h includes
ok mpi@ kspillner@

Bye bye /dev/crypto

The interface has been disabled by default for about 4 years and
currently there's not much value in having it around at all.

ok deraadt

use mallocarray()

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

cc_queued is not used for anything atm, remove it; ok jsing, markus

Respect CRYPTO_F_NOQUEUE flag when dispatching a crypto operation

ok jsing, markus

convert crypto work queue to the task_add(9) api; ok dlg

institute a hard cap on crypto devs instead of a useless wraparound check
ok beck

for key material that is being being discarded, convert bzero() to
explicit_bzero() where required
ok markus mikeb

Reintroduce most crypto/crypto.c r1.55:

Move pool initialization to init_crypto and zap the crypto_pool_initialized
variable. This way we don't have to check if the pool are initialized every
time we do a crypto_getreq().

However, also perform the crypto initialisation earlier in init_main so
that the crypto pools are initialised before they are used.

ok mikeb@ thib@ deraadt@

Backout r1.55 since this breaks anything which does crypto ops prior to
init_crypto() being called from late in init_main(). In particular, this
breaks softraid crypto volumes that are assembled at boot.

No cookies for thib/mikeb!

"Back it out, right now" deraadt@

Revert part of previous.

The splvm protection is needed after all, as we are walking the list
of registered crypto drivers and doing that unprotected is unwise.

Pointed out by kettenis@

Move pool initialization to init_crypto and zap the crypto_pool_initialized
variable. This way we don't have to check if the pool are initialized every
time we do a crypto_getreq().

Move splvm lower as it isnt need all through crypto_newsession().

tiny KNF nit.

From mikeb

OK deraadt@

Remove the CRYPTO_ALGORITHM_ALL define, fixup accordingly
and make the loop invartiants <= CRYPTO_ALGORITHM_MAX
Do this also for the CRK_ALGORITHM_MAX this also fixes
the a bug that caused us to skip CRK_DH_COMPUTE_KEY.

ok deraadt@

crypto hardware (eg, hifn) establishes its interrupt handler at
IPL_NET. when the hardware finishes some work for the crypto subsystem
and therefore something in the kernel that wanted crypto done, it
calls crypto_done from that interrupt handler.

one of the things that uses crypto is ipsec. when crypto is done
for ipsec it then pushes the packet along the network stack. the
problem is that all the structures inside the network stack are
only protected at splsoftnet. we could be in the middle of modifications
to the pf state table or the pfsync queues when we get a hifn
interrupt and then go stomp on the same structures.

the solution is to defer the completions so they can do the right
spl protections.

this basically reverts r1.46 of src/sys/crypto/crypto.c.

found by naddy@

reintroduce mutexes to workqs for locking.

tested by many on many archs including several alpha test.

ok tedu@ go for it deraadt@

finish conversion to workq. remove list remnants, and put spl in the right
places. handle the no workq case here. ok deraadt

convert crypto thread to workq. add WQ_DIRECTOK flag to workq.
combined, this lets us use crypto before the thread is running
and therefore cryptoraid can attach nice and early.
ok/testing deraadt mbalmer marco

do not call crypto_done() on errors, since the drivers already do this.
otherwise we call the callback twice; fixes panics on crypto errors as
seen on reboot; ok hshoexer

remove some silly casts. put spl calls after all declarations.
put one splx in a better spot. make a variable size MALLOC use malloc.
remove null test after malloc(M_WAITOK).
add PR_NOWAIT flag to pool_get instead of 0. change callbacks to correct type.
ok brad deraadt markus mickey

splimp -> splvm

ok miod@

Don't use crypto thread for callbacks.
This primarily improves IPsec performance when using crypto accelerators.
With help from markus@, tested by wvdputte@.

ok deraadt@, markus@

In crypto_thread(), always save return value from splimp(). We were only
storing it once on kernel startup. Scary. "holy crap" --deraadt. art@ ok

Unclear if this was actually a problem in practice, but this doesn't hurt.

Fastpath for userland crypto requests. This change makes userland
crypto requests attempt to call the crypto driver directly to process
crypto layer requests, as opposed to queueing them in the kernel
crypto thread. If we can't use the crypto devices (i.e. they're busy)
we fall back to queueing the request up in the crypto thread as
before. This does allow for faster performance in some cases (smaller
requests, how small seems to be dependent on the card/cpu combination)
where context switching is a major issue in performance.
ok deraadt@ jason@

Copy the ENTIRE table into the supported algorithms (how the hell did this
work before?!)

From Angelos:
- simplistic load balancing across multiple cards
- simplified registration process
- a few style nits.

I don't know why this breaks things for me when sshd starts on sparc64.
But after wasting the whole day trying to just locate the problem I don't care.
Back out since this wasn't tested and showed to anyone else.

Double-pass over drivers, first hardware only, then software (if we
are interested in software).

Fix a typo, cleanup on session migration code in crypto_invoke(), and
add a convention that if the driver returns ERESTART as an error
message of its process method, the crypto framework will unregister
the driver and migrate all its sessions. After discussion with Sam
Leffler and Jason Wright.

kernel changes to make asymmetric crypto work in userland
- modify getfeat to return something more useful to us on devices
(like lofn and everything else until jason fixes it) that can't
do rsa stuff, etc and can only do mod_exp..
- error handling fixes so we correctly fail to software when we can't
deal with a particular key size
- add sysctl kern.userasymcrypto to turn on/off userland asymmetric crypto
via /dev/crypto - 1 == on, 0 == off, default is off

fix ivory tower greek fix. ok angelos@

Don't use an int for the flags, when the structure uses
u_int8_t. Also, make sure the logic is correct (bad theo!)

initial hack at a CIOCSYMFEAT ioctl

driver queueing & callback code for keying operations

crypto_check_alg() is not needed

It looks like there has been one crack smoking and a few cut and pastes.
PR_FREEHEADER should not be set in pool_init by the caller. It shouldn't
be set in pool_init at all. Besides, it's going away soon anyway.

Pool deals fairly well with physical memory shortage, but it doesn't deal
well (not at all) with shortages of the vm_map where the pages are mapped
(usually kmem_map).

Try to deal with it:
- group all information the backend allocator for a pool in a separate
struct. The pool will only have a pointer to that struct.
- change the pool_init API to reflect that.
- link all pools allocating from the same allocator on a linked list.
- Since an allocator is responsible to wait for physical memory it will
only fail (waitok) when it runs out of its backing vm_map, carefully
drain pools using the same allocator so that va space is freed.
(see comments in code for caveats and details).
- change pool_reclaim to return if it actually succeeded to free some
memory, use that information to make draining easier and more efficient.
- get rid of PR_URGENT, noone uses it.

branches: 1.30.2;
and for the case where it allocates a bunch at a time, also make sure the
software flag gets set.

incorrect check

be way more sure that software cannot be used

indent

branches: 1.26.2;
put in tags for ARC4 to please ben, who now has no excuses

KNF

Remove space.

Add crypto_check_alg(), from jgarfiel@seas.upenn.edu

Update copyright; you can use this with or without fee (unless your
name is Theo Deraadt)

New prototype for crypto_register(), to take into account maximum key
length (for PK operations) and various flags.

Structures for public key operations (DH, RSA, DSA). A lot of this
work was done by jgarfiel@seas.upenn.edu

merge crypto/crypto{dev,}.h to crypto/cryptodev.h, to avoid name conflicts inside OpenSSL codebase

KNF

Use pool(9) for some of the structures, and splimp/splx to protect
from ourselves. Should solve PR 1850.

initial cut at /dev/crypto support. takes original mbuf "try, and discard
if we fail" semantics and extends to two varients of data movement: mbuf,
or an iovec style block.

Use the M_CRYPTO_DATA and M_CRYPTO_OPS malloc types.

make the 31-bit code work on 32-bit machines.

branches: 1.14.2;
avoid excessive wakeup(). we think this is safe...

MALLOC/FREE -> malloc/free ok art@ angelos@

Fix tail queue leakage (zzlevo@dd.chalmers.se)

crypto_done(), all it does for now is invoke the callback.

Add Rijndael (128-bit blocksize) in the software crypto driver.

Hacking at OpenBSD Crypto 2000 :-)

Move prototype to include file.

Crypto kernel thread interface; requests are enqueued and processed by
a kernel thread. This allows a much cleaner interface with respect to
spl levels.

OpenBSD tags

crypto_dispatch() only returns an error if the argument it was
provided was NULL or no callback was specified.

avoid using void * when we are talking about pointers

Change the type of freesession to take u_int64_t as argument.

Add a few newlines for readability.

branches: 1.2.2;
split crypto driver front-end from software crypto engine

Cryptographic services framework, and software "device driver". The
idea is to support various cryptographic hardware accelerators (which
may be (detachable) cards, secondary/tertiary/etc processors,
software crypto, etc). Supports session migration between crypto
devices. What it doesn't (yet) support:
- multiple instances of the same algorithm used in the same session
- use of multiple crypto drivers in the same session
- asymmetric crypto

No support for a userland device yet.

IPsec code path modified to allow for asynchronous cryptography
(callbacks used in both input and output processing). Some unrelated
code simplification done in the process (especially for AH).

Development of this code kindly supported by Network Security
Technologies (NSTI). The code was writen mostly in Greece, and is
being committed from Montreal.

The function crypto_dispatch() never returns an error. Make it
void and remove error handling in the callers.
OK patrick@ mvs@

The kernel crypto framework sometimes returned an error, sometimes
the callback was called, and sometimes both. So the caller of that
API could not release resources correctly.
A bunch of errors can or should not happen, replace them with an
assert. Remove redundant checks. crypto_invoke() should not return
the error, but pass it via callback.
Some old hardware drivers keep part of their inconsistency as I
cannot test them.
OK mpi@

Do not queue crypto operations for IPsec. The packet entries in
task queues were unlimited and could overflow during havy traffic.
Even if we still use hardware drivers that sleep, softnet task
instead of soft interrupt can handle this now. Without queues net
lock is inherited and kernel lock is only needed once per packet.
This results in less lock contention and faster IPsec.
Also protect tdb drop counters with net lock and avoid a leak in
crypto dispatch error handling.
intense testing Hrvoje Popovski; OK mpi@

Propagate errors from crypto_invoke() and count them in IPsec. They
should not happen, but always check error conditions. tq is never
NULL, remove the check. tdb->tdb_odrops++ is not MP safe, but will
be addressed separately in ipsec_output_cb().
OK mvs@

Remove unused variable cryptodesc_pool. Document global variables
in crypto.c and annotate locking protection. Assert kernel lock
where needed. Remove dead code from crypto_get_driverid(). Move
crypto_init() prototype into header file.
OK mpi@

Break crypto_unregister() sanity check into two expressions, making it
clearer what is being checked.

Original suggestion from kettenis@.

Don't access past end of cc_alg[] when trying to avoid
unregistering an invalid algorithm.

CID 1453298

ok kettenis@ (with suggested improvements to come)

Fix the IPL and flags of the MP-safe crypto taskq. Now a sane IPL
is passed to the mutex implementation, and the queue actually runs
without the kernel lock.

Tested by dhill@
OK mikeb@, dhill@, kettenis@

Reduce the per-packet allocation costs for crypto operations (cryptop)
by pre-allocating two cryptodesc objects and storing them in an array
instead of a linked list. If more than two cryptodesc objects are
required use mallocarray to fetch them. Adapt the drivers to the new
API.

This change results in one pool-get per ESP packet instead of three.
It also simplifies softraid crypto where more cryptodesc objects are
allocated than used.

From, with and ok markus@, ok bluhm@
"looks sane" mpi@

convert bcopy to memcpy. from david hill

all pools have their ipl set via pool_setipl, so fold it into pool_init.

the ioff argument to pool_init() is unused and has been for many
years, so this replaces it with an ipl argument. because the ipl
will be set on init we no longer need pool_setipl.

most of these changes have been done with coccinelle using the spatch
below. cocci sucks at formatting code though, so i fixed that by hand.

the manpage and subr_pool.c bits i did myself.

ok tedu@ jmatthew@

@ipl@
expression pp;
expression ipl;
expression s, a, o, f, m, p;
@@
-pool_init(pp, s, a, o, f, m, p);
-pool_setipl(pp, ipl);
+pool_init(pp, s, a, ipl, f, m, p);

Add a mechanism for dispatching mpsafe crypto operations. This adds a new
CRYPTOCAP_F_MPSAFE flag that crypto implementations can set to indicate that
their cc_process() implementation can safely run without holding the kernel
lock.

ok mikeb@

fairly simple sizes for free(); ok tedu

we want to defer work traditionally (in openbsd) handled in an
interrupt context to a taskq running in a thread. however, there
is a concern that if we do that then we allow accidental use of
sleeping APIs in this work, which will make it harder to move the
work back to interrupts in the future.

guenther and kettenis came up with the idea of marking a proc with
CANTSLEEP which the sleep paths can check and panic on.

this builds on that so you create taskqs that run with CANTSLEEP
set except when they need to sleep for more tasks to run.

the taskq_create api is changed to take a flags argument so users
can specify CANTSLEEP. MPSAFE is also passed via this flags field
now. this means archs that defined IPL_MPSAFE to 0 can now create
mpsafe taskqs too.

lots of discussion at s2k15
ok guenther@ miod@ mpi@ tedu@ pelikan@

remove the second void * argument on tasks.

when workqs were introduced, we provided a second argument so you
could pass a thing and some context to work on it in. there were
very few things that took advantage of the second argument, so when
i introduced pools i suggested removing it. since tasks were meant
to replace workqs, it was requested that we keep the second argument
to make porting from workqs to tasks easier.

now that workqs are gone, i had a look at the use of the second
argument again and found only one good use of it (vdsp(4) on sparc64
if you're interested) and a tiny handful of questionable uses. the
vast majority of tasks only used a single argument. i have since
modified all tasks that used two args to only use one, so now we
can remove the second argument.

so this is a mechanical change. all tasks only passed NULL as their
second argument, so we can just remove it.

ok krw@

pools lock themselves now, we just have to tell them what IPL they
will be used from.

this adds pool_setipl at IPL_VM to the crypto descriptor pools, and
removes all the splvm handling around the use of those pools.

tested by many via tech@
ok kettenis@ deraadt@

apply only the bit of r1.69 that should have been committed:

make the crypto taskq protect things at IPL_VM instead of IPL_HIGH.

everything else in crypto.c uses splvm/IPL_VM. it seems this IPL_HIGH
came about because the hand rolled task list and thread that crypto
used to use was converted to workqs, which unconditionally used
IPL_HIGH internally. when it was converted from workqs to tasks it
blindly ported the protection workqs gave.

tested by many via tech@ and snapshots
ok kettenis@

revert previous. it did more than the commit message said it did.

make the crypto taskq protect things at IPL_VM instead of IPL_HIGH.

everything else in crypto.c uses splvm/IPL_VM. it seems this IPL_HIGH
came about because the hand rolled task list and thread that crypto
used to use was converted to workqs, which unconditionally used
IPL_HIGH internally. when it was converted from workqs to tasks it
blindly ported the protection workqs gave.

tested by many via tech@ and snapshots
ok kettenis@

replace bzeros after allocations with M_ZERO and PR_ZERO as appropriate.

ok deraadt@

remove uneeded proc.h includes
ok mpi@ kspillner@

Bye bye /dev/crypto

The interface has been disabled by default for about 4 years and
currently there's not much value in having it around at all.

ok deraadt

use mallocarray()

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

cc_queued is not used for anything atm, remove it; ok jsing, markus

Respect CRYPTO_F_NOQUEUE flag when dispatching a crypto operation

ok jsing, markus

convert crypto work queue to the task_add(9) api; ok dlg

institute a hard cap on crypto devs instead of a useless wraparound check
ok beck

for key material that is being being discarded, convert bzero() to
explicit_bzero() where required
ok markus mikeb

Reintroduce most crypto/crypto.c r1.55:

Move pool initialization to init_crypto and zap the crypto_pool_initialized
variable. This way we don't have to check if the pool are initialized every
time we do a crypto_getreq().

However, also perform the crypto initialisation earlier in init_main so
that the crypto pools are initialised before they are used.

ok mikeb@ thib@ deraadt@

Backout r1.55 since this breaks anything which does crypto ops prior to
init_crypto() being called from late in init_main(). In particular, this
breaks softraid crypto volumes that are assembled at boot.

No cookies for thib/mikeb!

"Back it out, right now" deraadt@

Revert part of previous.

The splvm protection is needed after all, as we are walking the list
of registered crypto drivers and doing that unprotected is unwise.

Pointed out by kettenis@

Move pool initialization to init_crypto and zap the crypto_pool_initialized
variable. This way we don't have to check if the pool are initialized every
time we do a crypto_getreq().

Move splvm lower as it isnt need all through crypto_newsession().

tiny KNF nit.

From mikeb

OK deraadt@

Remove the CRYPTO_ALGORITHM_ALL define, fixup accordingly
and make the loop invartiants <= CRYPTO_ALGORITHM_MAX
Do this also for the CRK_ALGORITHM_MAX this also fixes
the a bug that caused us to skip CRK_DH_COMPUTE_KEY.

ok deraadt@

crypto hardware (eg, hifn) establishes its interrupt handler at
IPL_NET. when the hardware finishes some work for the crypto subsystem
and therefore something in the kernel that wanted crypto done, it
calls crypto_done from that interrupt handler.

one of the things that uses crypto is ipsec. when crypto is done
for ipsec it then pushes the packet along the network stack. the
problem is that all the structures inside the network stack are
only protected at splsoftnet. we could be in the middle of modifications
to the pf state table or the pfsync queues when we get a hifn
interrupt and then go stomp on the same structures.

the solution is to defer the completions so they can do the right
spl protections.

this basically reverts r1.46 of src/sys/crypto/crypto.c.

found by naddy@

reintroduce mutexes to workqs for locking.

tested by many on many archs including several alpha test.

ok tedu@ go for it deraadt@

finish conversion to workq. remove list remnants, and put spl in the right
places. handle the no workq case here. ok deraadt

convert crypto thread to workq. add WQ_DIRECTOK flag to workq.
combined, this lets us use crypto before the thread is running
and therefore cryptoraid can attach nice and early.
ok/testing deraadt mbalmer marco

do not call crypto_done() on errors, since the drivers already do this.
otherwise we call the callback twice; fixes panics on crypto errors as
seen on reboot; ok hshoexer

remove some silly casts. put spl calls after all declarations.
put one splx in a better spot. make a variable size MALLOC use malloc.
remove null test after malloc(M_WAITOK).
add PR_NOWAIT flag to pool_get instead of 0. change callbacks to correct type.
ok brad deraadt markus mickey

splimp -> splvm

ok miod@

Don't use crypto thread for callbacks.
This primarily improves IPsec performance when using crypto accelerators.
With help from markus@, tested by wvdputte@.

ok deraadt@, markus@

In crypto_thread(), always save return value from splimp(). We were only
storing it once on kernel startup. Scary. "holy crap" --deraadt. art@ ok

Unclear if this was actually a problem in practice, but this doesn't hurt.

Fastpath for userland crypto requests. This change makes userland
crypto requests attempt to call the crypto driver directly to process
crypto layer requests, as opposed to queueing them in the kernel
crypto thread. If we can't use the crypto devices (i.e. they're busy)
we fall back to queueing the request up in the crypto thread as
before. This does allow for faster performance in some cases (smaller
requests, how small seems to be dependent on the card/cpu combination)
where context switching is a major issue in performance.
ok deraadt@ jason@

Copy the ENTIRE table into the supported algorithms (how the hell did this
work before?!)

From Angelos:
- simplistic load balancing across multiple cards
- simplified registration process
- a few style nits.

I don't know why this breaks things for me when sshd starts on sparc64.
But after wasting the whole day trying to just locate the problem I don't care.
Back out since this wasn't tested and showed to anyone else.

Double-pass over drivers, first hardware only, then software (if we
are interested in software).

Fix a typo, cleanup on session migration code in crypto_invoke(), and
add a convention that if the driver returns ERESTART as an error
message of its process method, the crypto framework will unregister
the driver and migrate all its sessions. After discussion with Sam
Leffler and Jason Wright.

kernel changes to make asymmetric crypto work in userland
- modify getfeat to return something more useful to us on devices
(like lofn and everything else until jason fixes it) that can't
do rsa stuff, etc and can only do mod_exp..
- error handling fixes so we correctly fail to software when we can't
deal with a particular key size
- add sysctl kern.userasymcrypto to turn on/off userland asymmetric crypto
via /dev/crypto - 1 == on, 0 == off, default is off

fix ivory tower greek fix. ok angelos@

Don't use an int for the flags, when the structure uses
u_int8_t. Also, make sure the logic is correct (bad theo!)

initial hack at a CIOCSYMFEAT ioctl

driver queueing & callback code for keying operations

crypto_check_alg() is not needed

It looks like there has been one crack smoking and a few cut and pastes.
PR_FREEHEADER should not be set in pool_init by the caller. It shouldn't
be set in pool_init at all. Besides, it's going away soon anyway.

Pool deals fairly well with physical memory shortage, but it doesn't deal
well (not at all) with shortages of the vm_map where the pages are mapped
(usually kmem_map).

Try to deal with it:
- group all information the backend allocator for a pool in a separate
struct. The pool will only have a pointer to that struct.
- change the pool_init API to reflect that.
- link all pools allocating from the same allocator on a linked list.
- Since an allocator is responsible to wait for physical memory it will
only fail (waitok) when it runs out of its backing vm_map, carefully
drain pools using the same allocator so that va space is freed.
(see comments in code for caveats and details).
- change pool_reclaim to return if it actually succeeded to free some
memory, use that information to make draining easier and more efficient.
- get rid of PR_URGENT, noone uses it.

branches: 1.30.2;
and for the case where it allocates a bunch at a time, also make sure the
software flag gets set.

incorrect check

be way more sure that software cannot be used

indent

branches: 1.26.2;
put in tags for ARC4 to please ben, who now has no excuses

KNF

Remove space.

Add crypto_check_alg(), from jgarfiel@seas.upenn.edu

Update copyright; you can use this with or without fee (unless your
name is Theo Deraadt)

New prototype for crypto_register(), to take into account maximum key
length (for PK operations) and various flags.

Structures for public key operations (DH, RSA, DSA). A lot of this
work was done by jgarfiel@seas.upenn.edu

merge crypto/crypto{dev,}.h to crypto/cryptodev.h, to avoid name conflicts inside OpenSSL codebase

KNF

Use pool(9) for some of the structures, and splimp/splx to protect
from ourselves. Should solve PR 1850.

initial cut at /dev/crypto support. takes original mbuf "try, and discard
if we fail" semantics and extends to two varients of data movement: mbuf,
or an iovec style block.

Use the M_CRYPTO_DATA and M_CRYPTO_OPS malloc types.

make the 31-bit code work on 32-bit machines.

branches: 1.14.2;
avoid excessive wakeup(). we think this is safe...

MALLOC/FREE -> malloc/free ok art@ angelos@

Fix tail queue leakage (zzlevo@dd.chalmers.se)

crypto_done(), all it does for now is invoke the callback.

Add Rijndael (128-bit blocksize) in the software crypto driver.

Hacking at OpenBSD Crypto 2000 :-)

Move prototype to include file.

Crypto kernel thread interface; requests are enqueued and processed by
a kernel thread. This allows a much cleaner interface with respect to
spl levels.

OpenBSD tags

crypto_dispatch() only returns an error if the argument it was
provided was NULL or no callback was specified.

avoid using void * when we are talking about pointers

Change the type of freesession to take u_int64_t as argument.

Add a few newlines for readability.

branches: 1.2.2;
split crypto driver front-end from software crypto engine

Cryptographic services framework, and software "device driver". The
idea is to support various cryptographic hardware accelerators (which
may be (detachable) cards, secondary/tertiary/etc processors,
software crypto, etc). Supports session migration between crypto
devices. What it doesn't (yet) support:
- multiple instances of the same algorithm used in the same session
- use of multiple crypto drivers in the same session
- asymmetric crypto

No support for a userland device yet.

IPsec code path modified to allow for asynchronous cryptography
(callbacks used in both input and output processing). Some unrelated
code simplification done in the process (especially for AH).

Development of this code kindly supported by Network Security
Technologies (NSTI). The code was writen mostly in Greece, and is
being committed from Montreal.

Do not queue crypto operations for IPsec. The packet entries in
task queues were unlimited and could overflow during havy traffic.
Even if we still use hardware drivers that sleep, softnet task
instead of soft interrupt can handle this now. Without queues net
lock is inherited and kernel lock is only needed once per packet.
This results in less lock contention and faster IPsec.
Also protect tdb drop counters with net lock and avoid a leak in
crypto dispatch error handling.
intense testing Hrvoje Popovski; OK mpi@

Propagate errors from crypto_invoke() and count them in IPsec. They
should not happen, but always check error conditions. tq is never
NULL, remove the check. tdb->tdb_odrops++ is not MP safe, but will
be addressed separately in ipsec_output_cb().
OK mvs@

Remove unused variable cryptodesc_pool. Document global variables
in crypto.c and annotate locking protection. Assert kernel lock
where needed. Remove dead code from crypto_get_driverid(). Move
crypto_init() prototype into header file.
OK mpi@

Break crypto_unregister() sanity check into two expressions, making it
clearer what is being checked.

Original suggestion from kettenis@.

Don't access past end of cc_alg[] when trying to avoid
unregistering an invalid algorithm.

CID 1453298

ok kettenis@ (with suggested improvements to come)

Fix the IPL and flags of the MP-safe crypto taskq. Now a sane IPL
is passed to the mutex implementation, and the queue actually runs
without the kernel lock.

Tested by dhill@
OK mikeb@, dhill@, kettenis@

Reduce the per-packet allocation costs for crypto operations (cryptop)
by pre-allocating two cryptodesc objects and storing them in an array
instead of a linked list. If more than two cryptodesc objects are
required use mallocarray to fetch them. Adapt the drivers to the new
API.

This change results in one pool-get per ESP packet instead of three.
It also simplifies softraid crypto where more cryptodesc objects are
allocated than used.

From, with and ok markus@, ok bluhm@
"looks sane" mpi@

convert bcopy to memcpy. from david hill

all pools have their ipl set via pool_setipl, so fold it into pool_init.

the ioff argument to pool_init() is unused and has been for many
years, so this replaces it with an ipl argument. because the ipl
will be set on init we no longer need pool_setipl.

most of these changes have been done with coccinelle using the spatch
below. cocci sucks at formatting code though, so i fixed that by hand.

the manpage and subr_pool.c bits i did myself.

ok tedu@ jmatthew@

@ipl@
expression pp;
expression ipl;
expression s, a, o, f, m, p;
@@
-pool_init(pp, s, a, o, f, m, p);
-pool_setipl(pp, ipl);
+pool_init(pp, s, a, ipl, f, m, p);

Add a mechanism for dispatching mpsafe crypto operations. This adds a new
CRYPTOCAP_F_MPSAFE flag that crypto implementations can set to indicate that
their cc_process() implementation can safely run without holding the kernel
lock.

ok mikeb@

fairly simple sizes for free(); ok tedu

we want to defer work traditionally (in openbsd) handled in an
interrupt context to a taskq running in a thread. however, there
is a concern that if we do that then we allow accidental use of
sleeping APIs in this work, which will make it harder to move the
work back to interrupts in the future.

guenther and kettenis came up with the idea of marking a proc with
CANTSLEEP which the sleep paths can check and panic on.

this builds on that so you create taskqs that run with CANTSLEEP
set except when they need to sleep for more tasks to run.

the taskq_create api is changed to take a flags argument so users
can specify CANTSLEEP. MPSAFE is also passed via this flags field
now. this means archs that defined IPL_MPSAFE to 0 can now create
mpsafe taskqs too.

lots of discussion at s2k15
ok guenther@ miod@ mpi@ tedu@ pelikan@

remove the second void * argument on tasks.

when workqs were introduced, we provided a second argument so you
could pass a thing and some context to work on it in. there were
very few things that took advantage of the second argument, so when
i introduced pools i suggested removing it. since tasks were meant
to replace workqs, it was requested that we keep the second argument
to make porting from workqs to tasks easier.

now that workqs are gone, i had a look at the use of the second
argument again and found only one good use of it (vdsp(4) on sparc64
if you're interested) and a tiny handful of questionable uses. the
vast majority of tasks only used a single argument. i have since
modified all tasks that used two args to only use one, so now we
can remove the second argument.

so this is a mechanical change. all tasks only passed NULL as their
second argument, so we can just remove it.

ok krw@

pools lock themselves now, we just have to tell them what IPL they
will be used from.

this adds pool_setipl at IPL_VM to the crypto descriptor pools, and
removes all the splvm handling around the use of those pools.

tested by many via tech@
ok kettenis@ deraadt@

apply only the bit of r1.69 that should have been committed:

make the crypto taskq protect things at IPL_VM instead of IPL_HIGH.

everything else in crypto.c uses splvm/IPL_VM. it seems this IPL_HIGH
came about because the hand rolled task list and thread that crypto
used to use was converted to workqs, which unconditionally used
IPL_HIGH internally. when it was converted from workqs to tasks it
blindly ported the protection workqs gave.

tested by many via tech@ and snapshots
ok kettenis@

revert previous. it did more than the commit message said it did.

make the crypto taskq protect things at IPL_VM instead of IPL_HIGH.

everything else in crypto.c uses splvm/IPL_VM. it seems this IPL_HIGH
came about because the hand rolled task list and thread that crypto
used to use was converted to workqs, which unconditionally used
IPL_HIGH internally. when it was converted from workqs to tasks it
blindly ported the protection workqs gave.

tested by many via tech@ and snapshots
ok kettenis@

replace bzeros after allocations with M_ZERO and PR_ZERO as appropriate.

ok deraadt@

remove uneeded proc.h includes
ok mpi@ kspillner@

Bye bye /dev/crypto

The interface has been disabled by default for about 4 years and
currently there's not much value in having it around at all.

ok deraadt

use mallocarray()

add a size argument to free. will be used soon, but for now default to 0.
after discussions with beck deraadt kettenis.

cc_queued is not used for anything atm, remove it; ok jsing, markus

Respect CRYPTO_F_NOQUEUE flag when dispatching a crypto operation

ok jsing, markus

convert crypto work queue to the task_add(9) api; ok dlg

institute a hard cap on crypto devs instead of a useless wraparound check
ok beck

for key material that is being being discarded, convert bzero() to
explicit_bzero() where required
ok markus mikeb

Reintroduce most crypto/crypto.c r1.55:

Move pool initialization to init_crypto and zap the crypto_pool_initialized
variable. This way we don't have to check if the pool are initialized every
time we do a crypto_getreq().

However, also perform the crypto initialisation earlier in init_main so
that the crypto pools are initialised before they are used.

ok mikeb@ thib@ deraadt@

Backout r1.55 since this breaks anything which does crypto ops prior to
init_crypto() being called from late in init_main(). In particular, this
breaks softraid crypto volumes that are assembled at boot.

No cookies for thib/mikeb!

"Back it out, right now" deraadt@

Revert part of previous.

The splvm protection is needed after all, as we are walking the list
of registered crypto drivers and doing that unprotected is unwise.

Pointed out by kettenis@

Move pool initialization to init_crypto and zap the crypto_pool_initialized
variable. This way we don't have to check if the pool are initialized every
time we do a crypto_getreq().

Move splvm lower as it isnt need all through crypto_newsession().

tiny KNF nit.

From mikeb

OK deraadt@

Remove the CRYPTO_ALGORITHM_ALL define, fixup accordingly
and make the loop invartiants <= CRYPTO_ALGORITHM_MAX
Do this also for the CRK_ALGORITHM_MAX this also fixes
the a bug that caused us to skip CRK_DH_COMPUTE_KEY.

ok deraadt@

crypto hardware (eg, hifn) establishes its interrupt handler at
IPL_NET. when the hardware finishes some work for the crypto subsystem
and therefore something in the kernel that wanted crypto done, it
calls crypto_done from that interrupt handler.

one of the things that uses crypto is ipsec. when crypto is done
for ipsec it then pushes the packet along the network stack. the
problem is that all the structures inside the network stack are
only protected at splsoftnet. we could be in the middle of modifications
to the pf state table or the pfsync queues when we get a hifn
interrupt and then go stomp on the same structures.

the solution is to defer the completions so they can do the right
spl protections.

this basically reverts r1.46 of src/sys/crypto/crypto.c.

found by naddy@

reintroduce mutexes to workqs for locking.

tested by many on many archs including several alpha test.

ok tedu@ go for it deraadt@

finish conversion to workq. remove list remnants, and put spl in the right
places. handle the no workq case here. ok deraadt

convert crypto thread to workq. add WQ_DIRECTOK flag to workq.
combined, this lets us use crypto before the thread is running
and therefore cryptoraid can attach nice and early.
ok/testing deraadt mbalmer marco

do not call crypto_done() on errors, since the drivers already do this.
otherwise we call the callback twice; fixes panics on crypto errors as
seen on reboot; ok hshoexer

remove some silly casts. put spl calls after all declarations.
put one splx in a better spot. make a variable size MALLOC use malloc.
remove null test after malloc(M_WAITOK).
add PR_NOWAIT flag to pool_get instead of 0. change callbacks to correct type.
ok brad deraadt markus mickey