Zap a trailing blank that snuck into ssl3_get_client_hello()

Remove GOST and STREEBOG support from libssl.

This version of GOST is old and not anywhere close to compliant with
modern GOST standards. It is also very intrusive in libssl and
makes a mess everywhere. Efforts to entice a suitably minded anyone
to care about it have been unsuccessful.

At this point it is probably best to remove this, and if someone
ever showed up who truly needed a working version, it should be
a clean implementation from scratch, and have it use something
closer to the typical API in libcrypto so it would integrate less
painfully here.

This removes it from libssl in preparation for it's removal from
libcrypto with a future major bump

ok tb@

Neuter the SSL_set_debug(3) API

The TLSv1.3 stack didn't support this in the first place, and in the legacy
stack it only added some dubious BIO_flush(3) calls. The sleep call between
SSL_read(3) and SSL_write(3) advertised in the comment next to the flag has
been a sleep call in the s_server since time immemorial, nota bene between
calls to BIO_gets(3). Anyway. This can all go and what remains will go with
the next major bump.

ok jsing

Check for negative EVP_CIPHER_CTX_iv_length() return in libssl

ok beck

Hide all public symbols in libssl

With the guentherizer 9000

ok tb@

Convert legacy server kex to one-shot sign/verify

This converts ssl3_{get,send}_server_key_exchange() to EVP_DigestVerify()
and EVP_DigestSign(). In order to do this, build the full signed_params
up front and rework the way the key exchange parameters are constructed.
This way we can do the verify and sign steps in one go and at the same
use a more idiomatic approach with CBB/CBS.

with/ok jsing

Easy EVP_Digest{Sign,Verify} conversions for legacy stack

Convert ssl3_send_client_verify_{sigalgs,gost}() to EVP_DigestSign() and
ssl3_get_cert_verify() to EVP_DigestVerify().

ok jsing

spelling fixes; from paul tagliamonte
ok tb

Make internal header file names consistent

Libcrypto currently has a mess of *_lcl.h, *_locl.h, and *_local.h names
used for internal headers. Move all these headers we inherited from
OpenSSL to *_local.h, reserving the name *_internal.h for our own code.
Similarly, move dtls_locl.h and ssl_locl.h to dtls_local and ssl_local.h.
constant_time_locl.h is moved to constant_time.h since it's special.

Adjust all .c files in libcrypto, libssl and regress.

The diff is mechanical with the exception of tls13_quic.c, where
#include <ssl_locl.h> was fixed manually.

discussed with jsing,
no objection bcook

Get rid of SSL_CTX_INTERNAL and SSL_INTERNAL.

These are no longer necessary due to SSL_CTX and SSL now being fully
opaque. Merge SSL_CTX_INTERNAL back into SSL_CTX and SSL_INTERNAL back
into SSL.

Prompted by tb@

Move handshake message handling functions from ssl_both.c to client/server.

Currently, ssl_both.c contains several functions that are used by both the
legacy client and legacy server. This interwines the client and server,
making it harder to make progressive changes. While it does deduplicate
some code, it also ends up with code that is conditioned on s->server and
forces the caller to pass in SSL3_ST_* values.

Move these functions from ssl_both.c into ssl_clnt.c and ssl_srvr.c,
renaming as appropriate and removing the s->server conditionals. Also move
the client and server function prototypes from ssl_locl.h into the .c
files, making them static in the process.

ok tb@

Deduplicate peer certificate chain processing code.

Rather than reimplement this in each TLS client and server, deduplicate it
into a single function. Furthermore, rather than dealing with the API
hazard that is SSL_get_peer_cert_chain() in this code, simply produce two
chains - one that has the leaf and one that does not.
SSL_get_peer_cert_chain() can then return the appropriate one.

This also moves the peer cert chain from the SSL_SESSION to the
SSL_HANDSHAKE, which makes more sense since it is not available on
resumption.

ok tb@

Simplify certificate list handling code in legacy server.

A client is required to send an empty list if it does not have a suitable
certificate - handle this case up front, rather than going through the
normal code path and ending up with an empty certificate list. This matches
what we do in the TLSv1.3 stack and will allow for ruther clean up (in
addition to making the code more readable).

Also tidy up the CBS code and remove some unnecessary length checks. Use
'cert' and 'certs' for certificates, rather than 'x' and 'sk'.

ok tb@

Rename uses 'curve' to 'group' and rework tls1 group API.

This reworks various tls1_ curve APIs to indicate success via a boolean
return value and move the output to an out parameter. This makes the
caller code easier and more consistent.

Based on a suggestion by jsing

ok jsing

Add checks to ensure we do not initiate or negotiate handshakes with
versions below the minimum required by the security level.

input & ok jsing

Check the security of DH key shares

ok beck, looks good to jsing

Check the security level when building sigalgs

ok beck jsing

Free ciphers before assigning to them

While this is not a leak currently, it definitely looks like one.
Pointed out by jsing on review of a diff that touched the vicinity
a while ago.

ok jsing

Add error checking to tls_session_secret_cb() calls

Failure of this undocumented callback was previously silently ignored.
Follow OpenSSL's behavior and throw an internal error (for lack of a
better choice) if the callback failed or if it set the master_key_length
to a negative number.

Unindent the success path and clean up some strange idioms.

ok jsing

Bye bye S3I.

S3I has served us well, however now that libssl is fully opaque it is time
to say goodbye. Aside from removing the calloc/free/memset, the rest is
mechanical sed.

ok inoguchi@ tb@

Remove peer_pkeys from SSL_SESSION.

peer_pkeys comes from some world where peers can send multiple certificates
- in fact, one of each known type. Since we do not live in such a world,
get rid of peer_pkeys and simply use peer_cert instead (in both TLSv1.2
and TLSv1.3, both clients and servers can only send a single leaf
(aka end-entity) certificate).

ok inoguchi@ tb@

Rename 'peer' to 'peer_cert' in SSL_SESSION.

The 'peer' member of SSL_SESSION is the leaf/end-entity certificate
provided by our peer. Rename it since 'peer' on its own is unhelpful.

ok inoguchi@ tb@

Plumb decode errors through key share parsing code.

Distinguish between decode errors and other errors, so that we can send
a SSL_AD_DECODE_ERROR alert when appropriate.

Fixes a tlsfuzzer failure, due to it expecting a decode error alert and
not receiving one.

Prompted by anton@

ok tb@

Clean up ssl3_{send,get}_client_kex_gost()

Fix leaks, use sizeof() instead of hardcoded sizes, actually check return
codes, explicit_bzero() the premaster secret on the server side and
generally try to kick the GOST kex code into some sort of shape.

ok inoguchi@ tb@

Return 0/1 from ssl3_{send,get}_client_kex_gost()

Like other KEX handling functions, there is no need to return anything
other than failure/success here.

ok inoguchi@ tb@

Fix GOST skip certificate verify handling.

GOST skip certificate verify handling got broken in r1.132 of s3_srvr.c
circa 2016. Prior to this, ssl3_get_client_key_exchange() returned an
'extra special' value to indicate that the state machine should skip
certificate verify. Fix this by setting and checking the
TLS1_FLAGS_SKIP_CERT_VERIFY flag, which is the same as is done in the
client.

ok inoguchi@ tb@

Merge SESS_CERT into SSL_SESSION.

There is no reason for SESS_CERT to exist - remove it and merge its members
into SSL_SESSION for the time being. More clean up to follow.

ok inoguchi@ tb@

Rename CERT to SSL_CERT and CERT_PKEY to SSL_CERT_PKEY.

Nearly all structs in libssl start with an SSL_ suffix, rename CERT and
CERT_PKEY for consistency.

ok inoguchi@ tb@

Rename dh_tmp to dhe_params.

Support for non-ephemeral DH was removed a long time ago - as such, the
dh_tmp and dh_tmp_cb are used for DHE parameters. Rename them to reflect
reality.

ok inoguchi@ tb@

Convert legacy server to tls_key_share.

This requires a few more additions to the DHE key share code - we need to
be able to either set the DHE parameters or specify the number of key bits
for use with auto DHE parameters. Additionally, we need to be able to
serialise the DHE parameters to send to the client.

This removes the infamous 'tmp' struct from ssl3_state_internal_st.

ok inoguchi@ tb@

Return 0 on failure from send/get kex functions in the legacy stack.

In the legacy stack, a message handling function returns -1 for failure,
0 for need more data and 1 for success (although in extra special cases
2 may also be used). However, the various send/get kex functions only
need to indicate success or failure - switch these to return 0 on failure
(rather than -1) and use normal result testing.

This leaves GOST unchanged for now, as that code is special and needs
extra work.

ok inoguchi@ tb@

Hoist memset of CBB above EVP_MD_CTX_new() and HMAC_CTX_new() to avoid
a use of uninitialized in the unlikely event that either of them fails.
Problem introduced in r1.128.

CID 345113

ok jsing

Convert ssl_srvr.c to opaque EVP_MD_CTX.

ok inoguchi jsing

Clean up and refactor server side DHE key exchange.

Provide ssl_kex_generate_dhe_params_auto() which handles DHE key generation
based on parameters determined by the specified key bits. Convert the
existing DHE auto parameter selection code into a function that just tells
us how many key bits to use.

Untangle and rework the server side DHE key exchange to use the ssl_kex_*
functions.

ok inoguchi@ tb@

Convert server serialisation of DHE parameters/public key to new functions.

ok inoguchi@ tb@

Stop reaching into EVP_PKEY in the rest of libssl.

ok inoguchi jsing

libssl: don't reach for pkey->save_type.

For some strange historical reason ECDSA_sign() and ECDSA_verify}() have
a type argument that they ignore. For another strange historical reason,
the type passed to them from libssl is pkey->save_type, which is used to
avoid expensive engine lookups when setting the pkey type... Whatever the
aforementioned reasons were, we can't access pkey->save_type with the
OpenSSL 1.1 API, and this is thus in the way of making EVP_PKEY opaque.
Simply pass in 0 instead.

ok jsing

Fold SSL_SESSION_INTERNAL back into SSL_SESSION.

ok beck@ tb@

Provide a way to determine our maximum legacy version.

With the introduction of TLSv1.3, we need the ability to determine our
maximum legacy version and to track our peer's maximum legacy version.
This is needed for both the TLS record layer when using TLSv1.3, plus
it is needed for RSA key exhange in TLS prior to TLSv1.3, where the
maximum legacy version is incorporated in the pre-master secret to
avoid downgrade attacks.

This unbreaks RSA KEX for the TLS client when the non-version specific
method is used with TLSv1.0 or TLSv1.1 (clearly no one does this).

ok tb@

Fold DTLS1_STATE_INTERNAL into DTLS1_STATE.

Now that DTLS1_STATE is opaque, fold DTLS1_STATE_INTERNAL back into
DTLS1_STATE and remove D1I() usage.

ok tb@

Untangle ssl3_get_message() return values.

This function currently has a long return type that may be <= 0 on
error/retry (which is then cast to an int in order to return it up the
stack), or it returns the length of the handshake message (on success).
This obviously means that 0 can be returned for both success and failure,
which is the reason why a separate 'ok' argument has to exist.

Untangle this mess by changing the return value to an int that indicates
success (1) or error/retry (<= 0). The length never needs to actually be
returned as it is already stored in s->internal->init_num (which is where
the return value is read from anyway).

ok tb@

Ensure that a client hello does not have trailing data.

Found by tlsfuzzer.

ok beck@

Clean up and simplify info and msg callbacks.

The info and msg callbacks result in duplication - both for code that
refers to the function pointers and for the call sites. Avoid this by
providing typedefs for the function pointers and pulling the calling
sequences into their own functions.

ok inoguchi@ tb@

Track sigalg used by ourselves and our peer in the legacy stack.

This is needed for upcoming API additions.

Convert legacy stack server to ssl_sigalg_for_peer().

ok inoguchi@ tb@

Move the RSA-PSS check for TLSv1.3 to ssl_sigalg_pkey_ok().

Also, rather than passing in a check_curve flag, pass in the SSL * and
handle version checks internally to ssl_sigalg_pkey_ok(), simplifying
the callers.

ok inoguchi@ tb@

Change ssl_sigalgs_from_value() to perform sigalg list selection.

Rather that passing in a sigalg list at every call site, pass in the
appropriate TLS version and have ssl_sigalgs_from_value() perform the
sigalg list selection itself. This allows the sigalg lists to be made
internal to the sigalgs code.

ok tb@

Rename ssl_sigalg() to ssl_sigalg_from_value().

This makes the code more self-documenting and avoids the ambiguity between
ssl_sigalg the struct and ssl_sigalg the function.

ok tb@

Change ssl_sigalgs_build() to perform sigalg list selection.

Rather that doing sigalg list selection at every call site, pass in the
appropriate TLS version and have ssl_sigalgs_build() perform the sigalg
list selection itself. This reduces code duplication, simplifies the
calling code and is the first step towards internalising the sigalg lists.

ok tb@

Make local header inclusion consistent.

Consistently include local headers in the same location, using the same
grouping/sorting across all files.

Move DTLS structs/definitions/prototypes to dtls_locl.h.

Now that the DTLS structs are opaque, add a dtls_locl.h header and move
internal-only structs from dtls1.h, along with prototypes from ssl_locl.h.
Only pull this header in where DTLS code actually exists.

ok inoguchi@ tb@

Explicitly include <openssl/opensslconf.h> in files using OPENSSL_NO_*

Where a file references to OPENSSL_NO_* conditions, ensure that we
explicitly include <openssl/opensslconf.h> before any references, rather
than relying on another header to pull this in.

Clean up tls1_change_cipher_state().

Replace flag gymnastics at call sites with separate read and write,
functions which call the common code. Condition on s->server instead of
using SSL_ST_ACCEPT, for consistency and more readable code.

ok inoguchi@ tb@

In the TLSv1.2 server, set up the key block after sending the CCS.

This avoids calling into the key block setup code multiple times and makes
the server code consistent with the client.

ok inoguchi@ tb@

Clean up dtls1_reset_seq_numbers().

Rather than doing flag gymnastics, split dtls1_reset_seq_numbers() into
separate read and write functions. Move the calls of these functions into
tls1_change_cipher_state() so they directly follow the change of cipher
state in the record layer, which avoids having to duplicate the calls in
the client and server.

ok inoguchi@ tb@

Clean up and harden TLSv1.2 master key derivation.

The master key and its length are only stored in one location, so it makes
no sense to handle these outside of the derivation function (the current
'out' argument is unused). This simplifies the various call sites.

If derivation fails for some reason, fail hard rather than continuing on
and hoping that something deals with this correctly later.

ok inoguchi@ tb@

Clean up derivation of finished/peer finished.

Make this process more readable by having specific client/server functions,
calling the correct one based on s->server. This allows to remove various
SSL_ST_ACCEPT/SSL_ST_CONNECT checks, along with duplicate code.

ok inoguchi@ tb@

Clean up TLSv1.2 certificate request handshake data.

Currently cert_req is used by clients and cert_request is used by servers.
Replace this by a single cert_request used by either client or server.
Remove the certificate types as they are currently unused. This also fixes
a bug whereby if the number of certificate types exceeds SSL3_CT_NUMBER
the number of bytes read in is insufficient, which will break decoding.

ok inoguchi@ tb@

Move reuse_message, message_type, message_size and cert_verify into the
TLSv1.2 handshake struct.

ok inoguchi@ tb@

Avoid transcript initialisation when sending a TLS HelloRequest.

When server side renegotiation is triggered, the TLSv1.2 state machine
sends a HelloRequest before going to ST_SW_FLUSH and ST_OK. In this case
we do not need the transcript and currently hit the sanity check in ST_OK
that ensures the transcript has been freed, breaking server initiated
renegotiation. We do however need the transcript in the DTLS case.

ok tb@

Garbage collect s->internal->type

This variable is used in the legacy stack to decide whether we are
a server or a client. That's what s->server is for...

The new TLSv1.3 stack failed to set s->internal->type, which resulted
in hilarious mishandling of previous_{client,server}_finished. Indeed,
both client and server would first store the client's verify_data in
previous_server_finished and later overwrite it with the server's
verify_data. Consequently, renegotiation has been completely broken
for more than a year. In fact, server side renegotiation was broken
during the 6.5 release cycle. Clearly, no-one uses this.

This commit fixes client side renegotiation and restores the previous
behavior of SSL_get_client_CA_list(). Server side renegotiation will
be fixed in a later commit.

ok jsing

Rename new_cipher to cipher.

This is in the SSL_HANDSHAKE struct and is what we're currently
negotiating, so there is really nothing more "new" about the cipher
than there is the key block or other parts of the handshake data.

ok inoguchi@ tb@

Add SSL_HANDSHAKE_TLS12 for TLSv1.2 specific handshake data.

Move TLSv1.2 specific components over from SSL_HANDSHAKE.

ok inoguchi@ tb@

Remove ssl_downgrade_max_version().

Now that we store our maximum TLS version at the start of the handshake,
we can check against that directly.

ok inoguchi@ tb@

Improve internal version handling.

Add handshake fields for our minimum TLS version, our maximum TLS version
and the TLS version negotiated during the handshake. Initialise our min/max
versions at the start of the handshake and leave these unchanged. The
negotiated TLS version is set in the client once we receive the ServerHello
and in the server at the point we select the highest shared version.

Provide an ssl_effective_version() function that returns the negotiated TLS
version if known, otherwise our maximum TLS version - this is effectively
what is stored in s->version currently.

Convert most of the internal code to use one of these three version fields,
which greatly simplifies code (especially in the TLS extension handling
code).

ok tb@

ugly whitespace

Rename f_err into fatal_err.

discussed with jsing

Rename the truncated label into decode_err. This describes its purpose
better and doesn't look odd if there's trailing data for exapmle.
Indent a few labels in the neighborhood while there.

ok jsing

Revise HelloVerifyRequest handling for DTLSv1.2.

Per RFC 6347 section 4.2.1, the HelloVerifyRequest should always contain
DTLSv1.0 - ensure this is the case on the server side, allow both DTLSv1.0
and DTLSv1.2 on the client.

ok tb@

Factor out the legacy stack version checks.

Also check for explicit version numbers, rather than just the major version
value.

ok tb@

Move sequence numbers into the new TLSv1.2 record layer.

This allows for all of the DTLS sequence number save/restore code to be
removed.

ok inoguchi@ "whee!" tb@

Provide functions to determine if TLSv1.2 record protection is engaged.

Call these functions from code that needs to know if we've changed cipher
state and enabled record protection, rather than inconsistently checking
various pointers from other places in the code base. This also fixes a
minor bug where the wrong pointers are checked if we're operating with
AEAD.

ok inoguchi@ tb@

Replace SSL_IS_DTLS with SSL_is_dtls().

Garbage collect the now unused SSL_IS_DTLS macro.

ok tb@

Condense and simplify TLS methods.

Historically, OpenSSL has had client and server specific methods - the only
difference between these is that the .ssl_connect or .ssl_accept function
pointer is set to ssl_undefined_function, with the intention of reducing
code size for a statically linked binary that was only a client or server.
These days the difference is minimal or non-existant in many cases and
we can reduce the amount of code and complexity by having single method.

Internally remove all of the client and server specific methods,
simplifying code in the process. The external client/server specific API
remain, however these now return the same thing as TLS_method() does.

ok tb@

Merge d1_{clnt,srvr}.c into ssl_{clnt,srvr}.c

The d1_{clnt,srvr}.c contain a single function each - merge these into the
ssl_{clnt,srvr}.c, renaming them with an ssl3_ prefix for consistency.

ok beck@ tb@

Simplify the cleanup of init_buf via a ssl3_release_init_buffer() function.

ok beck@ inoguchi@ tb@

Simplify SSL method lookups.

There are three places where we call tls1_get_{client,server}_method() and
if that returns NULL, call dtls1_get_{client,server}_method(). Simplify
this by combining the lookup into a single function. While here also use
uint16_t for version types.

ok inoguchi@ millert@

Use the correct type for tls1_set_ec_id()

The curve_id is a uint16, not an int.

ok beck jsing

Remove cipher_list_by_id.

When parsing a cipher string, a cipher list is created, before being
duplicated and sorted - the second copy being stored as cipher_list_by_id.
This is done only so that a client can ensure that the cipher selected by
a server is in the cipher list. This is pretty pointless given that most
clients are short-lived and that we already had to iterate over the cipher
list in order to build the client hello. Additionally, any update to the
cipher list requires that cipher_list_by_id also be updated and kept in
sync.

Remove all of this and replace it with a simple linear scan - the overhead
of duplicating and sorting the cipher list likely exceeds that of a simple
linear scan over the cipher list (64 maximum, more typically ~9 or so).

ok beck@ tb@

Send alert on ssl_get_prev_session failure

ssl_get_prev_session() can fail for various reasons some of which
may be internal_error others decode_error alerts. Propagate the
appropriate alert up to the caller so we can abort the handshake
by sending a fatal alert instead of rudely closing the pipe.

Currently only 28 of 292 test cases of tlsfuzzer's test-extension.py pass.
With this diff, 272 pass. The rest will require fixes elsewhere.

ok beck inoguchi jsing

Improve argument order for the internal tlsext API

Move is_server and msg_type right after the SSL object so that CBS
and CBB and alert come last. This brings these functions more in
line with other internal functions and separates state from data.

requested by jsing

Enable GOST_SIG_FORMAT_RS_LE when verifying certificate signatures.

GOST cipher suites requires that CertVerify signatures be generated in a
special way (see ssl3_send_client_kex_gost(), ssl3_get_cert_verify()).
However, the GOST_SIG_FORMAT_RS_LE flag was not passed in case of TLS 1.2
connections (because they use different code path). Set this flag on
GOST PKEYs.

Diff from Dmitry Baryshkov <dbaryshkov@gmail.com>

Sponsored by ROSA Linux

ok inoguchi@ tb@

Send an illegal_parameter alert if a client sends us invalid DH key
shares. Previously we would fail and just close the pipe.

Fixes the remaining failing test-dhe-rsa-key-exchange-with-bad-messages.py
tests of tlsfuzzer.

ok beck (earlier version) jsing

Correct downgrade sentinels when a version pinned method is in use.

Previously only the enabled protocol versions were considered, however we
also have to consider the method in use which may be version pinned.

Found the hard way by danj@ with haproxy and force-tlsv12.

ok beck@ inoguchi@ tb@

Replace SSL_PKEY_RSA_ENC/SSL_PKEY_RSA_SIGN with SSL_PKEY_RSA.

Some time prior to SSLeay 0.8.1b, SSL_PKEY_RSA_SIGN got added with the
intention of handling RSA sign only certificates... this incomplete code
had the following comment:

/* check to see if this is a signing only certificate */
/* EAY EAY EAY EAY */

And while the comment was removed in 2005, the incomplete RSA sign-only
handling has remained ever since.

Remove SSL_PKEY_RSA_SIGN and rename SSL_PKEY_RSA_ENC to SSL_PKEY_RSA. While
here also remove the unused SSL_PKEY_DH_RSA.

ok tb@

Use size_t for OCSP response length.

The OCSP response length is currently an integer, which is overloaded with
-1 meaning "unset". Use a size_t for the OCSP response length and infer
unset from the OCSP response being NULL. This makes code more readable,
simpler and less error prone.

ok beck@

On receiving an overlong session ID terminate with an illegal_parameter
alert. Found with tlsfuzzer.

ok jsing

TLSv1.3 servers that intend to downgrade are required to set the last
eight bytes of the server's random to a magic cookie (RFC 8446, 4.1.3).
The TLSv1.3 spec changes the TLSv1.2 spec in that it recommends that
TLSv1.2 servers that negotiate TLSv1.1 or below do the same. This gives
a limited additional protection against downgrade attacks beyond what is
already present in the Finished exchange.

The TLSv1.3 part was already implemented in Hobart and can be trivially
modified to do the TLSv1.2 bit as well.

ok inoguchi, jsing

Avoid leak for tmp.x25519

Changed to use local variable to hold malloc address rather than directly
set to S3I(s)->tmp.x25519, and set that private_key pointer to
S3I(s)->tmp.x25519 after all the "goto err;".

Also added freezero for S3I(s)->tmp.x25519 to ssl3_free() and ssl3_clear().

ok jsing@ tb@

Factor out/rewrite the ECDHE EC point key exchange code.

This reduces replication between the existing TLS client/server and allows
the code to soon be reused for TLSv1.3.

With feedback from inoguchi@ and tb@

ok inoguchi@ tb@

Remove the ssl_get_message function pointer from SSL_METHOD_INTERNAL.

ssl_get_message is essentially a switch between ssl3_get_message and
dtls1_get_message, both only used by the legacy stack. Instead, use
SSL_IS_DTLS() in ssl3_get_message to call the DTLS function when
necessary.

ok beck@ inoguchi@ tb@

If we are building a legacy server hello, check to see if we are
downgrading from TLS 1.3. If we are, set the last 8 bytes of the
server_random value to the required values as per RFC 8446 section
4.1.3 indicating that we deliberately meant to downgrade.

ok jsing@

Pass the session ID down to the session/ticket handling code as a CBS.

Convert ssl_get_prev_session(), tls1_process_ticket() and
tls1_decrypt_ticket() to handle the session ID from the client hello
as a CBS. While here also swap the order of arguments for
tls1_decrypt_ticket() so that it is consistent with the other functions.

ok tb@

Inline and remove the tlsext_tick_md macro.

There is not much point having a tlsext_tick_md macro that replaces
EVP_sha256() in two places, when the cipher is just hardcoded.

ok tb@

Defer sigalgs selection until the certificate is known.

Previously the signature algorithm was selected when the TLS extension was
parsed (or the client received a certificate request), however the actual
certificate to be used is not known at this stage. This leads to various
problems, including the selection of a signature algorithm that cannot be
used with the certificate key size (as found by jeremy@ via ruby regress).

Instead, store the signature algorithms list and only select a signature
algorithm when we're ready to do signature generation.

Joint work with beck@.

Rework ssl3_output_cert_chain() to take a CERT_PKEY and consider chains.

We will now include the certificates in the chain in the certificate list,
or use the existing extra_certs if present. Failing that we fall back to
the automatic chain building if not disabled.

This also simplifies the code significantly.

ok beck@ tb@

Rename tls1_handshake_hash*() to tls1_transcript_hash*().

While handshake hash is correct (in as far as it is a hash of handshake
messages), using tls1_transcript_hash*() aligns them with the naming of the
tls1_transcript*() functions. Additionally, the TLSv1.3 specification uses
Transcript-Hash and "transcript hash", which this matches.

ok inoguchi@ tb@

Rename TLS extension handling to use less "hello".

When the TLS extension code was rewritten, TLS extensions could only exist
in ClientHello and ServerHello messages - as such, they were named in pairs
of *_clienthello_{needs,build} which would be called by the client and
*_clienthello_parse. Likewise for *_serverhello_{needs,build} which would
be called by a server and *_serverhello_parse, which would be called by a
client.

Enter TLSv1.3 - TLS extensions can now exist in one of seven messages,
with only certain types being allowed to appear in each, meaning the naming
scheme no longer works. Instead, rename them to indicate the caller rather
than the message type - this effectively means:

clienthello_needs -> client_needs
clienthello_build -> client_build
clienthello_parse -> server_parse
serverhello_needs -> server_needs
serverhello_build -> server_build
serverhello_parse -> client_parse

ok beck@ tb@

Remove an ugly hack in the client certificate verification code that works
around broken GOST implementations. It looks like client certificates with
GOST have been completely broken since reimport of the GOST code, so no-one
is using LibreSSL this way. The client side was fixed only last week for
TLSv1.0 and TLSv1.1. This workaround is now in the way of much needed
simplifcation and cleanup, so it is time for it to go.

suggested by and ok jsing

Fix DTLS transcript handling for HelloVerifyRequest.

If DTLS sees a HelloVerifyRequest the transcript is reset - the previous
tls1_init_finished_mac() function could be called multiple times and would
discard any existing state. The replacement tls1_transcript_init() is more
strict and fails if a transcript already exists.

Provide an explicit tls1_transcript_reset() function and call it from the
appropriate places. This also lets us make DTLS less of a special snowflake
and call tls1_transcript_init() in the same place as used for TLS.

ok beck@ tb@

Add check function to verify that pkey is usable with a sigalg.
Include check for appropriate RSA key size when used with PSS.
ok tb@

quiet warning on other compilers

ok beck@

Free the server tls transcript in case session reuse did not work.
Regression found by Perl module p5-IO-Socket-SSL tests.
with beck@ tb@

Add support for RSA PSS algorithims being used in sigalgs.
lightly tested, but will need sanity checks and regress test changes
before being added to any sigalgs list for real
ok jsing@ tb@

Convert signatures and verifcation to use the EVP_DigestXXX api
to allow for adding PSS, Nuke the now unneejded guard around the PSS
algorithms in the sigalgs table
ok jsing@ tb@

Stop keeping track of sigalgs by guessing it from digest and pkey,
just keep the sigalg around so we can remember what we actually
decided to use.
ok jsing@

Ensure we only choose sigalgs from our prefernce list, not the whole list
ok jsing@

Add the ability to have a separate priority list for sigalgs.
Add a priority list for tls 1.2
ok jsing@

Reimplement the sigalgs processing code into a new implementation
that will be usable with TLS 1.3 with less eye bleed.
ok jsing@ tb@

Clean up and simplify the handshake transcript code.

This provides a cleaner, simpler and more readable API, with code that uses
a BUF_MEM instead of a BIO.

ok beck@ ("hurry up") and tb@.

Stop pretending that a cert member in a SSL and SSL_CTX can be NULL.

ok beck@ tb@

Ensure the handshake transcript is cleaned up.

Add a check at the completion of the client/server handshake to ensure that
the handshake transcript has been freed. Fix the case where a server asks
the client for a certificate, but it does not have one, resulting in the
handshake transcript being left around post-handshake.

ok bcook@ tb@

Convert ssl3_get_cert_verify() to CBS and clean up somewhat.

ok inoguchi@

Add some missing statics.

Simplify new session ticket encoding/generation.

The original code did a crazy encode/malloc/encode/decode/modify/encode
dance, in order to encode a session in the form needed to encrypt then add
to a session ticket. By modifying the encoding functions slightly, we can
do this entire dance as a single encode.

Inspired by similar changes in BoringSSL.

ok inoguchi@ tb@

Simplify session ticket parsing/handling.

The original implementation is rather crazy and means that we effectively
have two lots of code that parse a ClientHello and two lots of code that
parse TLS extensions. Partially simplify this by passing a CBS containing
the extension block through to the session handling functions, removing the
need to reimplement the ClientHello parsing.

While here standarise on naming for session_id and session_id_len.

ok inoguchi@ tb@

Pull up the parsing of a ClientHello.

Parse up until the extensions (if any), then proceed with processing,
rather than gradually parsing while processing. This makes the code
cleaner, requires messages to be valid before processing and makes way
for upcoming changes.

ok inoguchi@ tb@

Clean up handshake message start/finish functions.

Now that all handshake messages are created using CBB, remove the non-CBB
ssl3_handshake_msg_start()/ssl3_handshake_msg_finish() functions. Rename
the CBB variants by dropping the _cbb suffix.

ok bcook@ inoguchi@ tb@

Correct session ticket encryption.

The CBB conversion resulted in the ticket encryption being handled
incorrectly, resulting in only the last block being used. Fix this and
restore the previous behaviour.

Issue found by inoguchi@ and sebastia@.

ok inoguchi@ and tb@

Rename and collapse tls12_get_sigandhash_cbb().

Now that all callers of tls12_get_sigandhash() have been converted to CBB,
collapse tls12_get_sigandhash() and tls12_get_sigandhash_cbb() into a
single function. Rename it to tls12_gethashandsig() to be representative
of the actual order of the sigalgs parameters, and perform some other
clean up.

ok inoguchi@ tb@

Convert ssl3_send_newsession_ticket() to CBB.

This removes a memorable BUF_MEM_grow() and associated comment.

ok inoguchi@ tb@

Simplify the add signature code/logic in ssl3_send_server_key_exchange().

ok tb@

Convert ssl3_send_server_key_exchange() to CBB.

ok inoguchi@ tb@

Actually check the return values for EVP_Sign* and EVP_Verify*.

ok bcook@ beck@ tb@

Simplify server kex exchange signature generation.

Everything can go through the single EVP_Sign* code path.

ok inoguchi@ tb@

Now that all of the server-side client key exchange processing functions
have been converted to CBS, pull it up a level.

ok inoguchi@ tb@

Allocate a dedicated buffer for use when deriving a shared key during
client KEX DHE processing, rather than reusing the buffer that is used
to send/receive handshake messages.

ok beck@ inoguchi@

Convert ssl3_get_client_kex_ecdhe_ecp() to CBS.

Also allocate a dedicated buffer to hold the shared secret, rather than
reusing init_buf.

ok inoguchi@ tb@

Clean out a pile of cruft from ssl3_get_client_kex_ecdhe_ecp().

For pure ECDHE we do not need to construct a new key using the one that
was set up during the other half of the key exchange. Also, since we do not
support any form of ECDH the n == 0 case is not valid (per RFC 4492 section
5.7), so we can ditch this entirely.

ok inoguchi@ tb@

Convert ssl3_get_client_kex_gost() to CBS.

ok beck@ tb@

More clean up of the RSA key exchange code.

Convert to CBS, use more appropriate variable names and improve validation.
Allocate a dedicated buffer to hold the decrypted result, rather than
decrypting into the handshake buffer (which is also used to send data).

ok beck@ inoguchi@ tb@

Nuke SSL_OP_TLS_ROLLBACK_BUG - this is a workaround for buggy clients from
around the SSLv3/TLSv1.0 period... and buggy clients are buggy. This also
helps to clean up the RSA key exchange code.

ok "kill it with fire" beck@ tb@

Initialize variables to avoid compiler warnings

ok jsing@

Complete the TLS extension handling rewrite for the server-side.

This removes ssl_parse_clienthello_tlsext() and allows the CBS to be
passed all the way through from ssl3_get_client_hello(). The renegotation
check gets pulled up into ssl3_get_client_hello() which is where other
such checks exist.

The TLS extension parsing now also ensures that we do not get duplicates
of any known extensions (the old pre-rewrite code only did this for some
extensions).

ok inoguchi@

Fold dtls1_accept() into ssl_accept(), removing a lot of duplicated code.

With review/feedback from inoguchi@

Fully convert ssl3_send_server_hello() to CBB.

Based on a diff from doug@

Make ssl_bytes_to_cipher_list() take a CBS, rather than a pointer and
length, since the caller has already been converted to CBS. A small amount
of additional clean up whilst here.

Fix some style/whitespace/indentation issues in ssl3_accept().

Convert TLS signature algorithms extension handling to the new framework.

ok beck@ doug@

Remove NPN support.

NPN was never standardised and the last draft expired in October 2012.
ALPN was standardised in July 2014 and has been supported in LibreSSL
since December 2014. NPN has also been removed from Chromium in May 2016.

TLS clients and servers that try to use/enable NPN will fail gracefully and
fallback to the default protocol, since it will essentially appear that the
otherside does not support NPN. At some point in the future we will
actually remove the NPN related symbols entirely.

ok bcook@ beck@ doug@

Remove support for DSS/DSA, since we removed the cipher suites a while
back.

ok guenther@

Convert ssl3_send_certificate_request() to CBB.

ok beck@ doug@

Clean up the EC key/curve configuration handling.

Over the years OpenSSL grew multiple ways of being able to specify EC keys
(and/or curves) for use with ECDH and ECDHE key exchange. You could specify
a static EC key (SSL{_CTX,}_set_tmp_ecdh()), use that as a curve and
generate ephemeral keys (SSL_OP_SINGLE_ECDH_USE), provide the EC key via
a callback that was provided with insufficient information
(SSL{_CTX,}_set_tmp_ecdh_cb()) or enable automatic selection and generation
of EC keys via SSL{_CTX,}_set_ecdh_auto(). This complexity leads to
problems (like ECDHE not being enabled) and potential weird configuration
(like being able to do ECDHE without the ephemeral part...).

We no longer support ECDH and ECDHE can be disabled by removing ECDHE
ciphers from the cipher list. As such, permanently enable automatic EC
curve selection and generation, effectively disabling all of the
configuration knobs. The only exception is the
SSL{_CTX,}_set_tmp_ecdh() functions, which retain part of their previous
behaviour by configuring the curve of the given EC key as the only curve
being enabled. Everything else becomes a no-op.

ok beck@ doug@

Move state from ssl->internal to the handshake structure.
while we are at it, convert SSLerror to use a function
internally, so that we may later allocate the handshake
structure and check for it
ok jsing@

Bring in an SSL_HANDSHAKE structure and commence the great shovelling
ok jsing@, gcc@, regress@

Fix a bug caused by the return value being set early to signal successful
DTLS cookie validation. This can mask a later failure and result in a
positive return value being returned from ssl3_get_client_hello(), when
it should return a negative value to propagate the error.

Ironically this was introduced in OpenSSL 2e9802b7a7b with the commit
message "Fix DTLS cookie management bugs".

Fix based on OpenSSL.

Issue reported by Nicolas Bouliane <nbouliane at jive dot com>.

ok beck@

Use freezero(3) when cleaning up session tickets - not only does it require
less code, but there is also a potential performance gain since they can be
larger allocations.

Use freezero(3) to clean up the X25519 keys - simpler, cleaner code.

Clean up server key exchange EC point handling. Encode the point directly
into the CBB memory, rather than mallocing and memcpying, which also makes
makes the code more consistent with the client. Add a missing check for the
first EC_POINT_point2oct() call.

ok beck@

branches: 1.11.4;
Remove the handshake digests and related code, replacing remaining uses
with the handshake hash. For now tls1_digest_cached_records() is retained
to release the handshake buffer.

ok beck@ inoguchi@

Provide a rolling handshake hash that commences as soon as the cipher
suite has been selected, and convert the final finish MAC to use this
handshake hash.

This is a first step towards cleaning up the current handshake
buffer/digest code.

ok beck@ inoguchi@

Convert various handshake message generation functions to CBB.

ok beck@ inoguchi@

Convert ssl3_{get,send}_server_key_exchange() to EVP_md5_sha1().

ok inoguchi@

Stop pretending that MD5 and SHA1 might not exist - rather than locating
"ssl3-md5" and "ssl-sha1", call the EVP_md5() and EVP_sha1() functions
directly.

ok beck@ inoguchi@

Change SSLerror() back to taking two args, with the first one being an SSL *.
Make a table of "function codes" which maps the internal state of the SSL *
to something like a useful name so in a typical error in the connection you
know in what sort of place in the handshake things happened. (instead of
by arcane function name).
Add SSLerrorx() for when we don't have an SSL *
ok jsing@ after us both being prodded by bluhm@ to make it not terrible

Convert ssl3_get_client_hello() to CBS.

ok beck@

Finish the fallout of the SSLerr->SSLerror cleanup to get rid of the ugly
line wraps that resulted

Send the error function codes to rot in the depths of hell where they belong
We leave a single funciton code (0xFFF) to say "SSL_internal" so the public
API will not break, and we replace all internal use of the two argument
SSL_err() with the internal only SSL_error() that only takes a reason code.
ok jsing@

Remove most of SSL3_ENC_METHOD - we can just inline the function calls
and defines since they are the same everywhere.

ok beck@

Rename s3_{both,clnt,pkt_srvr}.c to have an ssl_ prefix since they are no
longer SSLv3 code.

ok beck@

Neuter the SSL_set_debug(3) API

The TLSv1.3 stack didn't support this in the first place, and in the legacy
stack it only added some dubious BIO_flush(3) calls. The sleep call between
SSL_read(3) and SSL_write(3) advertised in the comment next to the flag has
been a sleep call in the s_server since time immemorial, nota bene between
calls to BIO_gets(3). Anyway. This can all go and what remains will go with
the next major bump.

ok jsing

Check for negative EVP_CIPHER_CTX_iv_length() return in libssl

ok beck

Hide all public symbols in libssl

With the guentherizer 9000

ok tb@

Convert legacy server kex to one-shot sign/verify

This converts ssl3_{get,send}_server_key_exchange() to EVP_DigestVerify()
and EVP_DigestSign(). In order to do this, build the full signed_params
up front and rework the way the key exchange parameters are constructed.
This way we can do the verify and sign steps in one go and at the same
use a more idiomatic approach with CBB/CBS.

with/ok jsing

Easy EVP_Digest{Sign,Verify} conversions for legacy stack

Convert ssl3_send_client_verify_{sigalgs,gost}() to EVP_DigestSign() and
ssl3_get_cert_verify() to EVP_DigestVerify().

ok jsing

spelling fixes; from paul tagliamonte
ok tb

Make internal header file names consistent

Libcrypto currently has a mess of *_lcl.h, *_locl.h, and *_local.h names
used for internal headers. Move all these headers we inherited from
OpenSSL to *_local.h, reserving the name *_internal.h for our own code.
Similarly, move dtls_locl.h and ssl_locl.h to dtls_local and ssl_local.h.
constant_time_locl.h is moved to constant_time.h since it's special.

Adjust all .c files in libcrypto, libssl and regress.

The diff is mechanical with the exception of tls13_quic.c, where
#include <ssl_locl.h> was fixed manually.

discussed with jsing,
no objection bcook

Get rid of SSL_CTX_INTERNAL and SSL_INTERNAL.

These are no longer necessary due to SSL_CTX and SSL now being fully
opaque. Merge SSL_CTX_INTERNAL back into SSL_CTX and SSL_INTERNAL back
into SSL.

Prompted by tb@

Move handshake message handling functions from ssl_both.c to client/server.

Currently, ssl_both.c contains several functions that are used by both the
legacy client and legacy server. This interwines the client and server,
making it harder to make progressive changes. While it does deduplicate
some code, it also ends up with code that is conditioned on s->server and
forces the caller to pass in SSL3_ST_* values.

Move these functions from ssl_both.c into ssl_clnt.c and ssl_srvr.c,
renaming as appropriate and removing the s->server conditionals. Also move
the client and server function prototypes from ssl_locl.h into the .c
files, making them static in the process.

ok tb@

Deduplicate peer certificate chain processing code.

Rather than reimplement this in each TLS client and server, deduplicate it
into a single function. Furthermore, rather than dealing with the API
hazard that is SSL_get_peer_cert_chain() in this code, simply produce two
chains - one that has the leaf and one that does not.
SSL_get_peer_cert_chain() can then return the appropriate one.

This also moves the peer cert chain from the SSL_SESSION to the
SSL_HANDSHAKE, which makes more sense since it is not available on
resumption.

ok tb@

Simplify certificate list handling code in legacy server.

A client is required to send an empty list if it does not have a suitable
certificate - handle this case up front, rather than going through the
normal code path and ending up with an empty certificate list. This matches
what we do in the TLSv1.3 stack and will allow for ruther clean up (in
addition to making the code more readable).

Also tidy up the CBS code and remove some unnecessary length checks. Use
'cert' and 'certs' for certificates, rather than 'x' and 'sk'.

ok tb@

Rename uses 'curve' to 'group' and rework tls1 group API.

This reworks various tls1_ curve APIs to indicate success via a boolean
return value and move the output to an out parameter. This makes the
caller code easier and more consistent.

Based on a suggestion by jsing

ok jsing

Add checks to ensure we do not initiate or negotiate handshakes with
versions below the minimum required by the security level.

input & ok jsing

Check the security of DH key shares

ok beck, looks good to jsing

Check the security level when building sigalgs

ok beck jsing

Free ciphers before assigning to them

While this is not a leak currently, it definitely looks like one.
Pointed out by jsing on review of a diff that touched the vicinity
a while ago.

ok jsing

Add error checking to tls_session_secret_cb() calls

Failure of this undocumented callback was previously silently ignored.
Follow OpenSSL's behavior and throw an internal error (for lack of a
better choice) if the callback failed or if it set the master_key_length
to a negative number.

Unindent the success path and clean up some strange idioms.

ok jsing

Bye bye S3I.

S3I has served us well, however now that libssl is fully opaque it is time
to say goodbye. Aside from removing the calloc/free/memset, the rest is
mechanical sed.

ok inoguchi@ tb@

Remove peer_pkeys from SSL_SESSION.

peer_pkeys comes from some world where peers can send multiple certificates
- in fact, one of each known type. Since we do not live in such a world,
get rid of peer_pkeys and simply use peer_cert instead (in both TLSv1.2
and TLSv1.3, both clients and servers can only send a single leaf
(aka end-entity) certificate).

ok inoguchi@ tb@

Rename 'peer' to 'peer_cert' in SSL_SESSION.

The 'peer' member of SSL_SESSION is the leaf/end-entity certificate
provided by our peer. Rename it since 'peer' on its own is unhelpful.

ok inoguchi@ tb@

Plumb decode errors through key share parsing code.

Distinguish between decode errors and other errors, so that we can send
a SSL_AD_DECODE_ERROR alert when appropriate.

Fixes a tlsfuzzer failure, due to it expecting a decode error alert and
not receiving one.

Prompted by anton@

ok tb@

Clean up ssl3_{send,get}_client_kex_gost()

Fix leaks, use sizeof() instead of hardcoded sizes, actually check return
codes, explicit_bzero() the premaster secret on the server side and
generally try to kick the GOST kex code into some sort of shape.

ok inoguchi@ tb@

Return 0/1 from ssl3_{send,get}_client_kex_gost()

Like other KEX handling functions, there is no need to return anything
other than failure/success here.

ok inoguchi@ tb@

Fix GOST skip certificate verify handling.

GOST skip certificate verify handling got broken in r1.132 of s3_srvr.c
circa 2016. Prior to this, ssl3_get_client_key_exchange() returned an
'extra special' value to indicate that the state machine should skip
certificate verify. Fix this by setting and checking the
TLS1_FLAGS_SKIP_CERT_VERIFY flag, which is the same as is done in the
client.

ok inoguchi@ tb@

Merge SESS_CERT into SSL_SESSION.

There is no reason for SESS_CERT to exist - remove it and merge its members
into SSL_SESSION for the time being. More clean up to follow.

ok inoguchi@ tb@

Rename CERT to SSL_CERT and CERT_PKEY to SSL_CERT_PKEY.

Nearly all structs in libssl start with an SSL_ suffix, rename CERT and
CERT_PKEY for consistency.

ok inoguchi@ tb@

Rename dh_tmp to dhe_params.

Support for non-ephemeral DH was removed a long time ago - as such, the
dh_tmp and dh_tmp_cb are used for DHE parameters. Rename them to reflect
reality.

ok inoguchi@ tb@

Convert legacy server to tls_key_share.

This requires a few more additions to the DHE key share code - we need to
be able to either set the DHE parameters or specify the number of key bits
for use with auto DHE parameters. Additionally, we need to be able to
serialise the DHE parameters to send to the client.

This removes the infamous 'tmp' struct from ssl3_state_internal_st.

ok inoguchi@ tb@

Return 0 on failure from send/get kex functions in the legacy stack.

In the legacy stack, a message handling function returns -1 for failure,
0 for need more data and 1 for success (although in extra special cases
2 may also be used). However, the various send/get kex functions only
need to indicate success or failure - switch these to return 0 on failure
(rather than -1) and use normal result testing.

This leaves GOST unchanged for now, as that code is special and needs
extra work.

ok inoguchi@ tb@

Hoist memset of CBB above EVP_MD_CTX_new() and HMAC_CTX_new() to avoid
a use of uninitialized in the unlikely event that either of them fails.
Problem introduced in r1.128.

CID 345113

ok jsing

Convert ssl_srvr.c to opaque EVP_MD_CTX.

ok inoguchi jsing

Clean up and refactor server side DHE key exchange.

Provide ssl_kex_generate_dhe_params_auto() which handles DHE key generation
based on parameters determined by the specified key bits. Convert the
existing DHE auto parameter selection code into a function that just tells
us how many key bits to use.

Untangle and rework the server side DHE key exchange to use the ssl_kex_*
functions.

ok inoguchi@ tb@

Convert server serialisation of DHE parameters/public key to new functions.

ok inoguchi@ tb@

Stop reaching into EVP_PKEY in the rest of libssl.

ok inoguchi jsing

libssl: don't reach for pkey->save_type.

For some strange historical reason ECDSA_sign() and ECDSA_verify}() have
a type argument that they ignore. For another strange historical reason,
the type passed to them from libssl is pkey->save_type, which is used to
avoid expensive engine lookups when setting the pkey type... Whatever the
aforementioned reasons were, we can't access pkey->save_type with the
OpenSSL 1.1 API, and this is thus in the way of making EVP_PKEY opaque.
Simply pass in 0 instead.

ok jsing

Fold SSL_SESSION_INTERNAL back into SSL_SESSION.

ok beck@ tb@

Provide a way to determine our maximum legacy version.

With the introduction of TLSv1.3, we need the ability to determine our
maximum legacy version and to track our peer's maximum legacy version.
This is needed for both the TLS record layer when using TLSv1.3, plus
it is needed for RSA key exhange in TLS prior to TLSv1.3, where the
maximum legacy version is incorporated in the pre-master secret to
avoid downgrade attacks.

This unbreaks RSA KEX for the TLS client when the non-version specific
method is used with TLSv1.0 or TLSv1.1 (clearly no one does this).

ok tb@

Fold DTLS1_STATE_INTERNAL into DTLS1_STATE.

Now that DTLS1_STATE is opaque, fold DTLS1_STATE_INTERNAL back into
DTLS1_STATE and remove D1I() usage.

ok tb@

Untangle ssl3_get_message() return values.

This function currently has a long return type that may be <= 0 on
error/retry (which is then cast to an int in order to return it up the
stack), or it returns the length of the handshake message (on success).
This obviously means that 0 can be returned for both success and failure,
which is the reason why a separate 'ok' argument has to exist.

Untangle this mess by changing the return value to an int that indicates
success (1) or error/retry (<= 0). The length never needs to actually be
returned as it is already stored in s->internal->init_num (which is where
the return value is read from anyway).

ok tb@

Ensure that a client hello does not have trailing data.

Found by tlsfuzzer.

ok beck@

Clean up and simplify info and msg callbacks.

The info and msg callbacks result in duplication - both for code that
refers to the function pointers and for the call sites. Avoid this by
providing typedefs for the function pointers and pulling the calling
sequences into their own functions.

ok inoguchi@ tb@

Track sigalg used by ourselves and our peer in the legacy stack.

This is needed for upcoming API additions.

Convert legacy stack server to ssl_sigalg_for_peer().

ok inoguchi@ tb@

Move the RSA-PSS check for TLSv1.3 to ssl_sigalg_pkey_ok().

Also, rather than passing in a check_curve flag, pass in the SSL * and
handle version checks internally to ssl_sigalg_pkey_ok(), simplifying
the callers.

ok inoguchi@ tb@

Change ssl_sigalgs_from_value() to perform sigalg list selection.

Rather that passing in a sigalg list at every call site, pass in the
appropriate TLS version and have ssl_sigalgs_from_value() perform the
sigalg list selection itself. This allows the sigalg lists to be made
internal to the sigalgs code.

ok tb@

Rename ssl_sigalg() to ssl_sigalg_from_value().

This makes the code more self-documenting and avoids the ambiguity between
ssl_sigalg the struct and ssl_sigalg the function.

ok tb@

Change ssl_sigalgs_build() to perform sigalg list selection.

Rather that doing sigalg list selection at every call site, pass in the
appropriate TLS version and have ssl_sigalgs_build() perform the sigalg
list selection itself. This reduces code duplication, simplifies the
calling code and is the first step towards internalising the sigalg lists.

ok tb@

Make local header inclusion consistent.

Consistently include local headers in the same location, using the same
grouping/sorting across all files.

Move DTLS structs/definitions/prototypes to dtls_locl.h.

Now that the DTLS structs are opaque, add a dtls_locl.h header and move
internal-only structs from dtls1.h, along with prototypes from ssl_locl.h.
Only pull this header in where DTLS code actually exists.

ok inoguchi@ tb@

Explicitly include <openssl/opensslconf.h> in files using OPENSSL_NO_*

Where a file references to OPENSSL_NO_* conditions, ensure that we
explicitly include <openssl/opensslconf.h> before any references, rather
than relying on another header to pull this in.

Clean up tls1_change_cipher_state().

Replace flag gymnastics at call sites with separate read and write,
functions which call the common code. Condition on s->server instead of
using SSL_ST_ACCEPT, for consistency and more readable code.

ok inoguchi@ tb@

In the TLSv1.2 server, set up the key block after sending the CCS.

This avoids calling into the key block setup code multiple times and makes
the server code consistent with the client.

ok inoguchi@ tb@

Clean up dtls1_reset_seq_numbers().

Rather than doing flag gymnastics, split dtls1_reset_seq_numbers() into
separate read and write functions. Move the calls of these functions into
tls1_change_cipher_state() so they directly follow the change of cipher
state in the record layer, which avoids having to duplicate the calls in
the client and server.

ok inoguchi@ tb@

Clean up and harden TLSv1.2 master key derivation.

The master key and its length are only stored in one location, so it makes
no sense to handle these outside of the derivation function (the current
'out' argument is unused). This simplifies the various call sites.

If derivation fails for some reason, fail hard rather than continuing on
and hoping that something deals with this correctly later.

ok inoguchi@ tb@

Clean up derivation of finished/peer finished.

Make this process more readable by having specific client/server functions,
calling the correct one based on s->server. This allows to remove various
SSL_ST_ACCEPT/SSL_ST_CONNECT checks, along with duplicate code.

ok inoguchi@ tb@

Clean up TLSv1.2 certificate request handshake data.

Currently cert_req is used by clients and cert_request is used by servers.
Replace this by a single cert_request used by either client or server.
Remove the certificate types as they are currently unused. This also fixes
a bug whereby if the number of certificate types exceeds SSL3_CT_NUMBER
the number of bytes read in is insufficient, which will break decoding.

ok inoguchi@ tb@

Move reuse_message, message_type, message_size and cert_verify into the
TLSv1.2 handshake struct.

ok inoguchi@ tb@

Avoid transcript initialisation when sending a TLS HelloRequest.

When server side renegotiation is triggered, the TLSv1.2 state machine
sends a HelloRequest before going to ST_SW_FLUSH and ST_OK. In this case
we do not need the transcript and currently hit the sanity check in ST_OK
that ensures the transcript has been freed, breaking server initiated
renegotiation. We do however need the transcript in the DTLS case.

ok tb@

Garbage collect s->internal->type

This variable is used in the legacy stack to decide whether we are
a server or a client. That's what s->server is for...

The new TLSv1.3 stack failed to set s->internal->type, which resulted
in hilarious mishandling of previous_{client,server}_finished. Indeed,
both client and server would first store the client's verify_data in
previous_server_finished and later overwrite it with the server's
verify_data. Consequently, renegotiation has been completely broken
for more than a year. In fact, server side renegotiation was broken
during the 6.5 release cycle. Clearly, no-one uses this.

This commit fixes client side renegotiation and restores the previous
behavior of SSL_get_client_CA_list(). Server side renegotiation will
be fixed in a later commit.

ok jsing

Rename new_cipher to cipher.

This is in the SSL_HANDSHAKE struct and is what we're currently
negotiating, so there is really nothing more "new" about the cipher
than there is the key block or other parts of the handshake data.

ok inoguchi@ tb@

Add SSL_HANDSHAKE_TLS12 for TLSv1.2 specific handshake data.

Move TLSv1.2 specific components over from SSL_HANDSHAKE.

ok inoguchi@ tb@

Remove ssl_downgrade_max_version().

Now that we store our maximum TLS version at the start of the handshake,
we can check against that directly.

ok inoguchi@ tb@

Improve internal version handling.

Add handshake fields for our minimum TLS version, our maximum TLS version
and the TLS version negotiated during the handshake. Initialise our min/max
versions at the start of the handshake and leave these unchanged. The
negotiated TLS version is set in the client once we receive the ServerHello
and in the server at the point we select the highest shared version.

Provide an ssl_effective_version() function that returns the negotiated TLS
version if known, otherwise our maximum TLS version - this is effectively
what is stored in s->version currently.

Convert most of the internal code to use one of these three version fields,
which greatly simplifies code (especially in the TLS extension handling
code).

ok tb@

ugly whitespace

Rename f_err into fatal_err.

discussed with jsing

Rename the truncated label into decode_err. This describes its purpose
better and doesn't look odd if there's trailing data for exapmle.
Indent a few labels in the neighborhood while there.

ok jsing

Revise HelloVerifyRequest handling for DTLSv1.2.

Per RFC 6347 section 4.2.1, the HelloVerifyRequest should always contain
DTLSv1.0 - ensure this is the case on the server side, allow both DTLSv1.0
and DTLSv1.2 on the client.

ok tb@

Factor out the legacy stack version checks.

Also check for explicit version numbers, rather than just the major version
value.

ok tb@

Move sequence numbers into the new TLSv1.2 record layer.

This allows for all of the DTLS sequence number save/restore code to be
removed.

ok inoguchi@ "whee!" tb@

Provide functions to determine if TLSv1.2 record protection is engaged.

Call these functions from code that needs to know if we've changed cipher
state and enabled record protection, rather than inconsistently checking
various pointers from other places in the code base. This also fixes a
minor bug where the wrong pointers are checked if we're operating with
AEAD.

ok inoguchi@ tb@

Replace SSL_IS_DTLS with SSL_is_dtls().

Garbage collect the now unused SSL_IS_DTLS macro.

ok tb@

Condense and simplify TLS methods.

Historically, OpenSSL has had client and server specific methods - the only
difference between these is that the .ssl_connect or .ssl_accept function
pointer is set to ssl_undefined_function, with the intention of reducing
code size for a statically linked binary that was only a client or server.
These days the difference is minimal or non-existant in many cases and
we can reduce the amount of code and complexity by having single method.

Internally remove all of the client and server specific methods,
simplifying code in the process. The external client/server specific API
remain, however these now return the same thing as TLS_method() does.

ok tb@

Merge d1_{clnt,srvr}.c into ssl_{clnt,srvr}.c

The d1_{clnt,srvr}.c contain a single function each - merge these into the
ssl_{clnt,srvr}.c, renaming them with an ssl3_ prefix for consistency.

ok beck@ tb@

Simplify the cleanup of init_buf via a ssl3_release_init_buffer() function.

ok beck@ inoguchi@ tb@

Simplify SSL method lookups.

There are three places where we call tls1_get_{client,server}_method() and
if that returns NULL, call dtls1_get_{client,server}_method(). Simplify
this by combining the lookup into a single function. While here also use
uint16_t for version types.

ok inoguchi@ millert@

Use the correct type for tls1_set_ec_id()

The curve_id is a uint16, not an int.

ok beck jsing

Remove cipher_list_by_id.

When parsing a cipher string, a cipher list is created, before being
duplicated and sorted - the second copy being stored as cipher_list_by_id.
This is done only so that a client can ensure that the cipher selected by
a server is in the cipher list. This is pretty pointless given that most
clients are short-lived and that we already had to iterate over the cipher
list in order to build the client hello. Additionally, any update to the
cipher list requires that cipher_list_by_id also be updated and kept in
sync.

Remove all of this and replace it with a simple linear scan - the overhead
of duplicating and sorting the cipher list likely exceeds that of a simple
linear scan over the cipher list (64 maximum, more typically ~9 or so).

ok beck@ tb@

Send alert on ssl_get_prev_session failure

ssl_get_prev_session() can fail for various reasons some of which
may be internal_error others decode_error alerts. Propagate the
appropriate alert up to the caller so we can abort the handshake
by sending a fatal alert instead of rudely closing the pipe.

Currently only 28 of 292 test cases of tlsfuzzer's test-extension.py pass.
With this diff, 272 pass. The rest will require fixes elsewhere.

ok beck inoguchi jsing

Improve argument order for the internal tlsext API

Move is_server and msg_type right after the SSL object so that CBS
and CBB and alert come last. This brings these functions more in
line with other internal functions and separates state from data.

requested by jsing

Enable GOST_SIG_FORMAT_RS_LE when verifying certificate signatures.

GOST cipher suites requires that CertVerify signatures be generated in a
special way (see ssl3_send_client_kex_gost(), ssl3_get_cert_verify()).
However, the GOST_SIG_FORMAT_RS_LE flag was not passed in case of TLS 1.2
connections (because they use different code path). Set this flag on
GOST PKEYs.

Diff from Dmitry Baryshkov <dbaryshkov@gmail.com>

Sponsored by ROSA Linux

ok inoguchi@ tb@

Send an illegal_parameter alert if a client sends us invalid DH key
shares. Previously we would fail and just close the pipe.

Fixes the remaining failing test-dhe-rsa-key-exchange-with-bad-messages.py
tests of tlsfuzzer.

ok beck (earlier version) jsing

Correct downgrade sentinels when a version pinned method is in use.

Previously only the enabled protocol versions were considered, however we
also have to consider the method in use which may be version pinned.

Found the hard way by danj@ with haproxy and force-tlsv12.

ok beck@ inoguchi@ tb@

Replace SSL_PKEY_RSA_ENC/SSL_PKEY_RSA_SIGN with SSL_PKEY_RSA.

Some time prior to SSLeay 0.8.1b, SSL_PKEY_RSA_SIGN got added with the
intention of handling RSA sign only certificates... this incomplete code
had the following comment:

/* check to see if this is a signing only certificate */
/* EAY EAY EAY EAY */

And while the comment was removed in 2005, the incomplete RSA sign-only
handling has remained ever since.

Remove SSL_PKEY_RSA_SIGN and rename SSL_PKEY_RSA_ENC to SSL_PKEY_RSA. While
here also remove the unused SSL_PKEY_DH_RSA.

ok tb@

Use size_t for OCSP response length.

The OCSP response length is currently an integer, which is overloaded with
-1 meaning "unset". Use a size_t for the OCSP response length and infer
unset from the OCSP response being NULL. This makes code more readable,
simpler and less error prone.

ok beck@

On receiving an overlong session ID terminate with an illegal_parameter
alert. Found with tlsfuzzer.

ok jsing

TLSv1.3 servers that intend to downgrade are required to set the last
eight bytes of the server's random to a magic cookie (RFC 8446, 4.1.3).
The TLSv1.3 spec changes the TLSv1.2 spec in that it recommends that
TLSv1.2 servers that negotiate TLSv1.1 or below do the same. This gives
a limited additional protection against downgrade attacks beyond what is
already present in the Finished exchange.

The TLSv1.3 part was already implemented in Hobart and can be trivially
modified to do the TLSv1.2 bit as well.

ok inoguchi, jsing

Avoid leak for tmp.x25519

Changed to use local variable to hold malloc address rather than directly
set to S3I(s)->tmp.x25519, and set that private_key pointer to
S3I(s)->tmp.x25519 after all the "goto err;".

Also added freezero for S3I(s)->tmp.x25519 to ssl3_free() and ssl3_clear().

ok jsing@ tb@

Factor out/rewrite the ECDHE EC point key exchange code.

This reduces replication between the existing TLS client/server and allows
the code to soon be reused for TLSv1.3.

With feedback from inoguchi@ and tb@

ok inoguchi@ tb@

Remove the ssl_get_message function pointer from SSL_METHOD_INTERNAL.

ssl_get_message is essentially a switch between ssl3_get_message and
dtls1_get_message, both only used by the legacy stack. Instead, use
SSL_IS_DTLS() in ssl3_get_message to call the DTLS function when
necessary.

ok beck@ inoguchi@ tb@

If we are building a legacy server hello, check to see if we are
downgrading from TLS 1.3. If we are, set the last 8 bytes of the
server_random value to the required values as per RFC 8446 section
4.1.3 indicating that we deliberately meant to downgrade.

ok jsing@

Pass the session ID down to the session/ticket handling code as a CBS.

Convert ssl_get_prev_session(), tls1_process_ticket() and
tls1_decrypt_ticket() to handle the session ID from the client hello
as a CBS. While here also swap the order of arguments for
tls1_decrypt_ticket() so that it is consistent with the other functions.

ok tb@

Inline and remove the tlsext_tick_md macro.

There is not much point having a tlsext_tick_md macro that replaces
EVP_sha256() in two places, when the cipher is just hardcoded.

ok tb@

Defer sigalgs selection until the certificate is known.

Previously the signature algorithm was selected when the TLS extension was
parsed (or the client received a certificate request), however the actual
certificate to be used is not known at this stage. This leads to various
problems, including the selection of a signature algorithm that cannot be
used with the certificate key size (as found by jeremy@ via ruby regress).

Instead, store the signature algorithms list and only select a signature
algorithm when we're ready to do signature generation.

Joint work with beck@.

Rework ssl3_output_cert_chain() to take a CERT_PKEY and consider chains.

We will now include the certificates in the chain in the certificate list,
or use the existing extra_certs if present. Failing that we fall back to
the automatic chain building if not disabled.

This also simplifies the code significantly.

ok beck@ tb@

Rename tls1_handshake_hash*() to tls1_transcript_hash*().

While handshake hash is correct (in as far as it is a hash of handshake
messages), using tls1_transcript_hash*() aligns them with the naming of the
tls1_transcript*() functions. Additionally, the TLSv1.3 specification uses
Transcript-Hash and "transcript hash", which this matches.

ok inoguchi@ tb@

Rename TLS extension handling to use less "hello".

When the TLS extension code was rewritten, TLS extensions could only exist
in ClientHello and ServerHello messages - as such, they were named in pairs
of *_clienthello_{needs,build} which would be called by the client and
*_clienthello_parse. Likewise for *_serverhello_{needs,build} which would
be called by a server and *_serverhello_parse, which would be called by a
client.

Enter TLSv1.3 - TLS extensions can now exist in one of seven messages,
with only certain types being allowed to appear in each, meaning the naming
scheme no longer works. Instead, rename them to indicate the caller rather
than the message type - this effectively means:

clienthello_needs -> client_needs
clienthello_build -> client_build
clienthello_parse -> server_parse
serverhello_needs -> server_needs
serverhello_build -> server_build
serverhello_parse -> client_parse

ok beck@ tb@

Remove an ugly hack in the client certificate verification code that works
around broken GOST implementations. It looks like client certificates with
GOST have been completely broken since reimport of the GOST code, so no-one
is using LibreSSL this way. The client side was fixed only last week for
TLSv1.0 and TLSv1.1. This workaround is now in the way of much needed
simplifcation and cleanup, so it is time for it to go.

suggested by and ok jsing

Fix DTLS transcript handling for HelloVerifyRequest.

If DTLS sees a HelloVerifyRequest the transcript is reset - the previous
tls1_init_finished_mac() function could be called multiple times and would
discard any existing state. The replacement tls1_transcript_init() is more
strict and fails if a transcript already exists.

Provide an explicit tls1_transcript_reset() function and call it from the
appropriate places. This also lets us make DTLS less of a special snowflake
and call tls1_transcript_init() in the same place as used for TLS.

ok beck@ tb@

Add check function to verify that pkey is usable with a sigalg.
Include check for appropriate RSA key size when used with PSS.
ok tb@

quiet warning on other compilers

ok beck@

Free the server tls transcript in case session reuse did not work.
Regression found by Perl module p5-IO-Socket-SSL tests.
with beck@ tb@

Add support for RSA PSS algorithims being used in sigalgs.
lightly tested, but will need sanity checks and regress test changes
before being added to any sigalgs list for real
ok jsing@ tb@

Convert signatures and verifcation to use the EVP_DigestXXX api
to allow for adding PSS, Nuke the now unneejded guard around the PSS
algorithms in the sigalgs table
ok jsing@ tb@

Stop keeping track of sigalgs by guessing it from digest and pkey,
just keep the sigalg around so we can remember what we actually
decided to use.
ok jsing@

Ensure we only choose sigalgs from our prefernce list, not the whole list
ok jsing@

Add the ability to have a separate priority list for sigalgs.
Add a priority list for tls 1.2
ok jsing@

Reimplement the sigalgs processing code into a new implementation
that will be usable with TLS 1.3 with less eye bleed.
ok jsing@ tb@

Clean up and simplify the handshake transcript code.

This provides a cleaner, simpler and more readable API, with code that uses
a BUF_MEM instead of a BIO.

ok beck@ ("hurry up") and tb@.

Stop pretending that a cert member in a SSL and SSL_CTX can be NULL.

ok beck@ tb@

Ensure the handshake transcript is cleaned up.

Add a check at the completion of the client/server handshake to ensure that
the handshake transcript has been freed. Fix the case where a server asks
the client for a certificate, but it does not have one, resulting in the
handshake transcript being left around post-handshake.

ok bcook@ tb@

Convert ssl3_get_cert_verify() to CBS and clean up somewhat.

ok inoguchi@

Add some missing statics.

Simplify new session ticket encoding/generation.

The original code did a crazy encode/malloc/encode/decode/modify/encode
dance, in order to encode a session in the form needed to encrypt then add
to a session ticket. By modifying the encoding functions slightly, we can
do this entire dance as a single encode.

Inspired by similar changes in BoringSSL.

ok inoguchi@ tb@

Simplify session ticket parsing/handling.

The original implementation is rather crazy and means that we effectively
have two lots of code that parse a ClientHello and two lots of code that
parse TLS extensions. Partially simplify this by passing a CBS containing
the extension block through to the session handling functions, removing the
need to reimplement the ClientHello parsing.

While here standarise on naming for session_id and session_id_len.

ok inoguchi@ tb@

Pull up the parsing of a ClientHello.

Parse up until the extensions (if any), then proceed with processing,
rather than gradually parsing while processing. This makes the code
cleaner, requires messages to be valid before processing and makes way
for upcoming changes.

ok inoguchi@ tb@

Clean up handshake message start/finish functions.

Now that all handshake messages are created using CBB, remove the non-CBB
ssl3_handshake_msg_start()/ssl3_handshake_msg_finish() functions. Rename
the CBB variants by dropping the _cbb suffix.

ok bcook@ inoguchi@ tb@

Correct session ticket encryption.

The CBB conversion resulted in the ticket encryption being handled
incorrectly, resulting in only the last block being used. Fix this and
restore the previous behaviour.

Issue found by inoguchi@ and sebastia@.

ok inoguchi@ and tb@

Rename and collapse tls12_get_sigandhash_cbb().

Now that all callers of tls12_get_sigandhash() have been converted to CBB,
collapse tls12_get_sigandhash() and tls12_get_sigandhash_cbb() into a
single function. Rename it to tls12_gethashandsig() to be representative
of the actual order of the sigalgs parameters, and perform some other
clean up.

ok inoguchi@ tb@

Convert ssl3_send_newsession_ticket() to CBB.

This removes a memorable BUF_MEM_grow() and associated comment.

ok inoguchi@ tb@

Simplify the add signature code/logic in ssl3_send_server_key_exchange().

ok tb@

Convert ssl3_send_server_key_exchange() to CBB.

ok inoguchi@ tb@

Actually check the return values for EVP_Sign* and EVP_Verify*.

ok bcook@ beck@ tb@

Simplify server kex exchange signature generation.

Everything can go through the single EVP_Sign* code path.

ok inoguchi@ tb@

Now that all of the server-side client key exchange processing functions
have been converted to CBS, pull it up a level.

ok inoguchi@ tb@

Allocate a dedicated buffer for use when deriving a shared key during
client KEX DHE processing, rather than reusing the buffer that is used
to send/receive handshake messages.

ok beck@ inoguchi@

Convert ssl3_get_client_kex_ecdhe_ecp() to CBS.

Also allocate a dedicated buffer to hold the shared secret, rather than
reusing init_buf.

ok inoguchi@ tb@

Clean out a pile of cruft from ssl3_get_client_kex_ecdhe_ecp().

For pure ECDHE we do not need to construct a new key using the one that
was set up during the other half of the key exchange. Also, since we do not
support any form of ECDH the n == 0 case is not valid (per RFC 4492 section
5.7), so we can ditch this entirely.

ok inoguchi@ tb@

Convert ssl3_get_client_kex_gost() to CBS.

ok beck@ tb@

More clean up of the RSA key exchange code.

Convert to CBS, use more appropriate variable names and improve validation.
Allocate a dedicated buffer to hold the decrypted result, rather than
decrypting into the handshake buffer (which is also used to send data).

ok beck@ inoguchi@ tb@

Nuke SSL_OP_TLS_ROLLBACK_BUG - this is a workaround for buggy clients from
around the SSLv3/TLSv1.0 period... and buggy clients are buggy. This also
helps to clean up the RSA key exchange code.

ok "kill it with fire" beck@ tb@

Initialize variables to avoid compiler warnings

ok jsing@

Complete the TLS extension handling rewrite for the server-side.

This removes ssl_parse_clienthello_tlsext() and allows the CBS to be
passed all the way through from ssl3_get_client_hello(). The renegotation
check gets pulled up into ssl3_get_client_hello() which is where other
such checks exist.

The TLS extension parsing now also ensures that we do not get duplicates
of any known extensions (the old pre-rewrite code only did this for some
extensions).

ok inoguchi@

Fold dtls1_accept() into ssl_accept(), removing a lot of duplicated code.

With review/feedback from inoguchi@

Fully convert ssl3_send_server_hello() to CBB.

Based on a diff from doug@

Make ssl_bytes_to_cipher_list() take a CBS, rather than a pointer and
length, since the caller has already been converted to CBS. A small amount
of additional clean up whilst here.

Fix some style/whitespace/indentation issues in ssl3_accept().

Convert TLS signature algorithms extension handling to the new framework.

ok beck@ doug@

Remove NPN support.

NPN was never standardised and the last draft expired in October 2012.
ALPN was standardised in July 2014 and has been supported in LibreSSL
since December 2014. NPN has also been removed from Chromium in May 2016.

TLS clients and servers that try to use/enable NPN will fail gracefully and
fallback to the default protocol, since it will essentially appear that the
otherside does not support NPN. At some point in the future we will
actually remove the NPN related symbols entirely.

ok bcook@ beck@ doug@

Remove support for DSS/DSA, since we removed the cipher suites a while
back.

ok guenther@

Convert ssl3_send_certificate_request() to CBB.

ok beck@ doug@

Clean up the EC key/curve configuration handling.

Over the years OpenSSL grew multiple ways of being able to specify EC keys
(and/or curves) for use with ECDH and ECDHE key exchange. You could specify
a static EC key (SSL{_CTX,}_set_tmp_ecdh()), use that as a curve and
generate ephemeral keys (SSL_OP_SINGLE_ECDH_USE), provide the EC key via
a callback that was provided with insufficient information
(SSL{_CTX,}_set_tmp_ecdh_cb()) or enable automatic selection and generation
of EC keys via SSL{_CTX,}_set_ecdh_auto(). This complexity leads to
problems (like ECDHE not being enabled) and potential weird configuration
(like being able to do ECDHE without the ephemeral part...).

We no longer support ECDH and ECDHE can be disabled by removing ECDHE
ciphers from the cipher list. As such, permanently enable automatic EC
curve selection and generation, effectively disabling all of the
configuration knobs. The only exception is the
SSL{_CTX,}_set_tmp_ecdh() functions, which retain part of their previous
behaviour by configuring the curve of the given EC key as the only curve
being enabled. Everything else becomes a no-op.

ok beck@ doug@

Move state from ssl->internal to the handshake structure.
while we are at it, convert SSLerror to use a function
internally, so that we may later allocate the handshake
structure and check for it
ok jsing@

Bring in an SSL_HANDSHAKE structure and commence the great shovelling
ok jsing@, gcc@, regress@

Fix a bug caused by the return value being set early to signal successful
DTLS cookie validation. This can mask a later failure and result in a
positive return value being returned from ssl3_get_client_hello(), when
it should return a negative value to propagate the error.

Ironically this was introduced in OpenSSL 2e9802b7a7b with the commit
message "Fix DTLS cookie management bugs".

Fix based on OpenSSL.

Issue reported by Nicolas Bouliane <nbouliane at jive dot com>.

ok beck@

Use freezero(3) when cleaning up session tickets - not only does it require
less code, but there is also a potential performance gain since they can be
larger allocations.

Use freezero(3) to clean up the X25519 keys - simpler, cleaner code.

Clean up server key exchange EC point handling. Encode the point directly
into the CBB memory, rather than mallocing and memcpying, which also makes
makes the code more consistent with the client. Add a missing check for the
first EC_POINT_point2oct() call.

ok beck@

branches: 1.11.4;
Remove the handshake digests and related code, replacing remaining uses
with the handshake hash. For now tls1_digest_cached_records() is retained
to release the handshake buffer.

ok beck@ inoguchi@

Provide a rolling handshake hash that commences as soon as the cipher
suite has been selected, and convert the final finish MAC to use this
handshake hash.

This is a first step towards cleaning up the current handshake
buffer/digest code.

ok beck@ inoguchi@

Convert various handshake message generation functions to CBB.

ok beck@ inoguchi@

Convert ssl3_{get,send}_server_key_exchange() to EVP_md5_sha1().

ok inoguchi@

Stop pretending that MD5 and SHA1 might not exist - rather than locating
"ssl3-md5" and "ssl-sha1", call the EVP_md5() and EVP_sha1() functions
directly.

ok beck@ inoguchi@

Change SSLerror() back to taking two args, with the first one being an SSL *.
Make a table of "function codes" which maps the internal state of the SSL *
to something like a useful name so in a typical error in the connection you
know in what sort of place in the handshake things happened. (instead of
by arcane function name).
Add SSLerrorx() for when we don't have an SSL *
ok jsing@ after us both being prodded by bluhm@ to make it not terrible

Convert ssl3_get_client_hello() to CBS.

ok beck@

Finish the fallout of the SSLerr->SSLerror cleanup to get rid of the ugly
line wraps that resulted

Send the error function codes to rot in the depths of hell where they belong
We leave a single funciton code (0xFFF) to say "SSL_internal" so the public
API will not break, and we replace all internal use of the two argument
SSL_err() with the internal only SSL_error() that only takes a reason code.
ok jsing@

Remove most of SSL3_ENC_METHOD - we can just inline the function calls
and defines since they are the same everywhere.

ok beck@

Rename s3_{both,clnt,pkt_srvr}.c to have an ssl_ prefix since they are no
longer SSLv3 code.

ok beck@

Check for negative EVP_CIPHER_CTX_iv_length() return in libssl

ok beck

Hide all public symbols in libssl

With the guentherizer 9000

ok tb@

Convert legacy server kex to one-shot sign/verify

This converts ssl3_{get,send}_server_key_exchange() to EVP_DigestVerify()
and EVP_DigestSign(). In order to do this, build the full signed_params
up front and rework the way the key exchange parameters are constructed.
This way we can do the verify and sign steps in one go and at the same
use a more idiomatic approach with CBB/CBS.

with/ok jsing

Easy EVP_Digest{Sign,Verify} conversions for legacy stack

Convert ssl3_send_client_verify_{sigalgs,gost}() to EVP_DigestSign() and
ssl3_get_cert_verify() to EVP_DigestVerify().

ok jsing

spelling fixes; from paul tagliamonte
ok tb

Make internal header file names consistent

Libcrypto currently has a mess of *_lcl.h, *_locl.h, and *_local.h names
used for internal headers. Move all these headers we inherited from
OpenSSL to *_local.h, reserving the name *_internal.h for our own code.
Similarly, move dtls_locl.h and ssl_locl.h to dtls_local and ssl_local.h.
constant_time_locl.h is moved to constant_time.h since it's special.

Adjust all .c files in libcrypto, libssl and regress.

The diff is mechanical with the exception of tls13_quic.c, where
#include <ssl_locl.h> was fixed manually.

discussed with jsing,
no objection bcook

Get rid of SSL_CTX_INTERNAL and SSL_INTERNAL.

These are no longer necessary due to SSL_CTX and SSL now being fully
opaque. Merge SSL_CTX_INTERNAL back into SSL_CTX and SSL_INTERNAL back
into SSL.

Prompted by tb@

Move handshake message handling functions from ssl_both.c to client/server.

Currently, ssl_both.c contains several functions that are used by both the
legacy client and legacy server. This interwines the client and server,
making it harder to make progressive changes. While it does deduplicate
some code, it also ends up with code that is conditioned on s->server and
forces the caller to pass in SSL3_ST_* values.

Move these functions from ssl_both.c into ssl_clnt.c and ssl_srvr.c,
renaming as appropriate and removing the s->server conditionals. Also move
the client and server function prototypes from ssl_locl.h into the .c
files, making them static in the process.

ok tb@

Deduplicate peer certificate chain processing code.

Rather than reimplement this in each TLS client and server, deduplicate it
into a single function. Furthermore, rather than dealing with the API
hazard that is SSL_get_peer_cert_chain() in this code, simply produce two
chains - one that has the leaf and one that does not.
SSL_get_peer_cert_chain() can then return the appropriate one.

This also moves the peer cert chain from the SSL_SESSION to the
SSL_HANDSHAKE, which makes more sense since it is not available on
resumption.

ok tb@

Simplify certificate list handling code in legacy server.

A client is required to send an empty list if it does not have a suitable
certificate - handle this case up front, rather than going through the
normal code path and ending up with an empty certificate list. This matches
what we do in the TLSv1.3 stack and will allow for ruther clean up (in
addition to making the code more readable).

Also tidy up the CBS code and remove some unnecessary length checks. Use
'cert' and 'certs' for certificates, rather than 'x' and 'sk'.

ok tb@

Rename uses 'curve' to 'group' and rework tls1 group API.

This reworks various tls1_ curve APIs to indicate success via a boolean
return value and move the output to an out parameter. This makes the
caller code easier and more consistent.

Based on a suggestion by jsing

ok jsing

Add checks to ensure we do not initiate or negotiate handshakes with
versions below the minimum required by the security level.

input & ok jsing

Check the security of DH key shares

ok beck, looks good to jsing

Check the security level when building sigalgs

ok beck jsing

Free ciphers before assigning to them

While this is not a leak currently, it definitely looks like one.
Pointed out by jsing on review of a diff that touched the vicinity
a while ago.

ok jsing

Add error checking to tls_session_secret_cb() calls

Failure of this undocumented callback was previously silently ignored.
Follow OpenSSL's behavior and throw an internal error (for lack of a
better choice) if the callback failed or if it set the master_key_length
to a negative number.

Unindent the success path and clean up some strange idioms.

ok jsing

Bye bye S3I.

S3I has served us well, however now that libssl is fully opaque it is time
to say goodbye. Aside from removing the calloc/free/memset, the rest is
mechanical sed.

ok inoguchi@ tb@

Remove peer_pkeys from SSL_SESSION.

peer_pkeys comes from some world where peers can send multiple certificates
- in fact, one of each known type. Since we do not live in such a world,
get rid of peer_pkeys and simply use peer_cert instead (in both TLSv1.2
and TLSv1.3, both clients and servers can only send a single leaf
(aka end-entity) certificate).

ok inoguchi@ tb@

Rename 'peer' to 'peer_cert' in SSL_SESSION.

The 'peer' member of SSL_SESSION is the leaf/end-entity certificate
provided by our peer. Rename it since 'peer' on its own is unhelpful.

ok inoguchi@ tb@

Plumb decode errors through key share parsing code.

Distinguish between decode errors and other errors, so that we can send
a SSL_AD_DECODE_ERROR alert when appropriate.

Fixes a tlsfuzzer failure, due to it expecting a decode error alert and
not receiving one.

Prompted by anton@

ok tb@

Clean up ssl3_{send,get}_client_kex_gost()

Fix leaks, use sizeof() instead of hardcoded sizes, actually check return
codes, explicit_bzero() the premaster secret on the server side and
generally try to kick the GOST kex code into some sort of shape.

ok inoguchi@ tb@

Return 0/1 from ssl3_{send,get}_client_kex_gost()

Like other KEX handling functions, there is no need to return anything
other than failure/success here.

ok inoguchi@ tb@

Fix GOST skip certificate verify handling.

GOST skip certificate verify handling got broken in r1.132 of s3_srvr.c
circa 2016. Prior to this, ssl3_get_client_key_exchange() returned an
'extra special' value to indicate that the state machine should skip
certificate verify. Fix this by setting and checking the
TLS1_FLAGS_SKIP_CERT_VERIFY flag, which is the same as is done in the
client.

ok inoguchi@ tb@

Merge SESS_CERT into SSL_SESSION.

There is no reason for SESS_CERT to exist - remove it and merge its members
into SSL_SESSION for the time being. More clean up to follow.

ok inoguchi@ tb@

Rename CERT to SSL_CERT and CERT_PKEY to SSL_CERT_PKEY.

Nearly all structs in libssl start with an SSL_ suffix, rename CERT and
CERT_PKEY for consistency.

ok inoguchi@ tb@

Rename dh_tmp to dhe_params.

Support for non-ephemeral DH was removed a long time ago - as such, the
dh_tmp and dh_tmp_cb are used for DHE parameters. Rename them to reflect
reality.

ok inoguchi@ tb@

Convert legacy server to tls_key_share.

This requires a few more additions to the DHE key share code - we need to
be able to either set the DHE parameters or specify the number of key bits
for use with auto DHE parameters. Additionally, we need to be able to
serialise the DHE parameters to send to the client.

This removes the infamous 'tmp' struct from ssl3_state_internal_st.

ok inoguchi@ tb@

Return 0 on failure from send/get kex functions in the legacy stack.

In the legacy stack, a message handling function returns -1 for failure,
0 for need more data and 1 for success (although in extra special cases
2 may also be used). However, the various send/get kex functions only
need to indicate success or failure - switch these to return 0 on failure
(rather than -1) and use normal result testing.

This leaves GOST unchanged for now, as that code is special and needs
extra work.

ok inoguchi@ tb@

Hoist memset of CBB above EVP_MD_CTX_new() and HMAC_CTX_new() to avoid
a use of uninitialized in the unlikely event that either of them fails.
Problem introduced in r1.128.

CID 345113

ok jsing

Convert ssl_srvr.c to opaque EVP_MD_CTX.

ok inoguchi jsing

Clean up and refactor server side DHE key exchange.

Provide ssl_kex_generate_dhe_params_auto() which handles DHE key generation
based on parameters determined by the specified key bits. Convert the
existing DHE auto parameter selection code into a function that just tells
us how many key bits to use.

Untangle and rework the server side DHE key exchange to use the ssl_kex_*
functions.

ok inoguchi@ tb@

Convert server serialisation of DHE parameters/public key to new functions.

ok inoguchi@ tb@

Stop reaching into EVP_PKEY in the rest of libssl.

ok inoguchi jsing

libssl: don't reach for pkey->save_type.

For some strange historical reason ECDSA_sign() and ECDSA_verify}() have
a type argument that they ignore. For another strange historical reason,
the type passed to them from libssl is pkey->save_type, which is used to
avoid expensive engine lookups when setting the pkey type... Whatever the
aforementioned reasons were, we can't access pkey->save_type with the
OpenSSL 1.1 API, and this is thus in the way of making EVP_PKEY opaque.
Simply pass in 0 instead.

ok jsing

Fold SSL_SESSION_INTERNAL back into SSL_SESSION.

ok beck@ tb@

Provide a way to determine our maximum legacy version.

With the introduction of TLSv1.3, we need the ability to determine our
maximum legacy version and to track our peer's maximum legacy version.
This is needed for both the TLS record layer when using TLSv1.3, plus
it is needed for RSA key exhange in TLS prior to TLSv1.3, where the
maximum legacy version is incorporated in the pre-master secret to
avoid downgrade attacks.

This unbreaks RSA KEX for the TLS client when the non-version specific
method is used with TLSv1.0 or TLSv1.1 (clearly no one does this).

ok tb@

Fold DTLS1_STATE_INTERNAL into DTLS1_STATE.

Now that DTLS1_STATE is opaque, fold DTLS1_STATE_INTERNAL back into
DTLS1_STATE and remove D1I() usage.

ok tb@

Untangle ssl3_get_message() return values.

This function currently has a long return type that may be <= 0 on
error/retry (which is then cast to an int in order to return it up the
stack), or it returns the length of the handshake message (on success).
This obviously means that 0 can be returned for both success and failure,
which is the reason why a separate 'ok' argument has to exist.

Untangle this mess by changing the return value to an int that indicates
success (1) or error/retry (<= 0). The length never needs to actually be
returned as it is already stored in s->internal->init_num (which is where
the return value is read from anyway).

ok tb@

Ensure that a client hello does not have trailing data.

Found by tlsfuzzer.

ok beck@

Clean up and simplify info and msg callbacks.

The info and msg callbacks result in duplication - both for code that
refers to the function pointers and for the call sites. Avoid this by
providing typedefs for the function pointers and pulling the calling
sequences into their own functions.

ok inoguchi@ tb@

Track sigalg used by ourselves and our peer in the legacy stack.

This is needed for upcoming API additions.

Convert legacy stack server to ssl_sigalg_for_peer().

ok inoguchi@ tb@

Move the RSA-PSS check for TLSv1.3 to ssl_sigalg_pkey_ok().

Also, rather than passing in a check_curve flag, pass in the SSL * and
handle version checks internally to ssl_sigalg_pkey_ok(), simplifying
the callers.

ok inoguchi@ tb@

Change ssl_sigalgs_from_value() to perform sigalg list selection.

Rather that passing in a sigalg list at every call site, pass in the
appropriate TLS version and have ssl_sigalgs_from_value() perform the
sigalg list selection itself. This allows the sigalg lists to be made
internal to the sigalgs code.

ok tb@

Rename ssl_sigalg() to ssl_sigalg_from_value().

This makes the code more self-documenting and avoids the ambiguity between
ssl_sigalg the struct and ssl_sigalg the function.

ok tb@

Change ssl_sigalgs_build() to perform sigalg list selection.

Rather that doing sigalg list selection at every call site, pass in the
appropriate TLS version and have ssl_sigalgs_build() perform the sigalg
list selection itself. This reduces code duplication, simplifies the
calling code and is the first step towards internalising the sigalg lists.

ok tb@

Make local header inclusion consistent.

Consistently include local headers in the same location, using the same
grouping/sorting across all files.

Move DTLS structs/definitions/prototypes to dtls_locl.h.

Now that the DTLS structs are opaque, add a dtls_locl.h header and move
internal-only structs from dtls1.h, along with prototypes from ssl_locl.h.
Only pull this header in where DTLS code actually exists.

ok inoguchi@ tb@

Explicitly include <openssl/opensslconf.h> in files using OPENSSL_NO_*

Where a file references to OPENSSL_NO_* conditions, ensure that we
explicitly include <openssl/opensslconf.h> before any references, rather
than relying on another header to pull this in.

Clean up tls1_change_cipher_state().

Replace flag gymnastics at call sites with separate read and write,
functions which call the common code. Condition on s->server instead of
using SSL_ST_ACCEPT, for consistency and more readable code.

ok inoguchi@ tb@

In the TLSv1.2 server, set up the key block after sending the CCS.

This avoids calling into the key block setup code multiple times and makes
the server code consistent with the client.

ok inoguchi@ tb@

Clean up dtls1_reset_seq_numbers().

Rather than doing flag gymnastics, split dtls1_reset_seq_numbers() into
separate read and write functions. Move the calls of these functions into
tls1_change_cipher_state() so they directly follow the change of cipher
state in the record layer, which avoids having to duplicate the calls in
the client and server.

ok inoguchi@ tb@

Clean up and harden TLSv1.2 master key derivation.

The master key and its length are only stored in one location, so it makes
no sense to handle these outside of the derivation function (the current
'out' argument is unused). This simplifies the various call sites.

If derivation fails for some reason, fail hard rather than continuing on
and hoping that something deals with this correctly later.

ok inoguchi@ tb@

Clean up derivation of finished/peer finished.

Make this process more readable by having specific client/server functions,
calling the correct one based on s->server. This allows to remove various
SSL_ST_ACCEPT/SSL_ST_CONNECT checks, along with duplicate code.

ok inoguchi@ tb@

Clean up TLSv1.2 certificate request handshake data.

Currently cert_req is used by clients and cert_request is used by servers.
Replace this by a single cert_request used by either client or server.
Remove the certificate types as they are currently unused. This also fixes
a bug whereby if the number of certificate types exceeds SSL3_CT_NUMBER
the number of bytes read in is insufficient, which will break decoding.

ok inoguchi@ tb@