Remove last peeking at TLS1_FLAGS_SKIP_CERT_VERIFY

This was used for some GOST weirdness. The flag is unused in ports and
there is no user in Debian's codesearch.

ok beck

Remove GOST and STREEBOG support from libssl.

This version of GOST is old and not anywhere close to compliant with
modern GOST standards. It is also very intrusive in libssl and
makes a mess everywhere. Efforts to entice a suitably minded anyone
to care about it have been unsuccessful.

At this point it is probably best to remove this, and if someone
ever showed up who truly needed a working version, it should be
a clean implementation from scratch, and have it use something
closer to the typical API in libcrypto so it would integrate less
painfully here.

This removes it from libssl in preparation for it's removal from
libcrypto with a future major bump

ok tb@

Neuter the SSL_set_debug(3) API

The TLSv1.3 stack didn't support this in the first place, and in the legacy
stack it only added some dubious BIO_flush(3) calls. The sleep call between
SSL_read(3) and SSL_write(3) advertised in the comment next to the flag has
been a sleep call in the s_server since time immemorial, nota bene between
calls to BIO_gets(3). Anyway. This can all go and what remains will go with
the next major bump.

ok jsing

Manually unifdef OPENSSL_NO_ENGINE in ssl_clnt.c

This allows us to simplify ssl_do_client_cert_cb() a bit.

ok jsing

Hide all public symbols in libssl

With the guentherizer 9000

ok tb@

Convert legacy server kex to one-shot sign/verify

This converts ssl3_{get,send}_server_key_exchange() to EVP_DigestVerify()
and EVP_DigestSign(). In order to do this, build the full signed_params
up front and rework the way the key exchange parameters are constructed.
This way we can do the verify and sign steps in one go and at the same
use a more idiomatic approach with CBB/CBS.

with/ok jsing

Easy EVP_Digest{Sign,Verify} conversions for legacy stack

Convert ssl3_send_client_verify_{sigalgs,gost}() to EVP_DigestSign() and
ssl3_get_cert_verify() to EVP_DigestVerify().

ok jsing

spelling fixes; from paul tagliamonte
ok tb

Make internal header file names consistent

Libcrypto currently has a mess of *_lcl.h, *_locl.h, and *_local.h names
used for internal headers. Move all these headers we inherited from
OpenSSL to *_local.h, reserving the name *_internal.h for our own code.
Similarly, move dtls_locl.h and ssl_locl.h to dtls_local and ssl_local.h.
constant_time_locl.h is moved to constant_time.h since it's special.

Adjust all .c files in libcrypto, libssl and regress.

The diff is mechanical with the exception of tls13_quic.c, where
#include <ssl_locl.h> was fixed manually.

discussed with jsing,
no objection bcook

Annotate misuse of EVP_Digest()

The session_id member of SSL_SESSION has 32 bytes for historical
reasons. This precisely accommodates a SHA-256 and is currently
computed using this hash. If the hash function is ever changed,
this will likely overflow. This should be fixed in code. Leave
it at an XXX comment for now.

Pointed out by jsing

Get rid of SSL_CTX_INTERNAL and SSL_INTERNAL.

These are no longer necessary due to SSL_CTX and SSL now being fully
opaque. Merge SSL_CTX_INTERNAL back into SSL_CTX and SSL_INTERNAL back
into SSL.

Prompted by tb@

Move handshake message handling functions from ssl_both.c to client/server.

Currently, ssl_both.c contains several functions that are used by both the
legacy client and legacy server. This interwines the client and server,
making it harder to make progressive changes. While it does deduplicate
some code, it also ends up with code that is conditioned on s->server and
forces the caller to pass in SSL3_ST_* values.

Move these functions from ssl_both.c into ssl_clnt.c and ssl_srvr.c,
renaming as appropriate and removing the s->server conditionals. Also move
the client and server function prototypes from ssl_locl.h into the .c
files, making them static in the process.

ok tb@

Deduplicate peer certificate chain processing code.

Rather than reimplement this in each TLS client and server, deduplicate it
into a single function. Furthermore, rather than dealing with the API
hazard that is SSL_get_peer_cert_chain() in this code, simply produce two
chains - one that has the leaf and one that does not.
SSL_get_peer_cert_chain() can then return the appropriate one.

This also moves the peer cert chain from the SSL_SESSION to the
SSL_HANDSHAKE, which makes more sense since it is not available on
resumption.

ok tb@

Remove redeclaration of ret

When ret was introduced in an outer scope in r1.113, this declaration
wasn't garbage collected.

ok jsing

Simplify certificate list handling code in legacy client.

Tidy up CBS code and remove some unnecessary length checks. Use 'cert' and
'certs' for certificates, rather than 'x' and 'sk'.

ok tb@

Rename uses 'curve' to 'group' and rework tls1 group API.

This reworks various tls1_ curve APIs to indicate success via a boolean
return value and move the output to an out parameter. This makes the
caller code easier and more consistent.

Based on a suggestion by jsing

ok jsing

Add checks to ensure we do not initiate or negotiate handshakes with
versions below the minimum required by the security level.

input & ok jsing

There's tentacles, tentacles everywhere

ok beck jsing

Check the security of DH key shares

ok beck, looks good to jsing

Simplify various CBS_write_bytes() calls

Now that session_id_length is a size_t, we can pass it directly to
CBS_write_bytes() instead of using a temporary variable.

ok jsing

Add missing error check call in ssl3_get_new_session_ticket()

EVP_Digest() can fail, so handle failure appropriately and prepare
switch of session_id_length to a size_t.

ok jsing

Another small readability tweak: compare explicitly against 0 and NULL,
respectively

ok jsing

Add error checking to tls_session_secret_cb() calls

Failure of this undocumented callback was previously silently ignored.
Follow OpenSSL's behavior and throw an internal error (for lack of a
better choice) if the callback failed or if it set the master_key_length
to a negative number.

Unindent the success path and clean up some strange idioms.

ok jsing

Remove incorrect and ungrammattical comment

The fallback to SHA-1 if SHA-256 is disabled fell victim to tedu many
moons ago when this file was still called s3_clnt.c and had no RCS ID.

Bye bye S3I.

S3I has served us well, however now that libssl is fully opaque it is time
to say goodbye. Aside from removing the calloc/free/memset, the rest is
mechanical sed.

ok inoguchi@ tb@

Cleanup/simplify ssl_cert_type()

Remove the X509 argument as it is unused - this was passed so that
ssl_cert_type() can get the public key from the X509 object if the
EVP_PKEY argument is NULL, however this is never the case.

ok tb@

Garbage collect unused ret in ssl3_get_new_session_ticket()

ret is used to bubble up an error from ssl3_get_message() and is unused
otherwise. Zap dead store and silly ret = 1; return ret;

ok inoguchi jsing

In ssl3_get_server_key_exchange() switch a 'goto err' to
'goto fatal_err' so that the illegal_parameter alert is
actually sent in case of X509_get0_pubkey() failure.

ok inoguchi jsing

Remove peer_pkeys from SSL_SESSION.

peer_pkeys comes from some world where peers can send multiple certificates
- in fact, one of each known type. Since we do not live in such a world,
get rid of peer_pkeys and simply use peer_cert instead (in both TLSv1.2
and TLSv1.3, both clients and servers can only send a single leaf
(aka end-entity) certificate).

ok inoguchi@ tb@

Rename 'peer' to 'peer_cert' in SSL_SESSION.

The 'peer' member of SSL_SESSION is the leaf/end-entity certificate
provided by our peer. Rename it since 'peer' on its own is unhelpful.

ok inoguchi@ tb@

Plumb decode errors through key share parsing code.

Distinguish between decode errors and other errors, so that we can send
a SSL_AD_DECODE_ERROR alert when appropriate.

Fixes a tlsfuzzer failure, due to it expecting a decode error alert and
not receiving one.

Prompted by anton@

ok tb@

ssl_check_srvr_ecc_cert_and_alg() only returns 0/1 - test accordingly.

Swap arguments to ssl_check_srvr_ecc_cert_and_alg()

If a libssl function takes an SSL *, it should normally be the first
argument.

Clean up ssl3_{send,get}_client_kex_gost()

Fix leaks, use sizeof() instead of hardcoded sizes, actually check return
codes, explicit_bzero() the premaster secret on the server side and
generally try to kick the GOST kex code into some sort of shape.

ok inoguchi@ tb@

Return 0/1 from ssl3_{send,get}_client_kex_gost()

Like other KEX handling functions, there is no need to return anything
other than failure/success here.

ok inoguchi@ tb@

Remove a comment from Captain Obvious.

Clean up pkey handling in ssl3_get_server_key_exchange()

With TLSv1.2 and earlier, the authentication algorithm used to sign the
ServerKeyExchange message is dependent on the cipher suite in use and has
nothing to do with the key exchange algorithm. As such, check the
authentication algorithm based on the cipher suite in
ssl3_get_server_key_exchange() and handle things accordingly.

ok inoguchi@ tb@

Merge SESS_CERT into SSL_SESSION.

There is no reason for SESS_CERT to exist - remove it and merge its members
into SSL_SESSION for the time being. More clean up to follow.

ok inoguchi@ tb@

Convert legacy TLS client to tls_key_share.

This requires adding DHE support to tls_key_share. In doing so,
tls_key_share_peer_public() has to lose the group argument and gains
an invalid_key argument. The one place that actually needs the group
check is tlsext_keyshare_client_parse(), so add code to do this.

ok inoguchi@ tb@

Return 0 on failure from send/get kex functions in the legacy stack.

In the legacy stack, a message handling function returns -1 for failure,
0 for need more data and 1 for success (although in extra special cases
2 may also be used). However, the various send/get kex functions only
need to indicate success or failure - switch these to return 0 on failure
(rather than -1) and use normal result testing.

This leaves GOST unchanged for now, as that code is special and needs
extra work.

ok inoguchi@ tb@

Use normal result testing for tls1_check_curve().

Refactor ssl3_get_server_kex_ecdhe() to separate parsing and validation.

If we receive something other than a "named curve", send a handshake
failure alert as we're unable to complete the handshake with the given
parameters. If the server responded with a curve that we did not advertise
send an illegal parameter alert.

ok inoguchi@ tb@

Convert ssl_clnt.c to opaque EVP_MD_CTX

ok inoguchi jsing

Move the minimum DHE key size check into ssl_kex_peer_params_dhe()

ok inoguchi@ tb@

Check DH public key in ssl_kex_peer_public_dhe().

Call DH_check_pub_key() after decoding the peer public key - this will be
needed for the server DHE key exchange, but also benefits the client.

ok inoguchi@ tb@

Factor out/rewrite DHE key exchange.

This follows what was done previously for ECDHE EC point key exchange and
will allow for deduplication and further code improvement.

Convert the TLSv1.2 client to use the new DHE key exchange functions.

ok inoguchi@ tb@

Stop reaching into EVP_PKEY in the rest of libssl.

ok inoguchi jsing

libssl: don't reach for pkey->save_type.

For some strange historical reason ECDSA_sign() and ECDSA_verify}() have
a type argument that they ignore. For another strange historical reason,
the type passed to them from libssl is pkey->save_type, which is used to
avoid expensive engine lookups when setting the pkey type... Whatever the
aforementioned reasons were, we can't access pkey->save_type with the
OpenSSL 1.1 API, and this is thus in the way of making EVP_PKEY opaque.
Simply pass in 0 instead.

ok jsing

Fold SSL_SESSION_INTERNAL back into SSL_SESSION.

ok beck@ tb@

Unhandroll X509_up_ref()

ok beck jsing

Provide a way to determine our maximum legacy version.

With the introduction of TLSv1.3, we need the ability to determine our
maximum legacy version and to track our peer's maximum legacy version.
This is needed for both the TLS record layer when using TLSv1.3, plus
it is needed for RSA key exhange in TLS prior to TLSv1.3, where the
maximum legacy version is incorporated in the pre-master secret to
avoid downgrade attacks.

This unbreaks RSA KEX for the TLS client when the non-version specific
method is used with TLSv1.0 or TLSv1.1 (clearly no one does this).

ok tb@

Fold DTLS1_STATE_INTERNAL into DTLS1_STATE.

Now that DTLS1_STATE is opaque, fold DTLS1_STATE_INTERNAL back into
DTLS1_STATE and remove D1I() usage.

ok tb@

Untangle ssl3_get_message() return values.

This function currently has a long return type that may be <= 0 on
error/retry (which is then cast to an int in order to return it up the
stack), or it returns the length of the handshake message (on success).
This obviously means that 0 can be returned for both success and failure,
which is the reason why a separate 'ok' argument has to exist.

Untangle this mess by changing the return value to an int that indicates
success (1) or error/retry (<= 0). The length never needs to actually be
returned as it is already stored in s->internal->init_num (which is where
the return value is read from anyway).

ok tb@

Change tlsext_tick_lifetime_hint to uint32_t.

Now that SSL_SESSION is opaque, change tlsext_tick_lifetime_hint from long
to uint32_t (matching RFC4507), rather than continuing to work around an
inappropriate type choice.

ok tb@

Ensure that a server hello does not have trailing data.

Found by tlsfuzzer.

ok beck@

We need to allow for either a CERTIFICATE or CERTIFICATE_STATUS message
here or we break the handshake with BAD_MESSAGE

ok tb@

RFC 6066 section 8 allows the server MAY choose not send the CertificateStatus
message, even if it has received a "status_request" extension in the client
hello message and has sent a "status_request" extention in the server hello
message. Genua found a site that is this broken. This makes it work.

ok jsing@

Clean up and simplify info and msg callbacks.

The info and msg callbacks result in duplication - both for code that
refers to the function pointers and for the call sites. Avoid this by
providing typedefs for the function pointers and pulling the calling
sequences into their own functions.

ok inoguchi@ tb@

Correct sigalg hash usage when signing content for client verify.

This was inadvertently broken during sigalgs refactoring.

Pull sigalg selection up into ssl3_send_client_verify().

This means that we do sigalg selection for all cases, including those
where are are not sending sigalgs. This is needed in order to track our
signature type in legacy cases.

ok tb@

Track sigalg used by ourselves and our peer in the legacy stack.

This is needed for upcoming API additions.

Convert legacy stack client to ssl_sigalg_for_peer().

ok inoguchi@ tb@

Move the RSA-PSS check for TLSv1.3 to ssl_sigalg_pkey_ok().

Also, rather than passing in a check_curve flag, pass in the SSL * and
handle version checks internally to ssl_sigalg_pkey_ok(), simplifying
the callers.

ok inoguchi@ tb@

Have ssl3_send_client_verify() pass *pkey to called functions.

ssl3_send_client_verify() already has a pointer to the EVP_PKEY for the
certificate - pass this as an argument to the functions that it calls,
rather than duplicating code/variable declarations.

Change ssl_sigalgs_from_value() to perform sigalg list selection.

Rather that passing in a sigalg list at every call site, pass in the
appropriate TLS version and have ssl_sigalgs_from_value() perform the
sigalg list selection itself. This allows the sigalg lists to be made
internal to the sigalgs code.

ok tb@

Rename ssl_sigalg() to ssl_sigalg_from_value().

This makes the code more self-documenting and avoids the ambiguity between
ssl_sigalg the struct and ssl_sigalg the function.

ok tb@

Make local header inclusion consistent.

Consistently include local headers in the same location, using the same
grouping/sorting across all files.

Move DTLS structs/definitions/prototypes to dtls_locl.h.

Now that the DTLS structs are opaque, add a dtls_locl.h header and move
internal-only structs from dtls1.h, along with prototypes from ssl_locl.h.
Only pull this header in where DTLS code actually exists.

ok inoguchi@ tb@

Explicitly include <openssl/opensslconf.h> in files using OPENSSL_NO_*

Where a file references to OPENSSL_NO_* conditions, ensure that we
explicitly include <openssl/opensslconf.h> before any references, rather
than relying on another header to pull this in.

Clean up tls1_change_cipher_state().

Replace flag gymnastics at call sites with separate read and write,
functions which call the common code. Condition on s->server instead of
using SSL_ST_ACCEPT, for consistency and more readable code.

ok inoguchi@ tb@

Clean up dtls1_reset_seq_numbers().

Rather than doing flag gymnastics, split dtls1_reset_seq_numbers() into
separate read and write functions. Move the calls of these functions into
tls1_change_cipher_state() so they directly follow the change of cipher
state in the record layer, which avoids having to duplicate the calls in
the client and server.

ok inoguchi@ tb@

Clean up and harden TLSv1.2 master key derivation.

The master key and its length are only stored in one location, so it makes
no sense to handle these outside of the derivation function (the current
'out' argument is unused). This simplifies the various call sites.

If derivation fails for some reason, fail hard rather than continuing on
and hoping that something deals with this correctly later.

ok inoguchi@ tb@

Clean up derivation of finished/peer finished.

Make this process more readable by having specific client/server functions,
calling the correct one based on s->server. This allows to remove various
SSL_ST_ACCEPT/SSL_ST_CONNECT checks, along with duplicate code.

ok inoguchi@ tb@

Clean up TLSv1.2 certificate request handshake data.

Currently cert_req is used by clients and cert_request is used by servers.
Replace this by a single cert_request used by either client or server.
Remove the certificate types as they are currently unused. This also fixes
a bug whereby if the number of certificate types exceeds SSL3_CT_NUMBER
the number of bytes read in is insufficient, which will break decoding.

ok inoguchi@ tb@

Move reuse_message, message_type, message_size and cert_verify into the
TLSv1.2 handshake struct.

ok inoguchi@ tb@

Update a stale comment and fix a typo.

Garbage collect s->internal->type

This variable is used in the legacy stack to decide whether we are
a server or a client. That's what s->server is for...

The new TLSv1.3 stack failed to set s->internal->type, which resulted
in hilarious mishandling of previous_{client,server}_finished. Indeed,
both client and server would first store the client's verify_data in
previous_server_finished and later overwrite it with the server's
verify_data. Consequently, renegotiation has been completely broken
for more than a year. In fact, server side renegotiation was broken
during the 6.5 release cycle. Clearly, no-one uses this.

This commit fixes client side renegotiation and restores the previous
behavior of SSL_get_client_CA_list(). Server side renegotiation will
be fixed in a later commit.

ok jsing

Rename new_cipher to cipher.

This is in the SSL_HANDSHAKE struct and is what we're currently
negotiating, so there is really nothing more "new" about the cipher
than there is the key block or other parts of the handshake data.

ok inoguchi@ tb@

Add SSL_HANDSHAKE_TLS12 for TLSv1.2 specific handshake data.

Move TLSv1.2 specific components over from SSL_HANDSHAKE.

ok inoguchi@ tb@

Remove ssl_downgrade_max_version().

Now that we store our maximum TLS version at the start of the handshake,
we can check against that directly.

ok inoguchi@ tb@

Improve internal version handling.

Add handshake fields for our minimum TLS version, our maximum TLS version
and the TLS version negotiated during the handshake. Initialise our min/max
versions at the start of the handshake and leave these unchanged. The
negotiated TLS version is set in the client once we receive the ServerHello
and in the server at the point we select the highest shared version.

Provide an ssl_effective_version() function that returns the negotiated TLS
version if known, otherwise our maximum TLS version - this is effectively
what is stored in s->version currently.

Convert most of the internal code to use one of these three version fields,
which greatly simplifies code (especially in the TLS extension handling
code).

ok tb@

Factor out/change some of the legacy client version handling code.

This consolidates the version handling code and will make upcoming changes
easier.

ok tb@

ugly whitespace

Rename f_err into fatal_err.

discussed with jsing

Rename the truncated label into decode_err. This describes its purpose
better and doesn't look odd if there's trailing data for exapmle.
Indent a few labels in the neighborhood while there.

ok jsing

Revise HelloVerifyRequest handling for DTLSv1.2.

Per RFC 6347 section 4.2.1, the HelloVerifyRequest should always contain
DTLSv1.0 - ensure this is the case on the server side, allow both DTLSv1.0
and DTLSv1.2 on the client.

ok tb@

Group HelloVerifyRequest decoding and add missing check for trailing data.

ok tb@

Absorb ssl3_get_algorithm2() into ssl_get_handshake_evp_md().

The mess that is ssl_get_algorithm2() only exists to upgrade the handshake
MAC of a pre-TLSv1.2 cipher suite to SHA256 when used with TLSv1.2. We can
readily do this in ssl_get_handshake_evp_md(), which is far more readable.

ok tb@

Factor out the legacy stack version checks.

Also check for explicit version numbers, rather than just the major version
value.

ok tb@

Replace SSL_IS_DTLS with SSL_is_dtls().

Garbage collect the now unused SSL_IS_DTLS macro.

ok tb@

Condense and simplify TLS methods.

Historically, OpenSSL has had client and server specific methods - the only
difference between these is that the .ssl_connect or .ssl_accept function
pointer is set to ssl_undefined_function, with the intention of reducing
code size for a statically linked binary that was only a client or server.
These days the difference is minimal or non-existant in many cases and
we can reduce the amount of code and complexity by having single method.

Internally remove all of the client and server specific methods,
simplifying code in the process. The external client/server specific API
remain, however these now return the same thing as TLS_method() does.

ok tb@

Merge d1_{clnt,srvr}.c into ssl_{clnt,srvr}.c

The d1_{clnt,srvr}.c contain a single function each - merge these into the
ssl_{clnt,srvr}.c, renaming them with an ssl3_ prefix for consistency.

ok beck@ tb@

Simplify the cleanup of init_buf via a ssl3_release_init_buffer() function.

ok beck@ inoguchi@ tb@

Simplify SSL method lookups.

There are three places where we call tls1_get_{client,server}_method() and
if that returns NULL, call dtls1_get_{client,server}_method(). Simplify
this by combining the lookup into a single function. While here also use
uint16_t for version types.

ok inoguchi@ millert@

Remove cipher_list_by_id.

When parsing a cipher string, a cipher list is created, before being
duplicated and sorted - the second copy being stored as cipher_list_by_id.
This is done only so that a client can ensure that the cipher selected by
a server is in the cipher list. This is pretty pointless given that most
clients are short-lived and that we already had to iterate over the cipher
list in order to build the client hello. Additionally, any update to the
cipher list requires that cipher_list_by_id also be updated and kept in
sync.

Remove all of this and replace it with a simple linear scan - the overhead
of duplicating and sorting the cipher list likely exceeds that of a simple
linear scan over the cipher list (64 maximum, more typically ~9 or so).

ok beck@ tb@

Improve argument order for the internal tlsext API

Move is_server and msg_type right after the SSL object so that CBS
and CBB and alert come last. This brings these functions more in
line with other internal functions and separates state from data.

requested by jsing

Enable GOST_SIG_FORMAT_RS_LE when verifying certificate signatures.

GOST cipher suites requires that CertVerify signatures be generated in a
special way (see ssl3_send_client_kex_gost(), ssl3_get_cert_verify()).
However, the GOST_SIG_FORMAT_RS_LE flag was not passed in case of TLS 1.2
connections (because they use different code path). Set this flag on
GOST PKEYs.

Diff from Dmitry Baryshkov <dbaryshkov@gmail.com>

Sponsored by ROSA Linux

ok inoguchi@ tb@

Correct downgrade sentinels when a version pinned method is in use.

Previously only the enabled protocol versions were considered, however we
also have to consider the method in use which may be version pinned.

Found the hard way by danj@ with haproxy and force-tlsv12.

ok beck@ inoguchi@ tb@

Replace SSL_PKEY_RSA_ENC/SSL_PKEY_RSA_SIGN with SSL_PKEY_RSA.

Some time prior to SSLeay 0.8.1b, SSL_PKEY_RSA_SIGN got added with the
intention of handling RSA sign only certificates... this incomplete code
had the following comment:

/* check to see if this is a signing only certificate */
/* EAY EAY EAY EAY */

And while the comment was removed in 2005, the incomplete RSA sign-only
handling has remained ever since.

Remove SSL_PKEY_RSA_SIGN and rename SSL_PKEY_RSA_ENC to SSL_PKEY_RSA. While
here also remove the unused SSL_PKEY_DH_RSA.

ok tb@

Use size_t for OCSP response length.

The OCSP response length is currently an integer, which is overloaded with
-1 meaning "unset". Use a size_t for the OCSP response length and infer
unset from the OCSP response being NULL. This makes code more readable,
simpler and less error prone.

ok beck@

Drop a redundant test. It's effectively doing the same test twice
and if the two lengths differed, the later CBS_write_bytes() would
correctly fail anyway.

Discussed with jsing

RFC 8446, section 4.1.3: If a TLSv1.2 client receives a ServerHello for
TLSv1.1 or below, it should check whether the server's random value
contains the magic downgrade protection cookie and in that case abort
the handshake with an illegal parameter alert.

ok inoguchi, jsing

Factor out/rewrite the ECDHE EC point key exchange code.

This reduces replication between the existing TLS client/server and allows
the code to soon be reused for TLSv1.3.

With feedback from inoguchi@ and tb@

ok inoguchi@ tb@

Remove the ssl_get_message function pointer from SSL_METHOD_INTERNAL.

ssl_get_message is essentially a switch between ssl3_get_message and
dtls1_get_message, both only used by the legacy stack. Instead, use
SSL_IS_DTLS() in ssl3_get_message to call the DTLS function when
necessary.

ok beck@ inoguchi@ tb@

Clean up and simplify the client verify code:

- Be consistent with _len naming.
- Use size_t where possible/appropriate.
- Group the CBB code.
- Use EVP_MAX_MD_SIZE consistently, instead of "magic" values.
- Switch GOST to EVP_DigestSign*, making it similar to sigalgs.

ok tb@ a while back.

Defer sigalgs selection until the certificate is known.

Previously the signature algorithm was selected when the TLS extension was
parsed (or the client received a certificate request), however the actual
certificate to be used is not known at this stage. This leads to various
problems, including the selection of a signature algorithm that cannot be
used with the certificate key size (as found by jeremy@ via ruby regress).

Instead, store the signature algorithms list and only select a signature
algorithm when we're ready to do signature generation.

Joint work with beck@.

Rework ssl3_output_cert_chain() to take a CERT_PKEY and consider chains.

We will now include the certificates in the chain in the certificate list,
or use the existing extra_certs if present. Failing that we fall back to
the automatic chain building if not disabled.

This also simplifies the code significantly.

ok beck@ tb@

Revert TLS1_get{,_client}_version simplification because DTLS.

Partially clean up the TLS1_get_{,client}_version macros.

LibreSSL only supports TLSv1.0 and above, hence the checks the macros are
performing are useless. Simplify them to their effective code. Also place
both under #ifndef LIBRESSL_INTERNAL and use the variables directly in our
code, which improves readability.

ok tb@

Rename tls1_handshake_hash*() to tls1_transcript_hash*().

While handshake hash is correct (in as far as it is a hash of handshake
messages), using tls1_transcript_hash*() aligns them with the naming of the
tls1_transcript*() functions. Additionally, the TLSv1.3 specification uses
Transcript-Hash and "transcript hash", which this matches.

ok inoguchi@ tb@

Modify sigalgs extension processing to accomodate TLS 1.3.
- Make a separate sigalgs list for TLS 1.3 including only modern
algorithm choices which we use when the handshake will not negotiate
TLS 1.2.
- Modify the legacy sigalgs for TLS 1.2 to include the RSA PSS algorithms as
mandated by RFC8446 when the handshake will permit negotiation of TLS 1.2
from a 1.3 handshake.
ok jsing@ tb@

revert previous, accidentally contained another diff in addition
to the one I intended to commit

Modify sigalgs extension processing for TLS 1.3.
- Make a separate sigalgs list for TLS 1.3 including only modern
algorithm choices which we use when the handshake will not negotiate
TLS 1.2
- Modify the legacy sigalgs for TLS 1.2 to include the RSA PSS algorithms as
mandated by RFC8446 when the handshake will permit negotiation of TLS 1.2
ok jsing@ tb@

Rename TLS extension handling to use less "hello".

When the TLS extension code was rewritten, TLS extensions could only exist
in ClientHello and ServerHello messages - as such, they were named in pairs
of *_clienthello_{needs,build} which would be called by the client and
*_clienthello_parse. Likewise for *_serverhello_{needs,build} which would
be called by a server and *_serverhello_parse, which would be called by a
client.

Enter TLSv1.3 - TLS extensions can now exist in one of seven messages,
with only certain types being allowed to appear in each, meaning the naming
scheme no longer works. Instead, rename them to indicate the caller rather
than the message type - this effectively means:

clienthello_needs -> client_needs
clienthello_build -> client_build
clienthello_parse -> server_parse
serverhello_needs -> server_needs
serverhello_build -> server_build
serverhello_parse -> client_parse

ok beck@ tb@

Refactor a tangle in ssl3_send_client_verify() into one function for each
type, sigalgs/rsa/ec/gost. Move a few special dances for GOST where they
belong now. This prompted a fix for a long-standing bug with GOST client
certificate authentication where tls1_transcript_data() fails since the
transcript was already freed before. Add a bit of missing error checking
and leave some further cleanup for later.

idea, guidance & ok jsing

Fix DTLS transcript handling for HelloVerifyRequest.

If DTLS sees a HelloVerifyRequest the transcript is reset - the previous
tls1_init_finished_mac() function could be called multiple times and would
discard any existing state. The replacement tls1_transcript_init() is more
strict and fails if a transcript already exists.

Provide an explicit tls1_transcript_reset() function and call it from the
appropriate places. This also lets us make DTLS less of a special snowflake
and call tls1_transcript_init() in the same place as used for TLS.

ok beck@ tb@

Revert previous - DTLSv1 uses MD5+SHA1 for RSA signature verification.

Discussed with beck@

Fix DTLS, because DTLS still remains a special flower, allows regress to pass

revert previous

Fix DTLS. Because the DTLS code is strange. I am really coming around to
joel's line of thinking about it

Unbreak legacy ciphers for prior to 1.1 by setting having a legacy
sigalg for MD5_SHA1 and using it as the non sigalgs default
ok jsing@

Add check function to verify that pkey is usable with a sigalg.
Include check for appropriate RSA key size when used with PSS.
ok tb@

Add support for RSA PSS algorithims being used in sigalgs.
lightly tested, but will need sanity checks and regress test changes
before being added to any sigalgs list for real
ok jsing@ tb@

Convert signatures and verifcation to use the EVP_DigestXXX api
to allow for adding PSS, Nuke the now unneejded guard around the PSS
algorithms in the sigalgs table
ok jsing@ tb@

Stop keeping track of sigalgs by guessing it from digest and pkey,
just keep the sigalg around so we can remember what we actually
decided to use.
ok jsing@

Ensure we free the handshake transcript upon session resumption.

Found the hard way by jmc@

ok tb@

Ensure we only choose sigalgs from our prefernce list, not the whole list
ok jsing@

Reimplement the sigalgs processing code into a new implementation
that will be usable with TLS 1.3 with less eye bleed.
ok jsing@ tb@

Clean up and simplify the handshake transcript code.

This provides a cleaner, simpler and more readable API, with code that uses
a BUF_MEM instead of a BIO.

ok beck@ ("hurry up") and tb@.

Stop pretending that a cert member in a SSL and SSL_CTX can be NULL.

ok beck@ tb@

Ensure the handshake transcript is cleaned up.

Add a check at the completion of the client/server handshake to ensure that
the handshake transcript has been freed. Fix the case where a server asks
the client for a certificate, but it does not have one, resulting in the
handshake transcript being left around post-handshake.

ok bcook@ tb@

Use the newer/more sensible names for EVP_MD_CTX_* functions.

EVP_MD_CTX_create -> EVP_MD_CTX_new
EVP_MD_CTX_destroy -> EVP_MD_CTX_free

This should make the intent more obvious and reduce head scratching during
code reviews.

Raised by tb@

Clean up handshake message start/finish functions.

Now that all handshake messages are created using CBB, remove the non-CBB
ssl3_handshake_msg_start()/ssl3_handshake_msg_finish() functions. Rename
the CBB variants by dropping the _cbb suffix.

ok bcook@ inoguchi@ tb@

Rename and collapse tls12_get_sigandhash_cbb().

Now that all callers of tls12_get_sigandhash() have been converted to CBB,
collapse tls12_get_sigandhash() and tls12_get_sigandhash_cbb() into a
single function. Rename it to tls12_gethashandsig() to be representative
of the actual order of the sigalgs parameters, and perform some other
clean up.

ok inoguchi@ tb@

Convert ssl3_send_client_verify() to CBB.

ok inoguchi@ tb@

Convert ssl3_get_server_key_exchange() to CBS.

ok inoguchi@ tb@

Remove now unused variable, that got left behind from a previous change.

Actually check the return values for EVP_Sign* and EVP_Verify*.

ok bcook@ beck@ tb@

Simplify server key exchange signature verification.

Everything can go through the EVP_Verify* code path.

ok inoguchi@ tb@

Check the return value from DH_size() in ssl3_send_client_kex_dhe().

ok beck@ inoguchi@

Fix a malloc() NULL check in ssl3_send_client_kex_ecdhe_ecp(), by adding
the missing goto. While here also remove a set of unnecessary parentheses.

Nuke SSL3_FLAGS_DELAY_CLIENT_FINISHED and SSL3_FLAGS_POP_BUFFER.

These flags enabled experimental behaviour in the write path, which nothing
uses. Removing this code greatly simplifies ssl3_write().

ok beck@ inoguchi@ sthen@ tb@

Complete the TLS extension rewrite on the client-side.

The RI logic gets pulled up into ssl3_get_server_hello() and
ssl_parse_serverhello_tlsext() gets replaced by tlsext_client_parse(),
which allows a CBS to be passed all the way down.

This also deduplicates the tlsext_client_build() and tlsext_server_build()
code.

ok beck@

Rename ssl3_client_hello() to ssl3_send_client_hello() for consistency.

Convert ssl3_client_hello() to CBB.

As part of this, change ssl_cipher_list_to_bytes() to take a CBB argument,
rather than a pointer/length. Some additional clean up/renames while here.

Based on a diff from doug@

((remove) (some) (unnecessary) (parentheses))

Part of a diff from doug@

Merge dtls1_connect() into ssl3_connect(), removing a large amount of
duplicated code. For now this is essentially adds a diff of the two
functions with 'if (SSL_IS_DTLS(s))' - further clean up and improvement
will follow.

ok inoguchi@

Fix some style/whitespace/indentation issues in ssl3_connect().

Convert TLS signature algorithms extension handling to the new framework.

ok beck@ doug@

Remove NPN support.

NPN was never standardised and the last draft expired in October 2012.
ALPN was standardised in July 2014 and has been supported in LibreSSL
since December 2014. NPN has also been removed from Chromium in May 2016.

TLS clients and servers that try to use/enable NPN will fail gracefully and
fallback to the default protocol, since it will essentially appear that the
otherside does not support NPN. At some point in the future we will
actually remove the NPN related symbols entirely.

ok bcook@ beck@ doug@

Remove support for DSS/DSA, since we removed the cipher suites a while
back.

ok guenther@

Move state from ssl->internal to the handshake structure.
while we are at it, convert SSLerror to use a function
internally, so that we may later allocate the handshake
structure and check for it
ok jsing@

Bring in an SSL_HANDSHAKE structure and commence the great shovelling
ok jsing@, gcc@, regress@

Convert various client key exchange functions to freezero(3). The memory
contents needs to be made inaccessible - this is simpler and less error
prone than the current "if not NULL, explicit_bzero(); free()" dance.

Remove the handshake digests and related code, replacing remaining uses
with the handshake hash. For now tls1_digest_cached_records() is retained
to release the handshake buffer.

ok beck@ inoguchi@

Provide a rolling handshake hash that commences as soon as the cipher
suite has been selected, and convert the final finish MAC to use this
handshake hash.

This is a first step towards cleaning up the current handshake
buffer/digest code.

ok beck@ inoguchi@

Convert various handshake message generation functions to CBB.

ok beck@ inoguchi@

Call ssl3_handshake_write() instead of ssl3_do_write() - this was missed
when ssl3_send_client_certificate() was converted to the standard handshake
functions in r1.150 of s3_clnt.c.

This has no impact on TLS, however it causes the DTLS client to fail if the
server sends a certificate request, since the TLS MAC is calculated on a
non-populated DTLS header.

Issue reported by umokk on github.

Convert ssl3_{get,send}_server_key_exchange() to EVP_md5_sha1().

ok inoguchi@

Stop pretending that MD5 and SHA1 might not exist - rather than locating
"ssl3-md5" and "ssl-sha1", call the EVP_md5() and EVP_sha1() functions
directly.

ok beck@ inoguchi@

Change SSLerror() back to taking two args, with the first one being an SSL *.
Make a table of "function codes" which maps the internal state of the SSL *
to something like a useful name so in a typical error in the connection you
know in what sort of place in the handshake things happened. (instead of
by arcane function name).
Add SSLerrorx() for when we don't have an SSL *
ok jsing@ after us both being prodded by bluhm@ to make it not terrible

Finish the fallout of the SSLerr->SSLerror cleanup to get rid of the ugly
line wraps that resulted

Send the error function codes to rot in the depths of hell where they belong
We leave a single funciton code (0xFFF) to say "SSL_internal" so the public
API will not break, and we replace all internal use of the two argument
SSL_err() with the internal only SSL_error() that only takes a reason code.
ok jsing@

Remove most of SSL3_ENC_METHOD - we can just inline the function calls
and defines since they are the same everywhere.

ok beck@

Rename s3_{both,clnt,pkt_srvr}.c to have an ssl_ prefix since they are no
longer SSLv3 code.

ok beck@

Neuter the SSL_set_debug(3) API

The TLSv1.3 stack didn't support this in the first place, and in the legacy
stack it only added some dubious BIO_flush(3) calls. The sleep call between
SSL_read(3) and SSL_write(3) advertised in the comment next to the flag has
been a sleep call in the s_server since time immemorial, nota bene between
calls to BIO_gets(3). Anyway. This can all go and what remains will go with
the next major bump.

ok jsing

Manually unifdef OPENSSL_NO_ENGINE in ssl_clnt.c

This allows us to simplify ssl_do_client_cert_cb() a bit.

ok jsing

Hide all public symbols in libssl

With the guentherizer 9000

ok tb@

Convert legacy server kex to one-shot sign/verify

This converts ssl3_{get,send}_server_key_exchange() to EVP_DigestVerify()
and EVP_DigestSign(). In order to do this, build the full signed_params
up front and rework the way the key exchange parameters are constructed.
This way we can do the verify and sign steps in one go and at the same
use a more idiomatic approach with CBB/CBS.

with/ok jsing

Easy EVP_Digest{Sign,Verify} conversions for legacy stack

Convert ssl3_send_client_verify_{sigalgs,gost}() to EVP_DigestSign() and
ssl3_get_cert_verify() to EVP_DigestVerify().

ok jsing

spelling fixes; from paul tagliamonte
ok tb

Make internal header file names consistent

Libcrypto currently has a mess of *_lcl.h, *_locl.h, and *_local.h names
used for internal headers. Move all these headers we inherited from
OpenSSL to *_local.h, reserving the name *_internal.h for our own code.
Similarly, move dtls_locl.h and ssl_locl.h to dtls_local and ssl_local.h.
constant_time_locl.h is moved to constant_time.h since it's special.

Adjust all .c files in libcrypto, libssl and regress.

The diff is mechanical with the exception of tls13_quic.c, where
#include <ssl_locl.h> was fixed manually.

discussed with jsing,
no objection bcook

Annotate misuse of EVP_Digest()

The session_id member of SSL_SESSION has 32 bytes for historical
reasons. This precisely accommodates a SHA-256 and is currently
computed using this hash. If the hash function is ever changed,
this will likely overflow. This should be fixed in code. Leave
it at an XXX comment for now.

Pointed out by jsing

Get rid of SSL_CTX_INTERNAL and SSL_INTERNAL.

These are no longer necessary due to SSL_CTX and SSL now being fully
opaque. Merge SSL_CTX_INTERNAL back into SSL_CTX and SSL_INTERNAL back
into SSL.

Prompted by tb@

Move handshake message handling functions from ssl_both.c to client/server.

Currently, ssl_both.c contains several functions that are used by both the
legacy client and legacy server. This interwines the client and server,
making it harder to make progressive changes. While it does deduplicate
some code, it also ends up with code that is conditioned on s->server and
forces the caller to pass in SSL3_ST_* values.

Move these functions from ssl_both.c into ssl_clnt.c and ssl_srvr.c,
renaming as appropriate and removing the s->server conditionals. Also move
the client and server function prototypes from ssl_locl.h into the .c
files, making them static in the process.

ok tb@

Deduplicate peer certificate chain processing code.

Rather than reimplement this in each TLS client and server, deduplicate it
into a single function. Furthermore, rather than dealing with the API
hazard that is SSL_get_peer_cert_chain() in this code, simply produce two
chains - one that has the leaf and one that does not.
SSL_get_peer_cert_chain() can then return the appropriate one.

This also moves the peer cert chain from the SSL_SESSION to the
SSL_HANDSHAKE, which makes more sense since it is not available on
resumption.

ok tb@

Remove redeclaration of ret

When ret was introduced in an outer scope in r1.113, this declaration
wasn't garbage collected.

ok jsing

Simplify certificate list handling code in legacy client.

Tidy up CBS code and remove some unnecessary length checks. Use 'cert' and
'certs' for certificates, rather than 'x' and 'sk'.

ok tb@

Rename uses 'curve' to 'group' and rework tls1 group API.

This reworks various tls1_ curve APIs to indicate success via a boolean
return value and move the output to an out parameter. This makes the
caller code easier and more consistent.

Based on a suggestion by jsing

ok jsing

Add checks to ensure we do not initiate or negotiate handshakes with
versions below the minimum required by the security level.

input & ok jsing

There's tentacles, tentacles everywhere

ok beck jsing

Check the security of DH key shares

ok beck, looks good to jsing

Simplify various CBS_write_bytes() calls

Now that session_id_length is a size_t, we can pass it directly to
CBS_write_bytes() instead of using a temporary variable.

ok jsing

Add missing error check call in ssl3_get_new_session_ticket()

EVP_Digest() can fail, so handle failure appropriately and prepare
switch of session_id_length to a size_t.

ok jsing

Another small readability tweak: compare explicitly against 0 and NULL,
respectively

ok jsing

Add error checking to tls_session_secret_cb() calls

Failure of this undocumented callback was previously silently ignored.
Follow OpenSSL's behavior and throw an internal error (for lack of a
better choice) if the callback failed or if it set the master_key_length
to a negative number.

Unindent the success path and clean up some strange idioms.

ok jsing

Remove incorrect and ungrammattical comment

The fallback to SHA-1 if SHA-256 is disabled fell victim to tedu many
moons ago when this file was still called s3_clnt.c and had no RCS ID.

Bye bye S3I.

S3I has served us well, however now that libssl is fully opaque it is time
to say goodbye. Aside from removing the calloc/free/memset, the rest is
mechanical sed.

ok inoguchi@ tb@

Cleanup/simplify ssl_cert_type()

Remove the X509 argument as it is unused - this was passed so that
ssl_cert_type() can get the public key from the X509 object if the
EVP_PKEY argument is NULL, however this is never the case.

ok tb@

Garbage collect unused ret in ssl3_get_new_session_ticket()

ret is used to bubble up an error from ssl3_get_message() and is unused
otherwise. Zap dead store and silly ret = 1; return ret;

ok inoguchi jsing

In ssl3_get_server_key_exchange() switch a 'goto err' to
'goto fatal_err' so that the illegal_parameter alert is
actually sent in case of X509_get0_pubkey() failure.

ok inoguchi jsing

Remove peer_pkeys from SSL_SESSION.

peer_pkeys comes from some world where peers can send multiple certificates
- in fact, one of each known type. Since we do not live in such a world,
get rid of peer_pkeys and simply use peer_cert instead (in both TLSv1.2
and TLSv1.3, both clients and servers can only send a single leaf
(aka end-entity) certificate).

ok inoguchi@ tb@

Rename 'peer' to 'peer_cert' in SSL_SESSION.

The 'peer' member of SSL_SESSION is the leaf/end-entity certificate
provided by our peer. Rename it since 'peer' on its own is unhelpful.

ok inoguchi@ tb@

Plumb decode errors through key share parsing code.

Distinguish between decode errors and other errors, so that we can send
a SSL_AD_DECODE_ERROR alert when appropriate.

Fixes a tlsfuzzer failure, due to it expecting a decode error alert and
not receiving one.

Prompted by anton@

ok tb@

ssl_check_srvr_ecc_cert_and_alg() only returns 0/1 - test accordingly.

Swap arguments to ssl_check_srvr_ecc_cert_and_alg()

If a libssl function takes an SSL *, it should normally be the first
argument.

Clean up ssl3_{send,get}_client_kex_gost()

Fix leaks, use sizeof() instead of hardcoded sizes, actually check return
codes, explicit_bzero() the premaster secret on the server side and
generally try to kick the GOST kex code into some sort of shape.

ok inoguchi@ tb@

Return 0/1 from ssl3_{send,get}_client_kex_gost()

Like other KEX handling functions, there is no need to return anything
other than failure/success here.

ok inoguchi@ tb@

Remove a comment from Captain Obvious.

Clean up pkey handling in ssl3_get_server_key_exchange()

With TLSv1.2 and earlier, the authentication algorithm used to sign the
ServerKeyExchange message is dependent on the cipher suite in use and has
nothing to do with the key exchange algorithm. As such, check the
authentication algorithm based on the cipher suite in
ssl3_get_server_key_exchange() and handle things accordingly.

ok inoguchi@ tb@

Merge SESS_CERT into SSL_SESSION.

There is no reason for SESS_CERT to exist - remove it and merge its members
into SSL_SESSION for the time being. More clean up to follow.

ok inoguchi@ tb@

Convert legacy TLS client to tls_key_share.

This requires adding DHE support to tls_key_share. In doing so,
tls_key_share_peer_public() has to lose the group argument and gains
an invalid_key argument. The one place that actually needs the group
check is tlsext_keyshare_client_parse(), so add code to do this.

ok inoguchi@ tb@

Return 0 on failure from send/get kex functions in the legacy stack.

In the legacy stack, a message handling function returns -1 for failure,
0 for need more data and 1 for success (although in extra special cases
2 may also be used). However, the various send/get kex functions only
need to indicate success or failure - switch these to return 0 on failure
(rather than -1) and use normal result testing.

This leaves GOST unchanged for now, as that code is special and needs
extra work.

ok inoguchi@ tb@

Use normal result testing for tls1_check_curve().

Refactor ssl3_get_server_kex_ecdhe() to separate parsing and validation.

If we receive something other than a "named curve", send a handshake
failure alert as we're unable to complete the handshake with the given
parameters. If the server responded with a curve that we did not advertise
send an illegal parameter alert.

ok inoguchi@ tb@

Convert ssl_clnt.c to opaque EVP_MD_CTX

ok inoguchi jsing

Move the minimum DHE key size check into ssl_kex_peer_params_dhe()

ok inoguchi@ tb@

Check DH public key in ssl_kex_peer_public_dhe().

Call DH_check_pub_key() after decoding the peer public key - this will be
needed for the server DHE key exchange, but also benefits the client.

ok inoguchi@ tb@

Factor out/rewrite DHE key exchange.

This follows what was done previously for ECDHE EC point key exchange and
will allow for deduplication and further code improvement.

Convert the TLSv1.2 client to use the new DHE key exchange functions.

ok inoguchi@ tb@

Stop reaching into EVP_PKEY in the rest of libssl.

ok inoguchi jsing

libssl: don't reach for pkey->save_type.

For some strange historical reason ECDSA_sign() and ECDSA_verify}() have
a type argument that they ignore. For another strange historical reason,
the type passed to them from libssl is pkey->save_type, which is used to
avoid expensive engine lookups when setting the pkey type... Whatever the
aforementioned reasons were, we can't access pkey->save_type with the
OpenSSL 1.1 API, and this is thus in the way of making EVP_PKEY opaque.
Simply pass in 0 instead.

ok jsing

Fold SSL_SESSION_INTERNAL back into SSL_SESSION.

ok beck@ tb@

Unhandroll X509_up_ref()

ok beck jsing

Provide a way to determine our maximum legacy version.

With the introduction of TLSv1.3, we need the ability to determine our
maximum legacy version and to track our peer's maximum legacy version.
This is needed for both the TLS record layer when using TLSv1.3, plus
it is needed for RSA key exhange in TLS prior to TLSv1.3, where the
maximum legacy version is incorporated in the pre-master secret to
avoid downgrade attacks.

This unbreaks RSA KEX for the TLS client when the non-version specific
method is used with TLSv1.0 or TLSv1.1 (clearly no one does this).

ok tb@

Fold DTLS1_STATE_INTERNAL into DTLS1_STATE.

Now that DTLS1_STATE is opaque, fold DTLS1_STATE_INTERNAL back into
DTLS1_STATE and remove D1I() usage.

ok tb@

Untangle ssl3_get_message() return values.

This function currently has a long return type that may be <= 0 on
error/retry (which is then cast to an int in order to return it up the
stack), or it returns the length of the handshake message (on success).
This obviously means that 0 can be returned for both success and failure,
which is the reason why a separate 'ok' argument has to exist.

Untangle this mess by changing the return value to an int that indicates
success (1) or error/retry (<= 0). The length never needs to actually be
returned as it is already stored in s->internal->init_num (which is where
the return value is read from anyway).

ok tb@

Change tlsext_tick_lifetime_hint to uint32_t.

Now that SSL_SESSION is opaque, change tlsext_tick_lifetime_hint from long
to uint32_t (matching RFC4507), rather than continuing to work around an
inappropriate type choice.

ok tb@

Ensure that a server hello does not have trailing data.

Found by tlsfuzzer.

ok beck@

We need to allow for either a CERTIFICATE or CERTIFICATE_STATUS message
here or we break the handshake with BAD_MESSAGE

ok tb@

RFC 6066 section 8 allows the server MAY choose not send the CertificateStatus
message, even if it has received a "status_request" extension in the client
hello message and has sent a "status_request" extention in the server hello
message. Genua found a site that is this broken. This makes it work.

ok jsing@

Clean up and simplify info and msg callbacks.

The info and msg callbacks result in duplication - both for code that
refers to the function pointers and for the call sites. Avoid this by
providing typedefs for the function pointers and pulling the calling
sequences into their own functions.

ok inoguchi@ tb@

Correct sigalg hash usage when signing content for client verify.

This was inadvertently broken during sigalgs refactoring.

Pull sigalg selection up into ssl3_send_client_verify().

This means that we do sigalg selection for all cases, including those
where are are not sending sigalgs. This is needed in order to track our
signature type in legacy cases.

ok tb@

Track sigalg used by ourselves and our peer in the legacy stack.

This is needed for upcoming API additions.

Convert legacy stack client to ssl_sigalg_for_peer().

ok inoguchi@ tb@

Move the RSA-PSS check for TLSv1.3 to ssl_sigalg_pkey_ok().

Also, rather than passing in a check_curve flag, pass in the SSL * and
handle version checks internally to ssl_sigalg_pkey_ok(), simplifying
the callers.

ok inoguchi@ tb@

Have ssl3_send_client_verify() pass *pkey to called functions.

ssl3_send_client_verify() already has a pointer to the EVP_PKEY for the
certificate - pass this as an argument to the functions that it calls,
rather than duplicating code/variable declarations.

Change ssl_sigalgs_from_value() to perform sigalg list selection.

Rather that passing in a sigalg list at every call site, pass in the
appropriate TLS version and have ssl_sigalgs_from_value() perform the
sigalg list selection itself. This allows the sigalg lists to be made
internal to the sigalgs code.

ok tb@

Rename ssl_sigalg() to ssl_sigalg_from_value().

This makes the code more self-documenting and avoids the ambiguity between
ssl_sigalg the struct and ssl_sigalg the function.

ok tb@

Make local header inclusion consistent.

Consistently include local headers in the same location, using the same
grouping/sorting across all files.

Move DTLS structs/definitions/prototypes to dtls_locl.h.

Now that the DTLS structs are opaque, add a dtls_locl.h header and move
internal-only structs from dtls1.h, along with prototypes from ssl_locl.h.
Only pull this header in where DTLS code actually exists.

ok inoguchi@ tb@

Explicitly include <openssl/opensslconf.h> in files using OPENSSL_NO_*

Where a file references to OPENSSL_NO_* conditions, ensure that we
explicitly include <openssl/opensslconf.h> before any references, rather
than relying on another header to pull this in.

Clean up tls1_change_cipher_state().

Replace flag gymnastics at call sites with separate read and write,
functions which call the common code. Condition on s->server instead of
using SSL_ST_ACCEPT, for consistency and more readable code.

ok inoguchi@ tb@

Clean up dtls1_reset_seq_numbers().

Rather than doing flag gymnastics, split dtls1_reset_seq_numbers() into
separate read and write functions. Move the calls of these functions into
tls1_change_cipher_state() so they directly follow the change of cipher
state in the record layer, which avoids having to duplicate the calls in
the client and server.

ok inoguchi@ tb@

Clean up and harden TLSv1.2 master key derivation.

The master key and its length are only stored in one location, so it makes
no sense to handle these outside of the derivation function (the current
'out' argument is unused). This simplifies the various call sites.

If derivation fails for some reason, fail hard rather than continuing on
and hoping that something deals with this correctly later.

ok inoguchi@ tb@

Clean up derivation of finished/peer finished.

Make this process more readable by having specific client/server functions,
calling the correct one based on s->server. This allows to remove various
SSL_ST_ACCEPT/SSL_ST_CONNECT checks, along with duplicate code.

ok inoguchi@ tb@

Clean up TLSv1.2 certificate request handshake data.

Currently cert_req is used by clients and cert_request is used by servers.
Replace this by a single cert_request used by either client or server.
Remove the certificate types as they are currently unused. This also fixes
a bug whereby if the number of certificate types exceeds SSL3_CT_NUMBER
the number of bytes read in is insufficient, which will break decoding.

ok inoguchi@ tb@

Move reuse_message, message_type, message_size and cert_verify into the
TLSv1.2 handshake struct.

ok inoguchi@ tb@

Update a stale comment and fix a typo.

Garbage collect s->internal->type

This variable is used in the legacy stack to decide whether we are
a server or a client. That's what s->server is for...

The new TLSv1.3 stack failed to set s->internal->type, which resulted
in hilarious mishandling of previous_{client,server}_finished. Indeed,
both client and server would first store the client's verify_data in
previous_server_finished and later overwrite it with the server's
verify_data. Consequently, renegotiation has been completely broken
for more than a year. In fact, server side renegotiation was broken
during the 6.5 release cycle. Clearly, no-one uses this.

This commit fixes client side renegotiation and restores the previous
behavior of SSL_get_client_CA_list(). Server side renegotiation will
be fixed in a later commit.

ok jsing

Rename new_cipher to cipher.

This is in the SSL_HANDSHAKE struct and is what we're currently
negotiating, so there is really nothing more "new" about the cipher
than there is the key block or other parts of the handshake data.

ok inoguchi@ tb@

Add SSL_HANDSHAKE_TLS12 for TLSv1.2 specific handshake data.

Move TLSv1.2 specific components over from SSL_HANDSHAKE.

ok inoguchi@ tb@

Remove ssl_downgrade_max_version().

Now that we store our maximum TLS version at the start of the handshake,
we can check against that directly.

ok inoguchi@ tb@

Improve internal version handling.

Add handshake fields for our minimum TLS version, our maximum TLS version
and the TLS version negotiated during the handshake. Initialise our min/max
versions at the start of the handshake and leave these unchanged. The
negotiated TLS version is set in the client once we receive the ServerHello
and in the server at the point we select the highest shared version.

Provide an ssl_effective_version() function that returns the negotiated TLS
version if known, otherwise our maximum TLS version - this is effectively
what is stored in s->version currently.

Convert most of the internal code to use one of these three version fields,
which greatly simplifies code (especially in the TLS extension handling
code).

ok tb@

Factor out/change some of the legacy client version handling code.

This consolidates the version handling code and will make upcoming changes
easier.

ok tb@

ugly whitespace

Rename f_err into fatal_err.

discussed with jsing

Rename the truncated label into decode_err. This describes its purpose
better and doesn't look odd if there's trailing data for exapmle.
Indent a few labels in the neighborhood while there.

ok jsing

Revise HelloVerifyRequest handling for DTLSv1.2.

Per RFC 6347 section 4.2.1, the HelloVerifyRequest should always contain
DTLSv1.0 - ensure this is the case on the server side, allow both DTLSv1.0
and DTLSv1.2 on the client.

ok tb@

Group HelloVerifyRequest decoding and add missing check for trailing data.

ok tb@

Absorb ssl3_get_algorithm2() into ssl_get_handshake_evp_md().

The mess that is ssl_get_algorithm2() only exists to upgrade the handshake
MAC of a pre-TLSv1.2 cipher suite to SHA256 when used with TLSv1.2. We can
readily do this in ssl_get_handshake_evp_md(), which is far more readable.

ok tb@

Factor out the legacy stack version checks.

Also check for explicit version numbers, rather than just the major version
value.

ok tb@

Replace SSL_IS_DTLS with SSL_is_dtls().

Garbage collect the now unused SSL_IS_DTLS macro.

ok tb@

Condense and simplify TLS methods.

Historically, OpenSSL has had client and server specific methods - the only
difference between these is that the .ssl_connect or .ssl_accept function
pointer is set to ssl_undefined_function, with the intention of reducing
code size for a statically linked binary that was only a client or server.
These days the difference is minimal or non-existant in many cases and
we can reduce the amount of code and complexity by having single method.

Internally remove all of the client and server specific methods,
simplifying code in the process. The external client/server specific API
remain, however these now return the same thing as TLS_method() does.

ok tb@

Merge d1_{clnt,srvr}.c into ssl_{clnt,srvr}.c

The d1_{clnt,srvr}.c contain a single function each - merge these into the
ssl_{clnt,srvr}.c, renaming them with an ssl3_ prefix for consistency.

ok beck@ tb@

Simplify the cleanup of init_buf via a ssl3_release_init_buffer() function.

ok beck@ inoguchi@ tb@

Simplify SSL method lookups.

There are three places where we call tls1_get_{client,server}_method() and
if that returns NULL, call dtls1_get_{client,server}_method(). Simplify
this by combining the lookup into a single function. While here also use
uint16_t for version types.

ok inoguchi@ millert@

Remove cipher_list_by_id.

When parsing a cipher string, a cipher list is created, before being
duplicated and sorted - the second copy being stored as cipher_list_by_id.
This is done only so that a client can ensure that the cipher selected by
a server is in the cipher list. This is pretty pointless given that most
clients are short-lived and that we already had to iterate over the cipher
list in order to build the client hello. Additionally, any update to the
cipher list requires that cipher_list_by_id also be updated and kept in
sync.

Remove all of this and replace it with a simple linear scan - the overhead
of duplicating and sorting the cipher list likely exceeds that of a simple
linear scan over the cipher list (64 maximum, more typically ~9 or so).

ok beck@ tb@

Improve argument order for the internal tlsext API

Move is_server and msg_type right after the SSL object so that CBS
and CBB and alert come last. This brings these functions more in
line with other internal functions and separates state from data.

requested by jsing

Enable GOST_SIG_FORMAT_RS_LE when verifying certificate signatures.

GOST cipher suites requires that CertVerify signatures be generated in a
special way (see ssl3_send_client_kex_gost(), ssl3_get_cert_verify()).
However, the GOST_SIG_FORMAT_RS_LE flag was not passed in case of TLS 1.2
connections (because they use different code path). Set this flag on
GOST PKEYs.

Diff from Dmitry Baryshkov <dbaryshkov@gmail.com>

Sponsored by ROSA Linux

ok inoguchi@ tb@

Correct downgrade sentinels when a version pinned method is in use.

Previously only the enabled protocol versions were considered, however we
also have to consider the method in use which may be version pinned.

Found the hard way by danj@ with haproxy and force-tlsv12.

ok beck@ inoguchi@ tb@

Replace SSL_PKEY_RSA_ENC/SSL_PKEY_RSA_SIGN with SSL_PKEY_RSA.

Some time prior to SSLeay 0.8.1b, SSL_PKEY_RSA_SIGN got added with the
intention of handling RSA sign only certificates... this incomplete code
had the following comment:

/* check to see if this is a signing only certificate */
/* EAY EAY EAY EAY */

And while the comment was removed in 2005, the incomplete RSA sign-only
handling has remained ever since.

Remove SSL_PKEY_RSA_SIGN and rename SSL_PKEY_RSA_ENC to SSL_PKEY_RSA. While
here also remove the unused SSL_PKEY_DH_RSA.

ok tb@

Use size_t for OCSP response length.

The OCSP response length is currently an integer, which is overloaded with
-1 meaning "unset". Use a size_t for the OCSP response length and infer
unset from the OCSP response being NULL. This makes code more readable,
simpler and less error prone.

ok beck@

Drop a redundant test. It's effectively doing the same test twice
and if the two lengths differed, the later CBS_write_bytes() would
correctly fail anyway.

Discussed with jsing

RFC 8446, section 4.1.3: If a TLSv1.2 client receives a ServerHello for
TLSv1.1 or below, it should check whether the server's random value
contains the magic downgrade protection cookie and in that case abort
the handshake with an illegal parameter alert.

ok inoguchi, jsing

Factor out/rewrite the ECDHE EC point key exchange code.

This reduces replication between the existing TLS client/server and allows
the code to soon be reused for TLSv1.3.

With feedback from inoguchi@ and tb@

ok inoguchi@ tb@

Remove the ssl_get_message function pointer from SSL_METHOD_INTERNAL.

ssl_get_message is essentially a switch between ssl3_get_message and
dtls1_get_message, both only used by the legacy stack. Instead, use
SSL_IS_DTLS() in ssl3_get_message to call the DTLS function when
necessary.

ok beck@ inoguchi@ tb@

Clean up and simplify the client verify code:

- Be consistent with _len naming.
- Use size_t where possible/appropriate.
- Group the CBB code.
- Use EVP_MAX_MD_SIZE consistently, instead of "magic" values.
- Switch GOST to EVP_DigestSign*, making it similar to sigalgs.

ok tb@ a while back.

Defer sigalgs selection until the certificate is known.

Previously the signature algorithm was selected when the TLS extension was
parsed (or the client received a certificate request), however the actual
certificate to be used is not known at this stage. This leads to various
problems, including the selection of a signature algorithm that cannot be
used with the certificate key size (as found by jeremy@ via ruby regress).

Instead, store the signature algorithms list and only select a signature
algorithm when we're ready to do signature generation.

Joint work with beck@.

Rework ssl3_output_cert_chain() to take a CERT_PKEY and consider chains.

We will now include the certificates in the chain in the certificate list,
or use the existing extra_certs if present. Failing that we fall back to
the automatic chain building if not disabled.

This also simplifies the code significantly.

ok beck@ tb@

Revert TLS1_get{,_client}_version simplification because DTLS.

Partially clean up the TLS1_get_{,client}_version macros.

LibreSSL only supports TLSv1.0 and above, hence the checks the macros are
performing are useless. Simplify them to their effective code. Also place
both under #ifndef LIBRESSL_INTERNAL and use the variables directly in our
code, which improves readability.

ok tb@

Rename tls1_handshake_hash*() to tls1_transcript_hash*().

While handshake hash is correct (in as far as it is a hash of handshake
messages), using tls1_transcript_hash*() aligns them with the naming of the
tls1_transcript*() functions. Additionally, the TLSv1.3 specification uses
Transcript-Hash and "transcript hash", which this matches.

ok inoguchi@ tb@

Modify sigalgs extension processing to accomodate TLS 1.3.
- Make a separate sigalgs list for TLS 1.3 including only modern
algorithm choices which we use when the handshake will not negotiate
TLS 1.2.
- Modify the legacy sigalgs for TLS 1.2 to include the RSA PSS algorithms as
mandated by RFC8446 when the handshake will permit negotiation of TLS 1.2
from a 1.3 handshake.
ok jsing@ tb@

revert previous, accidentally contained another diff in addition
to the one I intended to commit

Modify sigalgs extension processing for TLS 1.3.
- Make a separate sigalgs list for TLS 1.3 including only modern
algorithm choices which we use when the handshake will not negotiate
TLS 1.2
- Modify the legacy sigalgs for TLS 1.2 to include the RSA PSS algorithms as
mandated by RFC8446 when the handshake will permit negotiation of TLS 1.2
ok jsing@ tb@

Rename TLS extension handling to use less "hello".

When the TLS extension code was rewritten, TLS extensions could only exist
in ClientHello and ServerHello messages - as such, they were named in pairs
of *_clienthello_{needs,build} which would be called by the client and
*_clienthello_parse. Likewise for *_serverhello_{needs,build} which would
be called by a server and *_serverhello_parse, which would be called by a
client.

Enter TLSv1.3 - TLS extensions can now exist in one of seven messages,
with only certain types being allowed to appear in each, meaning the naming
scheme no longer works. Instead, rename them to indicate the caller rather
than the message type - this effectively means:

clienthello_needs -> client_needs
clienthello_build -> client_build
clienthello_parse -> server_parse
serverhello_needs -> server_needs
serverhello_build -> server_build
serverhello_parse -> client_parse

ok beck@ tb@

Refactor a tangle in ssl3_send_client_verify() into one function for each
type, sigalgs/rsa/ec/gost. Move a few special dances for GOST where they
belong now. This prompted a fix for a long-standing bug with GOST client
certificate authentication where tls1_transcript_data() fails since the
transcript was already freed before. Add a bit of missing error checking
and leave some further cleanup for later.

idea, guidance & ok jsing

Fix DTLS transcript handling for HelloVerifyRequest.

If DTLS sees a HelloVerifyRequest the transcript is reset - the previous
tls1_init_finished_mac() function could be called multiple times and would
discard any existing state. The replacement tls1_transcript_init() is more
strict and fails if a transcript already exists.

Provide an explicit tls1_transcript_reset() function and call it from the
appropriate places. This also lets us make DTLS less of a special snowflake
and call tls1_transcript_init() in the same place as used for TLS.

ok beck@ tb@

Revert previous - DTLSv1 uses MD5+SHA1 for RSA signature verification.

Discussed with beck@

Fix DTLS, because DTLS still remains a special flower, allows regress to pass

revert previous

Fix DTLS. Because the DTLS code is strange. I am really coming around to
joel's line of thinking about it

Unbreak legacy ciphers for prior to 1.1 by setting having a legacy
sigalg for MD5_SHA1 and using it as the non sigalgs default
ok jsing@

Add check function to verify that pkey is usable with a sigalg.
Include check for appropriate RSA key size when used with PSS.
ok tb@

Add support for RSA PSS algorithims being used in sigalgs.
lightly tested, but will need sanity checks and regress test changes
before being added to any sigalgs list for real
ok jsing@ tb@

Convert signatures and verifcation to use the EVP_DigestXXX api
to allow for adding PSS, Nuke the now unneejded guard around the PSS
algorithms in the sigalgs table
ok jsing@ tb@

Stop keeping track of sigalgs by guessing it from digest and pkey,
just keep the sigalg around so we can remember what we actually
decided to use.
ok jsing@

Ensure we free the handshake transcript upon session resumption.

Found the hard way by jmc@

ok tb@

Ensure we only choose sigalgs from our prefernce list, not the whole list
ok jsing@

Reimplement the sigalgs processing code into a new implementation
that will be usable with TLS 1.3 with less eye bleed.
ok jsing@ tb@

Clean up and simplify the handshake transcript code.

This provides a cleaner, simpler and more readable API, with code that uses
a BUF_MEM instead of a BIO.

ok beck@ ("hurry up") and tb@.

Stop pretending that a cert member in a SSL and SSL_CTX can be NULL.

ok beck@ tb@

Ensure the handshake transcript is cleaned up.

Add a check at the completion of the client/server handshake to ensure that
the handshake transcript has been freed. Fix the case where a server asks
the client for a certificate, but it does not have one, resulting in the
handshake transcript being left around post-handshake.

ok bcook@ tb@

Use the newer/more sensible names for EVP_MD_CTX_* functions.

EVP_MD_CTX_create -> EVP_MD_CTX_new
EVP_MD_CTX_destroy -> EVP_MD_CTX_free

This should make the intent more obvious and reduce head scratching during
code reviews.

Raised by tb@

Clean up handshake message start/finish functions.

Now that all handshake messages are created using CBB, remove the non-CBB
ssl3_handshake_msg_start()/ssl3_handshake_msg_finish() functions. Rename
the CBB variants by dropping the _cbb suffix.

ok bcook@ inoguchi@ tb@

Rename and collapse tls12_get_sigandhash_cbb().

Now that all callers of tls12_get_sigandhash() have been converted to CBB,
collapse tls12_get_sigandhash() and tls12_get_sigandhash_cbb() into a
single function. Rename it to tls12_gethashandsig() to be representative
of the actual order of the sigalgs parameters, and perform some other
clean up.

ok inoguchi@ tb@

Convert ssl3_send_client_verify() to CBB.

ok inoguchi@ tb@

Convert ssl3_get_server_key_exchange() to CBS.

ok inoguchi@ tb@

Remove now unused variable, that got left behind from a previous change.

Actually check the return values for EVP_Sign* and EVP_Verify*.

ok bcook@ beck@ tb@

Simplify server key exchange signature verification.

Everything can go through the EVP_Verify* code path.

ok inoguchi@ tb@

Check the return value from DH_size() in ssl3_send_client_kex_dhe().

ok beck@ inoguchi@

Fix a malloc() NULL check in ssl3_send_client_kex_ecdhe_ecp(), by adding
the missing goto. While here also remove a set of unnecessary parentheses.

Nuke SSL3_FLAGS_DELAY_CLIENT_FINISHED and SSL3_FLAGS_POP_BUFFER.

These flags enabled experimental behaviour in the write path, which nothing
uses. Removing this code greatly simplifies ssl3_write().

ok beck@ inoguchi@ sthen@ tb@

Complete the TLS extension rewrite on the client-side.

The RI logic gets pulled up into ssl3_get_server_hello() and
ssl_parse_serverhello_tlsext() gets replaced by tlsext_client_parse(),
which allows a CBS to be passed all the way down.

This also deduplicates the tlsext_client_build() and tlsext_server_build()
code.

ok beck@

Rename ssl3_client_hello() to ssl3_send_client_hello() for consistency.

Convert ssl3_client_hello() to CBB.

As part of this, change ssl_cipher_list_to_bytes() to take a CBB argument,
rather than a pointer/length. Some additional clean up/renames while here.

Based on a diff from doug@

((remove) (some) (unnecessary) (parentheses))

Part of a diff from doug@

Merge dtls1_connect() into ssl3_connect(), removing a large amount of
duplicated code. For now this is essentially adds a diff of the two
functions with 'if (SSL_IS_DTLS(s))' - further clean up and improvement
will follow.

ok inoguchi@

Fix some style/whitespace/indentation issues in ssl3_connect().

Convert TLS signature algorithms extension handling to the new framework.

ok beck@ doug@

Remove NPN support.

NPN was never standardised and the last draft expired in October 2012.
ALPN was standardised in July 2014 and has been supported in LibreSSL
since December 2014. NPN has also been removed from Chromium in May 2016.

TLS clients and servers that try to use/enable NPN will fail gracefully and
fallback to the default protocol, since it will essentially appear that the
otherside does not support NPN. At some point in the future we will
actually remove the NPN related symbols entirely.

ok bcook@ beck@ doug@

Remove support for DSS/DSA, since we removed the cipher suites a while
back.

ok guenther@

Move state from ssl->internal to the handshake structure.
while we are at it, convert SSLerror to use a function
internally, so that we may later allocate the handshake
structure and check for it
ok jsing@

Bring in an SSL_HANDSHAKE structure and commence the great shovelling
ok jsing@, gcc@, regress@

Convert various client key exchange functions to freezero(3). The memory
contents needs to be made inaccessible - this is simpler and less error
prone than the current "if not NULL, explicit_bzero(); free()" dance.

Remove the handshake digests and related code, replacing remaining uses
with the handshake hash. For now tls1_digest_cached_records() is retained
to release the handshake buffer.

ok beck@ inoguchi@

Provide a rolling handshake hash that commences as soon as the cipher
suite has been selected, and convert the final finish MAC to use this
handshake hash.

This is a first step towards cleaning up the current handshake
buffer/digest code.

ok beck@ inoguchi@

Convert various handshake message generation functions to CBB.

ok beck@ inoguchi@

Call ssl3_handshake_write() instead of ssl3_do_write() - this was missed
when ssl3_send_client_certificate() was converted to the standard handshake
functions in r1.150 of s3_clnt.c.

This has no impact on TLS, however it causes the DTLS client to fail if the
server sends a certificate request, since the TLS MAC is calculated on a
non-populated DTLS header.

Issue reported by umokk on github.

Convert ssl3_{get,send}_server_key_exchange() to EVP_md5_sha1().

ok inoguchi@

Stop pretending that MD5 and SHA1 might not exist - rather than locating
"ssl3-md5" and "ssl-sha1", call the EVP_md5() and EVP_sha1() functions
directly.

ok beck@ inoguchi@

Change SSLerror() back to taking two args, with the first one being an SSL *.
Make a table of "function codes" which maps the internal state of the SSL *
to something like a useful name so in a typical error in the connection you
know in what sort of place in the handshake things happened. (instead of
by arcane function name).
Add SSLerrorx() for when we don't have an SSL *
ok jsing@ after us both being prodded by bluhm@ to make it not terrible

Finish the fallout of the SSLerr->SSLerror cleanup to get rid of the ugly
line wraps that resulted

Send the error function codes to rot in the depths of hell where they belong
We leave a single funciton code (0xFFF) to say "SSL_internal" so the public
API will not break, and we replace all internal use of the two argument
SSL_err() with the internal only SSL_error() that only takes a reason code.
ok jsing@

Remove most of SSL3_ENC_METHOD - we can just inline the function calls
and defines since they are the same everywhere.

ok beck@

Rename s3_{both,clnt,pkt_srvr}.c to have an ssl_ prefix since they are no
longer SSLv3 code.

ok beck@

Manually unifdef OPENSSL_NO_ENGINE in ssl_clnt.c

This allows us to simplify ssl_do_client_cert_cb() a bit.

ok jsing

Hide all public symbols in libssl

With the guentherizer 9000

ok tb@

Convert legacy server kex to one-shot sign/verify

This converts ssl3_{get,send}_server_key_exchange() to EVP_DigestVerify()
and EVP_DigestSign(). In order to do this, build the full signed_params
up front and rework the way the key exchange parameters are constructed.
This way we can do the verify and sign steps in one go and at the same
use a more idiomatic approach with CBB/CBS.

with/ok jsing

Easy EVP_Digest{Sign,Verify} conversions for legacy stack

Convert ssl3_send_client_verify_{sigalgs,gost}() to EVP_DigestSign() and
ssl3_get_cert_verify() to EVP_DigestVerify().

ok jsing

spelling fixes; from paul tagliamonte
ok tb

Make internal header file names consistent

Libcrypto currently has a mess of *_lcl.h, *_locl.h, and *_local.h names
used for internal headers. Move all these headers we inherited from
OpenSSL to *_local.h, reserving the name *_internal.h for our own code.
Similarly, move dtls_locl.h and ssl_locl.h to dtls_local and ssl_local.h.
constant_time_locl.h is moved to constant_time.h since it's special.

Adjust all .c files in libcrypto, libssl and regress.

The diff is mechanical with the exception of tls13_quic.c, where
#include <ssl_locl.h> was fixed manually.

discussed with jsing,
no objection bcook

Annotate misuse of EVP_Digest()

The session_id member of SSL_SESSION has 32 bytes for historical
reasons. This precisely accommodates a SHA-256 and is currently
computed using this hash. If the hash function is ever changed,
this will likely overflow. This should be fixed in code. Leave
it at an XXX comment for now.

Pointed out by jsing

Get rid of SSL_CTX_INTERNAL and SSL_INTERNAL.

These are no longer necessary due to SSL_CTX and SSL now being fully
opaque. Merge SSL_CTX_INTERNAL back into SSL_CTX and SSL_INTERNAL back
into SSL.

Prompted by tb@

Move handshake message handling functions from ssl_both.c to client/server.

Currently, ssl_both.c contains several functions that are used by both the
legacy client and legacy server. This interwines the client and server,
making it harder to make progressive changes. While it does deduplicate
some code, it also ends up with code that is conditioned on s->server and
forces the caller to pass in SSL3_ST_* values.

Move these functions from ssl_both.c into ssl_clnt.c and ssl_srvr.c,
renaming as appropriate and removing the s->server conditionals. Also move
the client and server function prototypes from ssl_locl.h into the .c
files, making them static in the process.

ok tb@

Deduplicate peer certificate chain processing code.

Rather than reimplement this in each TLS client and server, deduplicate it
into a single function. Furthermore, rather than dealing with the API
hazard that is SSL_get_peer_cert_chain() in this code, simply produce two
chains - one that has the leaf and one that does not.
SSL_get_peer_cert_chain() can then return the appropriate one.

This also moves the peer cert chain from the SSL_SESSION to the
SSL_HANDSHAKE, which makes more sense since it is not available on
resumption.

ok tb@

Remove redeclaration of ret

When ret was introduced in an outer scope in r1.113, this declaration
wasn't garbage collected.

ok jsing

Simplify certificate list handling code in legacy client.

Tidy up CBS code and remove some unnecessary length checks. Use 'cert' and
'certs' for certificates, rather than 'x' and 'sk'.

ok tb@

Rename uses 'curve' to 'group' and rework tls1 group API.

This reworks various tls1_ curve APIs to indicate success via a boolean
return value and move the output to an out parameter. This makes the
caller code easier and more consistent.

Based on a suggestion by jsing

ok jsing

Add checks to ensure we do not initiate or negotiate handshakes with
versions below the minimum required by the security level.

input & ok jsing

There's tentacles, tentacles everywhere

ok beck jsing

Check the security of DH key shares

ok beck, looks good to jsing

Simplify various CBS_write_bytes() calls

Now that session_id_length is a size_t, we can pass it directly to
CBS_write_bytes() instead of using a temporary variable.

ok jsing

Add missing error check call in ssl3_get_new_session_ticket()

EVP_Digest() can fail, so handle failure appropriately and prepare
switch of session_id_length to a size_t.

ok jsing

Another small readability tweak: compare explicitly against 0 and NULL,
respectively

ok jsing

Add error checking to tls_session_secret_cb() calls

Failure of this undocumented callback was previously silently ignored.
Follow OpenSSL's behavior and throw an internal error (for lack of a
better choice) if the callback failed or if it set the master_key_length
to a negative number.

Unindent the success path and clean up some strange idioms.

ok jsing

Remove incorrect and ungrammattical comment

The fallback to SHA-1 if SHA-256 is disabled fell victim to tedu many
moons ago when this file was still called s3_clnt.c and had no RCS ID.

Bye bye S3I.

S3I has served us well, however now that libssl is fully opaque it is time
to say goodbye. Aside from removing the calloc/free/memset, the rest is
mechanical sed.

ok inoguchi@ tb@

Cleanup/simplify ssl_cert_type()

Remove the X509 argument as it is unused - this was passed so that
ssl_cert_type() can get the public key from the X509 object if the
EVP_PKEY argument is NULL, however this is never the case.

ok tb@

Garbage collect unused ret in ssl3_get_new_session_ticket()

ret is used to bubble up an error from ssl3_get_message() and is unused
otherwise. Zap dead store and silly ret = 1; return ret;

ok inoguchi jsing

In ssl3_get_server_key_exchange() switch a 'goto err' to
'goto fatal_err' so that the illegal_parameter alert is
actually sent in case of X509_get0_pubkey() failure.

ok inoguchi jsing

Remove peer_pkeys from SSL_SESSION.

peer_pkeys comes from some world where peers can send multiple certificates
- in fact, one of each known type. Since we do not live in such a world,
get rid of peer_pkeys and simply use peer_cert instead (in both TLSv1.2
and TLSv1.3, both clients and servers can only send a single leaf
(aka end-entity) certificate).

ok inoguchi@ tb@

Rename 'peer' to 'peer_cert' in SSL_SESSION.

The 'peer' member of SSL_SESSION is the leaf/end-entity certificate
provided by our peer. Rename it since 'peer' on its own is unhelpful.

ok inoguchi@ tb@

Plumb decode errors through key share parsing code.

Distinguish between decode errors and other errors, so that we can send
a SSL_AD_DECODE_ERROR alert when appropriate.

Fixes a tlsfuzzer failure, due to it expecting a decode error alert and
not receiving one.

Prompted by anton@

ok tb@

ssl_check_srvr_ecc_cert_and_alg() only returns 0/1 - test accordingly.

Swap arguments to ssl_check_srvr_ecc_cert_and_alg()

If a libssl function takes an SSL *, it should normally be the first
argument.

Clean up ssl3_{send,get}_client_kex_gost()

Fix leaks, use sizeof() instead of hardcoded sizes, actually check return
codes, explicit_bzero() the premaster secret on the server side and
generally try to kick the GOST kex code into some sort of shape.

ok inoguchi@ tb@

Return 0/1 from ssl3_{send,get}_client_kex_gost()

Like other KEX handling functions, there is no need to return anything
other than failure/success here.

ok inoguchi@ tb@

Remove a comment from Captain Obvious.

Clean up pkey handling in ssl3_get_server_key_exchange()

With TLSv1.2 and earlier, the authentication algorithm used to sign the
ServerKeyExchange message is dependent on the cipher suite in use and has
nothing to do with the key exchange algorithm. As such, check the
authentication algorithm based on the cipher suite in
ssl3_get_server_key_exchange() and handle things accordingly.

ok inoguchi@ tb@

Merge SESS_CERT into SSL_SESSION.

There is no reason for SESS_CERT to exist - remove it and merge its members
into SSL_SESSION for the time being. More clean up to follow.

ok inoguchi@ tb@

Convert legacy TLS client to tls_key_share.

This requires adding DHE support to tls_key_share. In doing so,
tls_key_share_peer_public() has to lose the group argument and gains
an invalid_key argument. The one place that actually needs the group
check is tlsext_keyshare_client_parse(), so add code to do this.

ok inoguchi@ tb@

Return 0 on failure from send/get kex functions in the legacy stack.

In the legacy stack, a message handling function returns -1 for failure,
0 for need more data and 1 for success (although in extra special cases
2 may also be used). However, the various send/get kex functions only
need to indicate success or failure - switch these to return 0 on failure
(rather than -1) and use normal result testing.

This leaves GOST unchanged for now, as that code is special and needs
extra work.

ok inoguchi@ tb@

Use normal result testing for tls1_check_curve().

Refactor ssl3_get_server_kex_ecdhe() to separate parsing and validation.

If we receive something other than a "named curve", send a handshake
failure alert as we're unable to complete the handshake with the given
parameters. If the server responded with a curve that we did not advertise
send an illegal parameter alert.

ok inoguchi@ tb@

Convert ssl_clnt.c to opaque EVP_MD_CTX

ok inoguchi jsing

Move the minimum DHE key size check into ssl_kex_peer_params_dhe()

ok inoguchi@ tb@

Check DH public key in ssl_kex_peer_public_dhe().

Call DH_check_pub_key() after decoding the peer public key - this will be
needed for the server DHE key exchange, but also benefits the client.

ok inoguchi@ tb@

Factor out/rewrite DHE key exchange.

This follows what was done previously for ECDHE EC point key exchange and
will allow for deduplication and further code improvement.

Convert the TLSv1.2 client to use the new DHE key exchange functions.

ok inoguchi@ tb@

Stop reaching into EVP_PKEY in the rest of libssl.

ok inoguchi jsing

libssl: don't reach for pkey->save_type.

For some strange historical reason ECDSA_sign() and ECDSA_verify}() have
a type argument that they ignore. For another strange historical reason,
the type passed to them from libssl is pkey->save_type, which is used to
avoid expensive engine lookups when setting the pkey type... Whatever the
aforementioned reasons were, we can't access pkey->save_type with the
OpenSSL 1.1 API, and this is thus in the way of making EVP_PKEY opaque.
Simply pass in 0 instead.

ok jsing

Fold SSL_SESSION_INTERNAL back into SSL_SESSION.

ok beck@ tb@

Unhandroll X509_up_ref()

ok beck jsing

Provide a way to determine our maximum legacy version.

With the introduction of TLSv1.3, we need the ability to determine our
maximum legacy version and to track our peer's maximum legacy version.
This is needed for both the TLS record layer when using TLSv1.3, plus
it is needed for RSA key exhange in TLS prior to TLSv1.3, where the
maximum legacy version is incorporated in the pre-master secret to
avoid downgrade attacks.

This unbreaks RSA KEX for the TLS client when the non-version specific
method is used with TLSv1.0 or TLSv1.1 (clearly no one does this).

ok tb@

Fold DTLS1_STATE_INTERNAL into DTLS1_STATE.

Now that DTLS1_STATE is opaque, fold DTLS1_STATE_INTERNAL back into
DTLS1_STATE and remove D1I() usage.

ok tb@

Untangle ssl3_get_message() return values.

This function currently has a long return type that may be <= 0 on
error/retry (which is then cast to an int in order to return it up the
stack), or it returns the length of the handshake message (on success).
This obviously means that 0 can be returned for both success and failure,
which is the reason why a separate 'ok' argument has to exist.

Untangle this mess by changing the return value to an int that indicates
success (1) or error/retry (<= 0). The length never needs to actually be
returned as it is already stored in s->internal->init_num (which is where
the return value is read from anyway).

ok tb@

Change tlsext_tick_lifetime_hint to uint32_t.

Now that SSL_SESSION is opaque, change tlsext_tick_lifetime_hint from long
to uint32_t (matching RFC4507), rather than continuing to work around an
inappropriate type choice.

ok tb@

Ensure that a server hello does not have trailing data.

Found by tlsfuzzer.

ok beck@

We need to allow for either a CERTIFICATE or CERTIFICATE_STATUS message
here or we break the handshake with BAD_MESSAGE

ok tb@

RFC 6066 section 8 allows the server MAY choose not send the CertificateStatus
message, even if it has received a "status_request" extension in the client
hello message and has sent a "status_request" extention in the server hello
message. Genua found a site that is this broken. This makes it work.

ok jsing@

Clean up and simplify info and msg callbacks.

The info and msg callbacks result in duplication - both for code that
refers to the function pointers and for the call sites. Avoid this by
providing typedefs for the function pointers and pulling the calling
sequences into their own functions.

ok inoguchi@ tb@

Correct sigalg hash usage when signing content for client verify.

This was inadvertently broken during sigalgs refactoring.

Pull sigalg selection up into ssl3_send_client_verify().

This means that we do sigalg selection for all cases, including those
where are are not sending sigalgs. This is needed in order to track our
signature type in legacy cases.

ok tb@