libnetapi: remove support for old OpenSSL versions, prepare for OpenSSL 3

- Remove backported function that was needed only for old OpenSSL versions
- Add compile time options to declare which version of OpenSSL API we
want to use

This should make it possible to build with OpenSSL 3 while keeping the
old APIs available. Once the migration to OpenSSL 3 is done, we can bump
the OPENSSL_API_COMPAT value to get compile time errors when using APIs
that are deprecated in OpenSSL3, and rewrite all the code that needs
changes.

Change-Id: I606633739ed12f9698a3013989025b68478352fc
Reviewed-on: https://review.haiku-os.org/c/haiku/+/6484
Tested-by: Commit checker robot <no-reply+buildbot@haiku-os.org>
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

BSecureSocket: Treat SSL-zero-return as B_IO_ERROR

Somehow B_CANCELED doesn't seem to convey the correct
meaning. Using B_IO_ERROR will also fit to the recent
changes in the Package Kit and will trigger a re-try of
package downloads.

Change-Id: I58c4d4faa705e6519e0ff9ec7c4654a6151e5486
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3635
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

BSecureSocket: fix read error handling

The current documentation
(https://www.openssl.org/docs/man1.1.1/man3/SSL_read.html) says that
SSL_read returning 0 should be handled as an error. So, let's do that.

Change-Id: I6781826ea700b6e597fd2d444fd96a1a5270c6cf
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3620
Reviewed-by: Stephan Aßmus <superstippi@gmx.de>

BSecureSocket: use openssl's default paths for certificates.

It is confusing to use different sets of certificates for BSecureSocket
based apps and ones using openssl directly. So, use the defaults in
BSecureSocket.

OpenSSL was modified to keep the user certificates in non-packaged so
this does not change the behavior for native apps (needs openssl 1.1.1g-3)

Change-Id: Ic398eec5efa9d036c3b810b7a3bb7142bdeb2d46
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3394
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

TLS: add user cert dir

BSecureSocket currently only checks for trust anchors in the CA file,
this change will add the openssl certs/ directory as a trust anchor dir.
this matches the behaviour openssl has on the commandline and allows
users to install their own trust anchors to verify against for TLS

Change-Id: I9db5c3f3b063607e092dded3d5b141dba340a8e2
Reviewed-on: https://review.haiku-os.org/c/haiku/+/3207
Reviewed-by: Fredrik Holmqvist <fredrik.holmqvist@gmail.com>

SecureSocket: Handle interrupted reads and writes

If a system call performed by SSL_read is interrupted by a signal, it
seems to set its error to SSL_ERROR_WANT_READ. This triggers logic
added in hrev53853 which assumes the caller is doing async reads and
returns B_WOULD_BLOCK.

This breaks uses of BSecureSocket that do blocking reads.

* Detect interrupted signal by checking for EINTR in errno.
* Adding this retry loop to BScureSocket::Write as well since it can
have the same problem.

Resolves issue #15853.

Change-Id: I8198a8496fa3a2ccee00bda87375a482a0d4ba3d
Reviewed-on: https://review.haiku-os.org/c/haiku/+/2825
Reviewed-by: Adrien Destugues <pulkomandy@gmail.com>

libnetapi: Adapt to OpenSSL 1.1 API changes.

Surprisingly easy: X509 was made opaque, and SSL_CTX_set_ecdh_auto
was removed (it is now the default internally.)

SecureSocket: fix non-blocking reads

OpenSSL says we should retry when a non-blocking read finds no data is
pending. But in that case we should not retry immediately, because the
operation should be non-blocking.

Build fix.

BSecureSocket: retry reads after interrupted syscalls.

SSL_AUTO_RETRY does not cover this case (it only covers SSL errors, not
underlying socket ones), so we still need to retry reads manually here.

Fixes #14638.

BSecureSocket: Don't continue with an untrusted certificate by default.

It has been multiple years since this comment was written; all relevant
apps have added their own hooks around this, so we should now be
"secure by default."

Also spotted by a random Hacker News commenter.

BSecureSocket: Pass the hostname to the X509 layer to validate it.

Now SSL certificates with the wrong hostname actually fail to validate.
While I'm at it, remove the usage of BString and just check [0] directly.

Spotted by a random commenter on Hacker News.

BSecureSocket: cleanup, no functional changes

Change-Id: I3355067efe4c5d71f8656d244e17bb11175600eb
Reviewed-on: https://review.haiku-os.org/c/910
Reviewed-by: waddlesplash <waddlesplash@gmail.com>

BSecureSocket: re-enable RSA cipher suite

my.justenergy.com allows only RSA, so we can't connect there without
this. Other websites may have a similar problem.

Also improve the handling of the error, as it was giving a generic
"general system error" to the user.

Fixes #13975.

BSecureSocket: use opt-out to select ciphersuites.

We just want to disable known broken ones, rather than opt-in for
working ones. This keeps the list of authentications shorter and is more
future proof.

libbnetapi: style fixes only.

Allow dumping SSL session keys on TRACE_SESSION_KEY

This dumps SSL session keys to a log file specified by the
SSLKEYLOGFILE environment variable.

This permits decrypting SSL trafic in wireshark with a tcpdump
capture for example.
cf.
https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSS/Key_Log_Format
https://security.stackexchange.com/questions/35639/decrypting-tls-in-wireshark-when-using-dhe-rsa-ciphersuites/42350#42350
https://wiki.wireshark.org/SSL#Using_the_.28Pre.29-Master-Secret
https://jimshaver.net/2015/02/11/decrypting-tls-browser-traffic-with-wireshark-the-easy-way/

Looks like we miss the required function from SSL...
SSL_SESSION_print_keylog is only in 1.1.0.
Also added dumping of client_random as it's required by wireshark.

SecureSocket: add code to trace SSL events.

Under a #define TRACE_SSL, should you need it.

Also load error strings when initializing the SSL context, so we get
human readable errors from SSL (also in the ser reported ones).

BSecureSocket: Let OpenSSL choose the most appropriate D-H curve.

BSecureSocket: Print SSL error message as well as the error code.

Helps with debugging failed SSL connections.

BSecureSocket: Explicitly set the list of ciphersuites.

OpenSSL's default cipherlist has a lot of spurious and arguably insecure
suites left in it for compatibility reasons. We have no need for all that,
so let's just use the suites Firefox/Chrome do.

BSecureSocket: Disable SSL/TLS compression to prevent the CRIME attack.

See wikipedia:CRIME (Compression-Rate Info-leak Made Easy) for details.

Add support for TLS SNI

Signed-off-by: Augustin Cavalier <waddlesplash@gmail.com>

libbnetapi: Extend socket classes.

B{Abstract,Datagram,Secure}Socket:
- Add functionality to listen for and accept new connections, thus allowing
one to use the socket classes for server functionality as well.

BSecureSocket:
- Adjust to take into account differences between how SSL needs to be called
when accepting an incoming connection vs initiating an outbound one.
The handshake on the accepted connection stills fails for unknown reasons
at the moment though.

Note that these changes break the ABI, and thus any packages making use of
them directly will need a rebuild.

SecureSocket: enable auto retry on connexion re-negociation

This is required to talk with the proxy in Thalys trains.
Now I'm online and I can push this!

SecureSocket: stubs for no-openssl version

Implement CONNECT pass-through for HTTPS proxy

* When using a proxy, HTTPS connexion must still go directly to the
target website. The proxy can then act as a TCP stream relay and just
transmit the raw SSL stream between the client and website.
* For this, we ask the proxy sending an HTTP request with the CONNECT
method. If the proxy supports this, we can then send anything as the
payload and it will be forwarded.
* Untested, as the network here in Dusseldorf doesn't let me use a
proxy.

ticket : #10973

BSecureSocket: use "trusted first" validation algorithm.

* This makes it work with the new ca_root_certificates package.

Fixes #12004.

BSecureSocket: prepared to do SSL error conversion.

* For now, just print them to stderr; we don't really have matching error
codes yet AFAICS.

SecureSocket: fix build without SSL.

Add more features in BCertificate class

* Make it possible to extract more useful data from the certificate
* Also get the OpenSSL error message when a certificate can't be
validated. Send it to the verification failure callback so it can be
shown to the user.

SecureSocket: disable SSL versions 2 and 3

There are known vulnerabilities in those. Everyone should be using TLS
by now.

Style fix: add parameter name

* As pointed out by Axel.

Fix double-free crash in BSecureSocket when cert. verification fails

* BSecureSocket::CertificateVerificationFailed() took a BCertificate
instance by value as parameter.
BCertificate deletes internal data in its destructor. Passing an
object by value creates a copy, so the copy attempted to delete
the internal data again during its destruction.
This caused mail_daemon to crash here when it came across a failed
certificate.

* Fix: pass BCertificate object as reference.

Style fixes and allocation checks

Yet anoter noSSL build fix.

Add some missing std::nothrow

... and allocation failure checks.

Fix concurrency issues in BSecureSocket

* Use pthread_once to initialize the SSL context once, in a thread-safe
way.
* Do not delete the BIO immediately when closing a connexion, instead
delay this to the destructor. This makes sure the protocol loop is done
running when we do that.
* Instead of creating a new BIO when we reconnect an already used
connection, create the BIO upfront, and reuse it with the new file
descriptor.
* Fix a memory leak: the SSL struct from OpenSSL was never freed, only
the BIO was.

Fixes #10414.

Style fixes, build fix with OpenSSL disabled.

Fix build.

SecureSocket: add some certificate support

* Instead of creating an OpenSSL context ofor each socket, use a global
one and initialize it lazily when the first SecureSocket is created
* Load the certificates from our certificate list so SSL certificates
sent by servers can be validated.
* Add a callback for signalling that certificate validation failed, the
default implementation proceeds with the connection anyway (to keep the
old behavior).
* Introduce BCertificate class, that provides some information about a
certificate. Currently it's only used by the callback mentionned above,
but it will be possible to get the leaf certificate for the connection
after it's established.

Review of the API and implementation is welcome, before I start making
use of this in HttpRequest and WebKit to allow the user to accept new
certificates.

SecureSocket: avoid crash on close

Deleting the BIO while it's still waiting on a read() in another thread
will lead to a crash when the socket is eventually closed. Close the
socket first, so the read() is unlocked, then safely delete the BIO.

BSecureSocket: Read and Write should return 0 for closed sockets.

Added experimental version of a Socket API with SSL support.

* Each class has a Socket() method to retrieve the underlaying file descriptor
to be able to do the more advanced stuff, if necessary.
* A server socket is yet missing, but the rest is pretty much covered.

BSecureSocket: prepared to do SSL error conversion.

* For now, just print them to stderr; we don't really have matching error
codes yet AFAICS.

SecureSocket: fix build without SSL.

Add more features in BCertificate class

* Make it possible to extract more useful data from the certificate
* Also get the OpenSSL error message when a certificate can't be
validated. Send it to the verification failure callback so it can be
shown to the user.

SecureSocket: disable SSL versions 2 and 3

There are known vulnerabilities in those. Everyone should be using TLS
by now.

Style fix: add parameter name

* As pointed out by Axel.

Fix double-free crash in BSecureSocket when cert. verification fails

* BSecureSocket::CertificateVerificationFailed() took a BCertificate
instance by value as parameter.
BCertificate deletes internal data in its destructor. Passing an
object by value creates a copy, so the copy attempted to delete
the internal data again during its destruction.
This caused mail_daemon to crash here when it came across a failed
certificate.

* Fix: pass BCertificate object as reference.

Style fixes and allocation checks

Yet anoter noSSL build fix.

Add some missing std::nothrow

... and allocation failure checks.

Fix concurrency issues in BSecureSocket

* Use pthread_once to initialize the SSL context once, in a thread-safe
way.
* Do not delete the BIO immediately when closing a connexion, instead
delay this to the destructor. This makes sure the protocol loop is done
running when we do that.
* Instead of creating a new BIO when we reconnect an already used
connection, create the BIO upfront, and reuse it with the new file
descriptor.
* Fix a memory leak: the SSL struct from OpenSSL was never freed, only
the BIO was.

Fixes #10414.

Style fixes, build fix with OpenSSL disabled.

Fix build.

SecureSocket: add some certificate support

* Instead of creating an OpenSSL context ofor each socket, use a global
one and initialize it lazily when the first SecureSocket is created
* Load the certificates from our certificate list so SSL certificates
sent by servers can be validated.
* Add a callback for signalling that certificate validation failed, the
default implementation proceeds with the connection anyway (to keep the
old behavior).
* Introduce BCertificate class, that provides some information about a
certificate. Currently it's only used by the callback mentionned above,
but it will be possible to get the leaf certificate for the connection
after it's established.

Review of the API and implementation is welcome, before I start making
use of this in HttpRequest and WebKit to allow the user to accept new
certificates.

SecureSocket: avoid crash on close

Deleting the BIO while it's still waiting on a read() in another thread
will lead to a crash when the socket is eventually closed. Close the
socket first, so the read() is unlocked, then safely delete the BIO.

BSecureSocket: Read and Write should return 0 for closed sockets.

Added experimental version of a Socket API with SSL support.

* Each class has a Socket() method to retrieve the underlaying file descriptor
to be able to do the more advanced stuff, if necessary.
* A server socket is yet missing, but the rest is pretty much covered.