openssl in 20.04 can't connect to site that was fine in 19.10 and is fine in Chrome and Firefox

[Jonathan Kamens]

openssl in Ubuntu 20.04 (focal) refuses to connect to a web site that openssl in Ubuntu 19.10 (eoan), Chrome, and Firefox are all happy to connect to.

Reproduce with: `curl -v https://www.toodledo.com/'

or: `openssl s_client -connect www.toodledo.com:443`

or: `python3 -c 'import requests; requests.get("https://www.toodledo.com/")'`

or: `wget https://www.toodledo.com/`

These worked in Ubuntu 19.10 and don't work in 20.04.

I've tried all sorts of things to debug this further and I've just run into walls. I hope someone who understands more about this stuff will be able to figure it out.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: openssl 1.1.1d-2ubuntu3
ProcVersionSignature: Ubuntu 5.4.0-14.17-generic 5.4.18
Uname: Linux 5.4.0-14-generic x86_64
ApportVersion: 2.20.11-0ubuntu18
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Tue Feb 25 13:01:22 2020
InstallationDate: Installed on 2019-08-16 (192 days ago)
InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
SourcePackage: openssl
UpgradeStatus: Upgraded to focal on 2020-01-31 (25 days ago)

[Jonathan Kamens (jik)]

Example output:

jik@jik-x1:~$ curl -v https://www.toodledo.com/
* Trying 146.20.52.175:443...
* TCP_NODELAY set
* Connected to www.toodledo.com (146.20.52.175) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, handshake failure (552):
* error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
* Closing connection 0
curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

[Jonathan Kamens (jik)]

I may be misunderstanding something, but as far as I can tell this is not a cipher mismatch problem.

According to https://www.ssllabs.com/ssltest/analyze.html?d=www.toodledo.com, the site supports TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256. According to "openssl ciphers", openssl supports ECDHE-RSA-AES128-GCM-SHA256, which apperas to be the same thing. But `curl -v --ciphers ECDHE-RSA-AES128-GCM-SHA256 --tlsv1.2 --tls-max 1.2 https://www.toodledo.com/` still fails.

[Jonathan Kamens (jik)]

Aha! `curl -v --ciphers 'DEFAULT:@SECLEVEL=1' https://www.toodledo.com/` works but `curl -v --ciphers 'DEFAULT:@SECLEVEL=2' https://www.toodledo.com/` fails.

According to https://www.openssl.org/docs/man1.1.0/man3/SSL_CTX_get_security_level.html, the default security level for the library is 1 if it isn't specified at compile time. Has Canonical made a decision to set a higher security level by default?

Oh, wait, it appears that yes it has. `openssl version -a` says `-DOPENSSL_TLS_SECURITY_LEVEL=2`.

It appears that this was an intentional change? I question the advisability of this, especially since it doesn't appear that there's any way to override it in a configuration file (is there?).

I am not sure it is advisable for command-line tools in the OS to have stricter security level requirements than users' browsers?

[Dimitri John Ledkov (xnox)]

The browsers will require SECLEVEL=2 from April 2020, the change in libraries has already landed.

You can override this via command-line, a system config file, or a local config file + environment variable pointing to it.

On Ubuntu 20.04 LTS:

man SSL_CTX_get_security_level.3ssl

does have:
--
Level 2
On Ubuntu, TLS versions below 1.2 are not permitted.

NOTES
The default security level can be configured when OpenSSL is compiled by setting -DOPENSSL_TLS_SECURITY_LEVEL=level. On Ubuntu, 2 is used.
--

In addition to having TLSv1.2 min, DTLSv1.2 min, bigger keys ie 2048 bit RSA, we are still to land requirement to reject SHA1 signatures on certs.

This is an intentional change.

[Dimitri John Ledkov (xnox)]

About the server:

- It's certificate is valid for 2 years, and Apple has started to rejecting any servers for which certificate is valid for more than 13 months.

- It has a SHA1withRSA certificate in its chain, which will be rejected by all clients soon.

- It supports many weak TLSv1.2 ciphersuites

- It is susceptible to OpenSSL 0-Length attack

Please contact the administrators of said website to secure it immediately & acquire certificate with shorter timespan and no SHA1 signatures.

[Jonathan Kamens (jik)]

Fair enough, I will contact the web site maintainer. However, regarding this:

>You can override this via command-line, a system config file, or a local config file + environment variable pointing to it.
>
>On Ubuntu 20.04 LTS:
>
>man SSL_CTX_get_security_level.3ssl

1) I searched high, low, and sideways, and couldn't find any documentation of how to override this via a config file or environment variable. Regarding command line, I imagine that is going to be different for each program, and I did manage to figure out how to do it for curl, but it took a huge amount of educated effort that the vast majority of users are not going to be able to figure out.

2) I don't know what deb the man pages for the OpenSSL API are in, but whichever one it is, it's not in the package install set that most people get. That man page is not available on my system, for example.

[Jonathan Kamens (jik)]

There is still something wrong here.

The site in question has fixed the issue in response to my query, and SSL Labs now gives it an A grade:

https://www.ssllabs.com/ssltest/analyze.html?d=www.toodledo.com

According to SSL Labs, it supports these two ciphers for TLS 1.2:

TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)

According to openssl ciphers -s -V, both of these are supported by openssl:

$ openssl ciphers -s -V | egrep '0xC0,0x(30|2F)'
          0xC0,0x30 - ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH Au=RSA
Enc=AESGCM(256) Mac=AEAD
          0xC0,0x2F - ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD

And yet I'm still unable to connect to this server through openssl unless I downgrade the security level to 1.

You mentioned there being an SHA1 certificate in the chain, but I don't see one. The certs all seem to be SHA256.

I cannot find any evidence that security level 2 blocks the use of certificates with lifetimes of more than a year. Is that an undocumented "feature" of security level 2?

[Dimitri John Ledkov (xnox)]

Lifetimes of more than a year is only implemented by Apple on their products. Longevity of the certifications do not matter on Ubuntu.

[Dimitri John Ledkov (xnox)]

So, in their chain of certs that they present there is still an RSA-SHA1 certificate. It shouldn't affect validation, as the other certs in the chain are sufficient (for example gnutls-cli toodledo.com connects fine) but it does trip up openssl:

- Certificate[3] info:
 - subject `OU=Go Daddy Class 2 Certification Authority,O=The Go Daddy Group\, Inc.,C=US', issuer `OU=Go Daddy Class 2 Certification Authority,O=The Go Daddy Group\, Inc.,C=US', serial 0x00, RSA key 2048 bits, signed using RSA-SHA1 (broken!), activated `2004-06-29 17:06:20 UTC', expires `2034-06-29 17:06:20 UTC', pin-sha256="VjLZe/p3W/PJnd6lL8JVNBCGQBZynFLdZSTIqcO0SJ8="

Now, they could remove that cert from the chain that their server uses. But also they should not need to do this and openssl should just work.

[Dimitri John Ledkov (xnox)]

I've now opened upstream issues about this on the OpenSSL bug tracker.

However, todoodle.com could refresh their certificate chain with an up to date G2 godaddy SSL certificate chain, instead of the one that ends on a CA Root certificate which is no longer trustworthy.

[Gordon Lack (gordon-lack)]

>> You can override this via command-line, a system config file, or a local config file + environment variable pointing to it.

Some hints as to how to achieve that "local config file + environment variable" would be extremely useful.
I've tried it and got nowhere, although I know that setting SECLEVEL=1 would fix my immediate problem, as I can set it on a command line for openssl s_client.

See: https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1878519

Forcing a minimum SECLEVEL of 2 by default is fine, but there has to be some-way of letting users reduce this when they are talking to external services, or providing services for external clients that cannot, for some reason,change at the moment.

[Dr. Uwe Meyer-Gruhl (meyergru)]

This bug affects me too, with a client certificate that now "magically" does not match the requirements.

Ironically, the error message says only:

OpenSSL error error:140AB18E:SSL routines:SSL_CTX_use_certificate:ca md too weak, (no key found, wrong pass phrase, or wrong file format?)

although there was no MD5 signature involved at all. So, even when you know that with OpenSSL 1.1, an "SSL security level" has been introduced, and that Ubuntu has set that level to 2, it is hard to find that it deprecates SHA1 now (see https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_security_level.html).

Thus, even for more knowledgable people than me this is a major hassle to find and/or fix. I wonder why Ubuntu has chosen to raise the level that high considering that the documentation page contains a clear warning indication:

"WARNING at this time setting the security level higher than 1 for general internet use is likely to cause considerable interoperability issues and is not recommended. This is because the SHA1 algorithm is very widely used in certificates and will be rejected at levels higher than 1 because it only offers 80 bits of security."

I think that this is an extremely unwise choice for an OS to make.

That being said, here is the fix (also hard to find):

In /etc/ssl/openssl.cnf, add this line before the start of the file:

 openssl_conf = default_conf

At the end of the file, add these lines:

 [default_conf]
 ssl_conf = ssl_sect

 [ssl_sect]
 system_default = system_default_sect

 [system_default_sect]
 CipherString = DEFAULT:@SECLEVEL=1

This will bring down the SSL security level to the former level of 1.

[matsonfamily (david-matsonfamily)]

Also affects me with many HTTPS sites, such as teams.microsoft.com, but haven't been able to post a bug here because SSL or whatever isn't working. I imagine many can't post that it's broken, but tons of websites aren't working in Firefox or chrome or wget for me.

[matsonfamily (david-matsonfamily)]

the openssl.cnf fix doesn't really work... allows me to connect to sites, but when loading their pages, it takes forever. unusable. Also tried SECLEVEL=0... same.

[Adrien Nader (adrien)]

Looking at the bug report opened upstream ( https://github.com/openssl/openssl/issues/11236 ), this is considered a bug on the server side and I'm inclined to follow openssl upstream on this.

Moreover, I've tried all the tests provided in this bug and all have succeeded.

I'll mark the bug as Invalid.

[Jonathan Kamens (jik)]

I mean, the bug is three years old and presumably by now the vast majority of web sites have renewed their certificates and the new ones are compatible with SECLEVEL=2 since all of the signing authorities stopped issuing incompatible ones years ago, so it's kind of moot at this point.