Python's test_ssl fails starting from Ubuntu 20.04

[Bug Reporter]

Please take a look at https://bugs.python.org/issue41561. Developers who work on Python think that the issue is due to a change in Ubuntu 20.04 that is best described by https://bugs.python.org/issue41561#msg378089:

"It sounds like a Debian/Ubuntu patch is breaking an assumption. Did somebody report the bug with Debian/Ubuntu maintainers of OpenSSL already? Fedora also configures OpenSSL with minimum protocol version of TLS 1.2. The distribution does it in a slightly different way that makes the restriction discoverable and that is compatible with Python's test suite."

[Bug Reporter (bugsrep)]

Sorry, this is the correct link: https://bugs.python.org/issue41561#msg378103

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Christian Heimes (heimes)]

FYI, yesterday GitHub Actions moved "ubuntu-latest" to 20.04. This caused breakage in CPython's CI, https://bugs.python.org/issue43382

[Dimitri John Ledkov (xnox)]

Fedora & Debian & Ubuntu implement openssl differently.

In Ubuntu, as an Ubuntu-specific patch, we set default security level to 2, and prohibit protocols lower than TLSv1.2 / DTLSv1.2.

This is documented in the Ubuntu manpages for OpenSSL

http://manpages.ubuntu.com/manpages/hirsute/en/man3/SSL_CTX_set_security_level.3ssl.html

"""
The default security level can be configured when OpenSSL is compiled by setting -DOPENSSL_TLS_SECURITY_LEVEL=level. On Ubuntu, 2 is used.

Level 2
   Security level set to 112 bits of security. As a result RSA, DSA and DH keys shorter
   than 2048 bits and ECC keys shorter than 224 bits are prohibited. In addition to the
   level 1 exclusions any cipher suite using RC4 is also prohibited. On Ubuntu, TLS
   versions below 1.2 are not permitted. Compression is disabled.
"""

This is the only way that we have able to configure minimum key sizes, protocol versions for both TLS and DTLS without making any openssl cnf changes, whilst remaining compatible with both openssl cnf from 1.0.2x, 1.1.0x and 1.1.1x series. As min/max API calls are not available across all openssl series and software that allows to configure openssl cipherstrings but not min/max versions.

If you need access to (D)TLS below 1.2 or weak cryptography you can use openssl 1.1.1 API to set_security level to 1. Or you can set CipherString to DEFAULT@SECLEVEL=1. Without modifying the software at all, libssl can be configured via envrionment variables too.

I.e. exporting

export OPENSSL_CONF=`pwd`/openssl.cnf
cat openssl.cnf
openssl_conf = default_conf
[default_conf]
ssl_conf = ssl_sect
[ssl_sect]
system_default = system_default_sect
[system_default_sect]
CipherString = DEFAULT@SECLEVEL=1

Note that this openssl.cnf is compatible with _any_ openssl series.

In debian, they set min versions of TLS communication only, which breaks with openssl 1.0.2x series as it fails to parse those settings. That was unacceptable for Ubuntu.

I don't know how Fedora implements this, I guess I should take a look.

It would be nice for OpenSSL upstream to provide a standard configure time option to set these things in a consistent manner, as at the moment each distribution has to invent their own way of doing this. My proposals to bump minimum protocol versions to TLSv1.2 in OpenSSL 3.0.0 for the time being got rejected, as it is deemed too soon.

In Ubuntu, we also configure GnuTLS with similar parameters, the override mechanism there is different see https://discourse.ubuntu.com/t/default-to-tls-v1-2-in-all-tls-libraries-in-20-04-lts/12464/8 for both OpenSSL and GnuTLS details.

I'm not sure what is expected from this bug report. Ubuntu changes are documented and publicized and are trivial to find. Were you expecting to find this documentation somewhere else? Where did you look? I am happy to add more documentation in more places, or change the implementation.

What does Fedora do? And is it portable to distributions that do not use the crypto-policies package to maintain configs?

[Christian Heimes (heimes)]

This is really just a problem with Ubuntu 20.04 and 20.10 (and maybe older/newer releases). The same tests are passing fine on latest Debian testing as well as Fedora. Debian testing and Fedora use a crypto policy that raises security level to 2 and disallows TLS 1.0 and 1.1. Python's test suite introspects OpenSSL settings and skips tests of disabled TLS versions.

There seems to be a major difference between Debian and Ubuntu's downstream patches of OpenSSL. Ubuntu's patch causes two problems

1) Python cannot figure out that TLS 1.0 and 1.1 are disabled on Ubuntu. The same code works fine on Debian and Fedora.
2) With some configuration, OpenSSL's SSL_do_handshake() function fails with an "internal error" message (SSL_AD_INTERNAL_ERROR / TLS1_AD_INTERNAL_ERROR) somewhere in its internal state machine.

I suggest that you involve Kurt and look at the difference between Debian's downstream patch and Ubuntu's downstream patch.

[Christian Heimes (heimes)]

PS: Ultimately it is your call how you want to handle the issue. You can either treat it as a bug in Ubuntu's downstream patch or close the issue as WONTFIX. In the latter case I'll add some hacks to our test suite and update our documentation that some legacy TLS features are not working on Ubuntu. I would appreciate if you fix the underlying issue, though. An internal error in the TLS state machine sounds troubling.

[Christian Heimes (heimes)]

CC Kurt Roeckx

Kurt, could you please take a look? An Ubuntu's downstream patch for OpenSSL causes TLS1_AD_INTERNAL_ERROR internal error in OpenSSL's state machine and breaks CPython's test suite. Debian is not affected. It's the same problem I mentioned in my gist a couple of months ago.

[Dimitri John Ledkov (xnox)]

But Debian & Fedora implementation are buggy, because they break 1.0.2x users & they do not prohibit DTLSv1.1 whilst enforcing TLSv1.2+.

So although Debian & Fedora look "nice" they are security vulnerable configurations.

I can set min_version to TLSv1.2, in addition to security level 2 but that will not make current stable test_ssl test suite pass, as it will require not only changing min_level but also setting security level to 1.

I do not see a way to make things secure, for both TLS and DTLS, and discoverable and not pain to use. Because when default context is created it is not known if TLS or DTLS will be used, and the enums for TLS & DTLS are not compatible with each other.

Ultimately it is deficiency in the OpenSSL APIs because it is impossible to know what is or isn't allowed by a given client OpenSSL context, against which server context, and vice versa. Even when enums are available, and one sets them as appropriate min/max. There are no inspection APIs available into the security levels. For example, it impossible to query if ones client certificate is suitable for a given security level, apart from trying to establish the connection.

Re Kurt => i have spoken to Kurt about this, he is aware that Debian's implementation is buggy and he does prefer Ubuntu's one, however Ubuntu's one is not without drawbacks either. I.e. at the moment in Debian people simply choose to not install openssl package and thus end up operating without public certificates and with TLS v1.1/v1.0 enabled, meaning the system is insecure by accident against the intentions. Especially if one tries to be secure, and use private CA certificates only.

"""
2) With some configuration, OpenSSL's SSL_do_handshake() function fails with an "internal error" message (SSL_AD_INTERNAL_ERROR / TLS1_AD_INTERNAL_ERROR) somewhere in its internal state machine.
"""

I'm not sure how this is related to anything of the above, can you please open a new bug report with details? crashes in handshake are typically specific to the connection type, context on both client & server, and well bugs. The one thing that I know failing badly, is when server has redundant tls certificates in its chain that are considered insecured (i.e. old CA cross-signed new CA). And OpenSSL client currently rejects establishing the connection, despite the server chain having alternative paths of certs that are secure throughout.

[Marc Deslauriers (mdeslaur)]

The python2.7 security updates that will be released today will fix this issue.

[Marc Deslauriers (mdeslaur)]

Please disregard comment #9, this is a different issue.

[Christian Heimes (heimes)]

(Short answer, I'm in meetings for the rest of the day)

Python's ssl module doesn't support TLS over UDP. DTLS is not an issue for CPython. I have limited experience with DTLS and cannot contribute much to that part of the discussion. Python's test suite uses its own set of certificates for testing. We don't rely on system's CA store. All certs and DH parameters are chosen to work on security level 2. (I've been bitten by old settings when I made Python core and tests FIPS compliant.)

Does Ubuntu's OpenSSL 1.0.x compat package use the same config file as OpenSSL 1.1.1? I can see how that can cause trouble. Fedora's compat-openssl10 package works around incompatible config files by using openssl10.cnf instead of openssl.cnf with patch https://src.fedoraproject.org/rpms/compat-openssl10/blob/f33/f/openssl-1.0.2o-conf-10.patch.

If you get SSL_CTX_get_min_proto_version() to return TLS1_2_VERSION, then we can detect the minimum version in Python. The macro currently returns "0" on Ubuntu. Python then falls back to "#if defined(TLS1_VERSION) && !defined(OPENSSL_NO_TLS1)" to detect if TLS 1.0 is available. https://github.com/python/cpython/blob/b04f1cb9df7ad93366ef0ef7d8088effc576c5ae/Lib/test/test_ssl.py#L155-L210

>>> import ssl
>>> ctx = ssl.create_default_context()
>>> ctx.minimum_version
<TLSVersion.MINIMUM_SUPPORTED: -2>

A reproducer for the "internal error" during handshake is:

    def test_min_max_version_tlsv1_1(self):
        client_context, server_context, hostname = testing_context()
        # client 1.0 to 1.2, server 1.0 to 1.1
        client_context.minimum_version = ssl.TLSVersion.TLSv1
        client_context.maximum_version = ssl.TLSVersion.TLSv1_2
        server_context.minimum_version = ssl.TLSVersion.TLSv1
        server_context.maximum_version = ssl.TLSVersion.TLSv1_1

        with ThreadedEchoServer(context=server_context) as server:
            with client_context.wrap_socket(socket.socket(),
                                            server_hostname=hostname) as s:
                s.connect((HOST, server.port))
                self.assertEqual(s.version(), 'TLSv1.1')

I'll try to find some time to create a new report.

[Christian Heimes (heimes)]

> I'm not sure how this is related to anything of the above, can you please open a new bug report with details?

It looks like both issues are caused by the same patch. I have filed https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1917625 for the internal error message and uploaded a Dockerfiles with a reproducer.

[Kurt Roeckx (kurt-roeckx)]

I was expecting SSL_CTX_get_min_proto_version() to return the default value (TLS1_2_VERSION). It's currently documented that 0 means the lowest supported by the library. If it returns 0 and the library supports TLS 1.0, it should be able to negotiate TLS 1.0.

On reflection, I'm not sure that for the get function 0 is ever a good value to return, it should always return a version.

[Kurt Roeckx (kurt-roeckx)]

My understanding of things is that Ubuntu does this:
- Set the default security level to 2 (at compile time)
- Disable TLS 1.0 and 1.1 at security level 2, only keeping TLS 1.2 by default

This is what Debian does:
- Set the default security level to 2 (using a config file)
- Set the minimum version to TLS 1.2 (using a config file)

To be able to use TLS 1.0 on Ubuntu you need to:
- Change the security level to 1

To be able to use TLS 1.0 on Debian you need to:
- Set the minimum allowed version to TLS 1.0

My understanding of the issue is that python doesn't know which TLS version can be negotiated, and uses or overrides the minimum TLS version which happens to currently work on Debian. But for the test suite it should also override the security level.

Note that in upstream OpenSSL 3.0, TLS 1.0 and 1.1 no longer meet the requirements for security level 1, if you want to use TLS 1.0 in the test suite you need to set the security level to 0.

[Christian Heimes (heimes)]

Actually, I don't want to enable TLS 1.0 and 1.1 when these versions are disabled by crypto policy or openssl.cnf. It's totally ok that these versions are disabled and don't work any more! I just need a reliable way to *detect* that the versions are disabled at runtime.

The feature detection logic boils down to this pseudo code:

    min_version = SSL_CTX_get_min_proto_version(ctx)
    if not defined(TLS1_VERSION) or defined(OPENSSL_NO_TLS1):
        return False
    elif min_version == 0 or min_version >= TLS1_VERSION:
        return True
    else:
        return False

On Debian, SSL_CTX_get_min_proto_version() returns TLS1_2_VERSION and Python considers TLS 1.0 and 1.1 as disabled by system policy. All test cases for TLS 1.0 and TLS 1.1 are skipped.

Ubuntu has TLS1_VERSION defined, OPENSSL_NO_TLS1 not defined, and SSL_CTX_get_min_proto_version() returns 0. So Python assumes that TLS 1.0 is compiled in, supported, and enabled. But it's actually disabled and tests are failing.

I completely agree with your comment 13: 0 is not a good return value. Could you modify Ubuntu's patch so that SSL_CTX_get_min_proto_version(ctx) returns TLS1_2_VERSION and SSL_CTX_set_min_proto_version(ctx, TLS1_VERSION) returns an error? This would fix Python's problem.

[Kurt Roeckx (kurt-roeckx)]

There are 3 things that can possibly returned by such a function:
1) The value that's set as minimum/maximum
2) The minimum and maximum version that's supported by the library, not depending on settings
3) The minimum and maximum version that's supported by the library, depending on current settings

The function now does 1), and you seem to want 3). And I'm not sure it's a good idea to change the function, I would prefer a new function.

[Christian Heimes (heimes)]

In upstream OpenSSL, (3) is the same as (1) for a pristine SSL_CTX_new(TLS_method()) context.

As far as I can see, the Ubuntu patch effectively sets the minimum version to TLS 1.2 and prevents users from setting TLS 1.0 and 1.1. I propose that the patch also changes the value of minimum protocol on the CTX, so (1) reports TLS1_2_VERSION as current value.

OpenSSL upstream is not going to accept a new function into 1.1.1 LTS branch.

[Kurt Roeckx (kurt-roeckx)]

2) and 3) would never return 0, which is what the upstream OpenSSL version returns now.

2) would make it return TLS1_VERSION for the minimum and TLS1_3_VERSION for the maximum with default build options. If you enable SSlv3 support at compile time, the minimum would return SSL3_VERSION. Note that there is a TLS_MAX_VERSION define that's equal to TLS1_3_VERSION, but no such define for the minimum.

1) would just return what is set, which is 0 upstream and for Ubuntu for both minimum and maximum, where 0 is defined as no limit set. Debian sets the minimum to TLS1_2_VERSION, and so returns that value. But maybe if no limit is set (and so the value is 0), it can instead return the minimum/maximum version that's supported at compile time.

3) Would have as effect that upstream in 1.1.1 returns TLS1_VERSION and TLS1_3_VERSION, but TLS1_2_VERSION and TLS1_3_VERSION in 3.0. For Ubuntu and Debian it would return TLS1_2_VERSION and TLS1_3_VERISON, since that is what they do using a different method.

Setting the default minimum to TLS1_2_VERSION (at compile time) will clearly fix your problem. But I think you're going to run in the same problem with 3.0, and we should probably add a new API in 3.0 for it.

[Marc Deslauriers (mdeslaur)]

I've read through this bug and I don't see a good way forward with a solution here. OpenSSL 1.1.1 doesn't provide the exact API that is required to solve it, which would probably be 3) as suggested above, but I don't think Ubuntu should change the meaning of the value returned by that API.

Ubuntu is returning the value that is expected of the upstream 1.1.1 code when security levels are used.

While Ubuntu, Debian, and Fedora use different approaches to achieve a similar goal, and Ubuntu did modify the meaning of security level 2 to restrict it further than the upstream default, the fact remains that building OpenSSL with a higher default security level results in a situation where there seems to be no good API to request what the minimum TLS version is.

It may appear reasonable that a simple query to the current security level could have given a hardcoded indication of what is available. Unfortunately, that assumption doesn't stand the test of time as it is no longer valid once the security levels have been modified to align with the current best practices, either in a new OpenSSL release, or by a distro.

This is a less than ideal situation, but I think the only way forward currently available when obsolete TLS versions need to be used is to set the minimum security level accepted by OpenSSL.

We should work with upstream OpenSSL to make sure 3.0 provides the right APIs, documentation, and guidance that are required to handle changing defaults like this more elegantly in the future.

[Dimitri John Ledkov (xnox)]

On SSLcontext, security callback has prototype

/* Security callback */
    int (*sec_cb) (const SSL *s, const SSL_CTX *ctx, int op, int bits, int nid,
                   void *other, void *ex);

if one calls that function, with context passed in, "op" set to SSL_SECOP_VERSION, "bits" set to zero, "nid" set to protocol version, other set to NULL, and ex set to null => then the security callback will tell us if at the current configuration a given protocol version is acceptable.

This should work on OpenSSL 1.1.0+

[Adrien Nader (adrien)]

Hi, AFAIU the crux of the issue is that the behaviour on Ubuntu differs from upstream and is not programmatically discoverable.

OpenSSL 3.2 (which is not released yet and will most likely not be used in Ubuntu 24.04) switches to seclevel 2 and also has a different meaning for it. It's (almost?) completely in line with what Ubuntu does. The story is actually a bit more complicated because upstream wanted to change this before 3.2 (not sure anymore if that was planned for 3.1 or 3.0) and some changes happened but not others, and it's difficult to track that now.

Considering this bug is more than two years old and considering where we're heading, I think I'm going to mark this bug as won't fix. Ubuntu will continue to use 3.0 until the next openssl LTS release and the behavior is not expected to change. When the next openssl LTS release happens, Ubuntu will start using it soon after and the meaning of seclevel should be unchanged from upstream again (no guarantee though since I don't control openssl upstream).

The function mentioned by Dimitry also looks interesting if something finer grained is needed.

[Adrien Nader (adrien)]

Closing this as won't fix but the proper status would be "will-not-change-anything-but-it-will-get-fixed-when-upstream-changes-to-the-same-as-us-but-there-is-no-guarantee-there-wont-be-other-differences-until-their-version-is-released".