[SRU] OpenSSL 1.1.1 to 18.04 LTS

Bug #1797386 reported by Dimitri John Ledkov on 2018-10-11
286
This bug affects 61 people
Affects Status Importance Assigned to Milestone
libio-socket-ssl-perl (Ubuntu)
Bionic
Undecided
Unassigned
libnet-ssleay-perl (Ubuntu)
Bionic
Undecided
Unassigned
nova (Ubuntu)
Bionic
Undecided
Unassigned
openssl (Ubuntu)
Undecided
Unassigned
Bionic
Undecided
Unassigned
python-cryptography (Ubuntu)
Bionic
Undecided
Unassigned
python2.7 (Ubuntu)
Bionic
Undecided
Unassigned
python3.6 (Ubuntu)
Bionic
Undecided
Unassigned
python3.7 (Ubuntu)
Bionic
Undecided
Unassigned
r-cran-openssl (Ubuntu)
Bionic
Undecided
Unassigned
ruby-openssl (Ubuntu)
Bionic
Undecided
Unassigned
ruby2.5 (Ubuntu)
Bionic
Undecided
Unassigned

Bug Description

[Impact]

 * OpenSSL 1.1.1 is an LTS release upstream, which will continue to receive security support for much longer than 1.1.0 series will.

 * OpenSSL 1.1.1 comes with support for TLS v1.3 which is expected to be rapidly adopted due to increased set of supported hashes & algoes, as well as improved handshake [re-]negotiation.

 * OpenSSL 1.1.1 comes with improved hw-acceleration capabilities.

 * OpenSSL 1.1.1 is ABI/API compatible with 1.1.0, however some software is sensitive to the negotiation handshake and may either need patches/improvements or clamp-down to maximum v1.2.

[Test Case]

 * Rebuild all reverse dependencies

 * Execute autopkg tests for all of them

 * Clamp down to TLS v1.2 software that does not support TLS v1.3 (e.g. mongodb)

 * Backport TLS v1.3 support patches, where applicable

[Regression Potential]

 * Connectivity interop is the biggest issues which will be unavoidable with introducing TLS v1.3. However, tests on cosmic demonstrate that curl/nginx/google-chrome/mozilla-firefox connect and negotiate TLS v1.3 without issues.

 * Mitigation of discovered connectivity issues will be possible by clamping down to TLS v1.2 in either server-side or client-side software or by backporting relevant support fixes

 * Notable changes are listed here https://wiki.openssl.org/index.php/TLS1.3

 * Most common connectivity issues so far:
   - client verifies SNI in TLSv1.3 mode, yet client doesn't set hostname. Solution is client change to set hostname, or to clamp down the client to TLSv1.2.

   - session negotiation is different in TLSv1.3, existing client code may fail to create/negotiate/resume session. Clients need to learn how to use session callback.

 * This update bundles python 3.6 and 3.7 point releases

 * Following the change in Cosmic and up, this SRU also includes a distro patch that lowers OPENSSL_TLS_SECURITY_LEVEL from 1 to 0, to allow for establishing client->server server->client connections with lower grade security settings (e.g. sub-80bits keys, MD5/SHA1 certificate checksums, and other crap like that). This is to continue allow bionic clients to connect to servers operating with older 1.0.x based openssl, as typically clients are at no mercy to reject servers that do not have any better certs/keys/signatures. Thus potentially weak-security connections that previously would fail to establish to/from bionic, may now be accepted. Some may view this as a regression. In that case adjust openssl.cnf to a higher TLS_SECURITY_LEVEL, or use the openssl ctx APIs to set a higher TLS security level. See further comments in this bug report as to when we will be raising this LEVEL up (currently timeline is to raise to 2, in 20.04 LTS).

[Other Info]

 * Previous FFe for OpenSSL in 18.10 is at
   https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1793092

 * TLS v1.3 support in NSS is expected to make it to 18.04 via security updates

 * TLS v1.3 support in GnuTLS is expected to be available in 19.04

 * Test OpenSSL is being prepared in
   https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3473

tags: added: bionic
summary: - SRU OpenSSL 1.1.1 to 18.04 LTS
+ [SRU] OpenSSL 1.1.1 to 18.04 LTS
description: updated
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openssl (Ubuntu):
status: New → Confirmed
Sven Neuhaus (sven0) wrote :

I'm very much in favor of this.

Does this imply an update to Apache 2.4.37, too? (see https://github.com/apache/httpd/blob/2.4.x/CHANGES)

Giraffe (dodger-forum) wrote :

I would love to be able to Use OpenSSL 1.1.1 (TLS 1.3) with Apache2 on 18.04 LTS.

Marc Peña (pachulo) wrote :

Any news on this? Thanks!

Dimitri John Ledkov (xnox) wrote :

Here is a quick update on this SRU.

Bringing in Apache support is currently not in scope. However, this can be investigated separately and possibly would most likely look like a targetted backport of mod_ssl, rather than a full upgrade of all of the apache2. But again only after OpenSSL 1.1.1 SRU is completed.

It is investigated to bring OpenSSH compiled against libcrypto 1.1.1 support. But again only after OpenSSL 1.1.1 SRU is complete.

The current goal is to SRU OpenSSL 1.1.1 without causing any regressions to the dependent packages, which is quite a large task. In practice that does mean enabling TLS1.3 support in a few packages that are affected by the new handshake.

As stated, this SRU is being staged https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3473 possibly with a better stage page of the currently expected runtime regressions as shown at this page https://bileto.ubuntu.com/excuses/3473/bionic.html

As you can see there, this upgrade cannot land until after relevant python/perl/ruby/R changes are also brought in. Python stack is mostly ready now, the others will be quite easier to test and land.

If you can, I do urge you to test https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3473 PPA on bionic with your workloads to spot breakage, incompatibility, and/or any unexpected connectivity issues (client<->server protocol negotiation failures).

My personal goal is to land this in time / well ahead of the next bionic point release (currently penciled in for 7th February). But this is not a guarantee or a firm commitment that one can bank on.

I hope this update helps.

Matt Ruffalo (mruffalo) wrote :

Thank you very much, Dimitri -- I am interested in this also.

I tested that PPA on a test web server running nginx, uwsgi, uwsgi-plugin-python3, Django 1.11(.16), and a Python 3.6 'pyvenv' virtual environment using 'psycopg2' to connect to a PostgreSQL 10 server via the pre-built Python wheel for 'psycopg2_binary' version 2.7.5.

I could immediately connect to nginx over TLS 1.3 without any problems, and the Qualys SSL Labs scan also reported that all was well with TLS 1.3.

However, the web app under uwsgi crashed (segfaulted) on any request, with a stack trace at https://pastebin.com/DLGiuKfR

I was relatively surprised that the 'psycopg2_binary' Python wheel seemed to bundle its own version of libssl-8bb9b3dd.so.1.0.2o -- and it looks like there's some incompatibility with this build of Python and OpenSSL 1.1.1. I removed this Python package and installed 'psycopg2' instead, and saw the same behavior.

I was able to fix this by reinstalling psycopg2 from source with 'pip install --no-binary=":all:" psycopg2', and now everything works well with the web app.

I'm not sure how much of a problem this is at this stage, or who has the responsibility to address it (Ubuntu developers or whoever built the psycopg2 wheel), but I figured I may as well mention this anyway.

It's great that everything was fine with nginx without any effort on my part; thanks!

description: updated
Dimitri John Ledkov (xnox) wrote :

@mruffalo

This is about distribution provided packages only, installed system-wide. Not about binaries side installed from wheels from third-party providers. Can you test using
   $ sudo apt install python3-psycopg2

Without any wheels installed from pypi...

description: updated
Changed in openssl (Ubuntu):
status: Confirmed → In Progress

Hello Dimitri, or anyone else affected,

Accepted r-cran-openssl into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/r-cran-openssl/1.0.1-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in r-cran-openssl (Ubuntu Bionic):
status: New → Fix Committed
no longer affects: r-cran-openssl (Ubuntu)
no longer affects: libio-socket-ssl-perl (Ubuntu)
tags: added: verification-needed verification-needed-bionic
no longer affects: libnet-ssleay-perl (Ubuntu)
no longer affects: nova (Ubuntu)
no longer affects: python2.7 (Ubuntu)
no longer affects: python-cryptography (Ubuntu)
Steve Langasek (vorlon) wrote :

Hello Dimitri, or anyone else affected,

Accepted ruby-openssl into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ruby-openssl/2.0.9-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

no longer affects: python3.6 (Ubuntu)
no longer affects: python3.7 (Ubuntu)
Changed in ruby-openssl (Ubuntu Bionic):
status: New → Fix Committed
no longer affects: ruby-openssl (Ubuntu)
no longer affects: ruby2.5 (Ubuntu)
Dr. Uwe Meyer-Gruhl (meyergru) wrote :

Hi,

good idea in theory, but I want to add my 2cents: Please coordinate this update with ALL affected packages, like apache2 and nginx.

My reason is:

I just tried the PPA and found that nginx works with TLS 1.3 after that right out of the box.

HOWEVER, there is a problem: openssl 1.1.1 has changed the way the cipher suites are configured - the ones for TLS 1.3 are configured separately, see here:

https://github.com/openssl/openssl/commit/f865b08143b453962ad4afccd69e698d13c60f77

Nginx on the other hand has chosen to not support that new configuration at all, see:

https://trac.nginx.org/nginx/ticket/1529

That means that the predefined order of TLS 1.3 is:

TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256

and cannot be changed - it took me hours to find that out since the Nginx 1.15 documentation does not tell you that the TLS 1.3 ciphers cannot be changed by ssl_ciphers, but are silently ignored.

The default set and order of ciphersuites may suit your needs or not - matter-of-fact it makes my SSLLabs score worse because of the AES128 cipher used. I have tried to apply othe defaults in /etc/ssl/openssl.conf but they do not seem to work for nginx. Neither could I just disable TLS 1.3 in order to restore the old behaviour other than to restore OpenSSL 1.1.0 by using "ppa-purge ppa:ci-train-ppa-service/3473".

King regards,

Uwe

Steve Langasek (vorlon) wrote :

Acceptance of openssl currently blocked on coverage of the (distro patch) OPENSSL_TLS_SECURITY_LEVEL change as part of the SRU template.

Changed in openssl (Ubuntu Bionic):
status: New → Incomplete

On Mon, 11 Mar 2019 at 21:20, Steve Langasek
<email address hidden> wrote:
>
> Acceptance of openssl currently blocked on coverage of the (distro
> patch) OPENSSL_TLS_SECURITY_LEVEL change as part of the SRU template.
>

In Debian (but never ubuntu) they have bumped the default security
level from 1, to 2.

In Ubuntu, we have further decreased security level from 1 to 0, for
connectivity compatibility with openssl 1.0.2. This change was done in
cosmic, and is part of this SRU backport.

The reason for the decreased security level is to aid with
connectivity compatibily with older Ubuntu LTS releases based on
openssl 1.0.2. Such that bionic clients can connect to older servers,
even if the server uses small keys / md5 / etc.

I do not believe it is possible to set higher default security level
"for servers only". Thus we rely on server/daemon apps to have
stronger configuration, large keys, better certs, etc.

There are 1.1.0/1.1.1 APIs available to dynamically set higher
security levels, which highly active servers are using to increase
security levels in servers/daemons.

These changes are documented in the cosmic+ changelog with the
following entries:
- Revert "Enable system default config to enforce TLS1.2 as a
  minimum" & "Increase default security level from 1 to 2".
- Further decrease security level from 1 to 0, for compatibility with
  openssl 1.0.2.

Migration path to stonger defaults is to be done in 2020. This is
inline with major web-browsers too. All of them still support weaker
defaults. And all of them however have committed to drop support for
those in 2020. My expectation is to follow suit, and set default
security level to 2, and require TLS1.2 shortly after 19.10 release.

For the webbrowsers announcements please see these references:
https://blogs.windows.com/msedgedev/2018/10/15/modernizing-tls-edge-ie11/
https://webkit.org/blog/8462/deprecation-of-legacy-tls-1-0-and-1-1-versions/
https://security.googleblog.com/2018/10/modernizing-transport-security.html
https://blog.mozilla.org/security/2018/10/15/removing-old-versions-of-tls/

--
Regards,

Dimitri.

Seth Arnold (seth-arnold) wrote :

On Tue, Mar 12, 2019 at 04:05:45PM -0000, Dimitri John Ledkov wrote:
> defaults. And all of them however have committed to drop support for
> those in 2020. My expectation is to follow suit, and set default
> security level to 2, and require TLS1.2 shortly after 19.10 release.

Can you expand upon this point a bit?

Do you mean we will require tls 1.2 across all our supported releases
at the same time?

Or do you mean we will require tls 1.2 for 19.10 and newer? Will this be
done as part of rolling out 19.10 or will we push an update to 19.10 that
will change behaviour?

Or something else?

Thanks

Dimitri John Ledkov (xnox) wrote :

On Tue, 12 Mar 2019 at 19:35, Seth Arnold <email address hidden> wrote:
>
> On Tue, Mar 12, 2019 at 04:05:45PM -0000, Dimitri John Ledkov wrote:
> > defaults. And all of them however have committed to drop support for
> > those in 2020. My expectation is to follow suit, and set default
> > security level to 2, and require TLS1.2 shortly after 19.10 release.
>
> Can you expand upon this point a bit?
>
> Do you mean we will require tls 1.2 across all our supported releases
> at the same time?
>
> Or do you mean we will require tls 1.2 for 19.10 and newer? Will this be
> done as part of rolling out 19.10 or will we push an update to 19.10 that
> will change behaviour?
>
> Or something else?
>

I mean that, after 19.10 ships, and 20.04 development cycle opens, I
will upload openssl which sets compiled in TLS security default to
value 2, and sets minimum TLS 1.2 into 20.04 series.
Clients and servers, will be able to continue to configure lower
values via e.g. the SSL_CTX_set_security_level [1] and so on, to
establish less than TLS 1.2 / weaker keys / etc.
That's what my plan is for 20.04. I do not plan to backport this
change to prior releases. Mostly because apps would need to learn how
to use set_security_level etc, which stable software in bionic does
not currently do en mass.

W.r.t. web-browsers, I do expect them to release those changes to
their stable browser on all platforms. It it would mean that
eventually we'd backport stable Firefox with that change into bionic.
And google chrome from google on bionic will also drop tls1.0 and
tls1.1. So a limited exposure to dropping TLS1.0/1.1 in the clients
will be observed in 2020 on Ubuntu 18.04 LTS.

[1] https://www.openssl.org/docs/man1.1.0/man3/SSL_CTX_set_security_level.html

--
Regards,

Dimitri.

description: updated
Changed in openssl (Ubuntu Bionic):
status: Incomplete → Confirmed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.