expiring trust anchor compatibility issue

Bug #1928989 reported by Dimitri John Ledkov
276
This bug affects 3 people
Affects Status Importance Assigned to Milestone
openssl (Ubuntu)
Fix Released
Undecided
Unassigned
Trusty
Confirmed
Critical
Unassigned
Xenial
Fix Released
Critical
Steve Beattie

Bug Description

[Impact]

 * openssl fails to talk to letsencrypt website past September 2021, despite trusting the letsencrypt root certificate.

[Test Plan]

 * Import staging cert equivalent to ISRG Root X1 https://letsencrypt.org/certs/staging/letsencrypt-stg-root-x1.pem

 * Import expired staging cert equivalen tto DST Root CA X3
https://letsencrypt.org/certs/staging/letsencrypt-stg-root-dst.pem

 * Test connectivity to the expired-root-ca test website
https://expired-root-ca-test.germancoding.com

setup:

apt install openssl ca-certificates wget
wget https://letsencrypt.org/certs/staging/letsencrypt-stg-root-x1.pem
wget https://letsencrypt.org/certs/staging/letsencrypt-stg-root-dst.pem
cat letsencrypt-stg-root-x1.pem letsencrypt-stg-root-dst.pem >> ca.pem

test case:
openssl s_client -connect expired-root-ca-test.germancoding.com:443 -servername expired-root-ca-test.germancoding.com -verify 1 -verifyCAfile ca.pem

bad result:
connection failed
verify depth is 1
CONNECTED(00000003)
depth=3 C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Doctored Durian Root CA X3
verify error:num=10:certificate has expired
notAfter=Jan 30 14:01:15 2021 GMT
140672978626200:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:s3_clnt.c:1264:

good result:
connection successful

verify depth is 1
CONNECTED(00000003)
depth=2 C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
verify return:1
depth=1 C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
verify return:1
depth=0 CN = expired-root-ca-test.germancoding.com
verify return:1
---
Certificate chain
 0 s:/CN=expired-root-ca-test.germancoding.com
   i:/C=US/O=(STAGING) Let's Encrypt/CN=(STAGING) Artificial Apricot R3
 1 s:/C=US/O=(STAGING) Let's Encrypt/CN=(STAGING) Artificial Apricot R3
   i:/C=US/O=(STAGING) Internet Security Research Group/CN=(STAGING) Pretend Pear X1
 2 s:/C=US/O=(STAGING) Internet Security Research Group/CN=(STAGING) Pretend Pear X1
   i:/C=US/O=(STAGING) Internet Security Research Group/CN=(STAGING) Doctored Durian Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----

Connection should be successful and trusted with correctly working openssl s_client that can manage to ignore expired CA, and build a valid trust path using non-expired CA in the chain.

[Testcase #2]

$ sudo apt install ca-certificates wget faketime

# Good connectivity
$ wget -O /dev/null https://canonical.com
--2021-07-13 11:54:20-- https://canonical.com/
Resolving canonical.com (canonical.com)... 2001:67c:1360:8001::2b, 2001:67c:1360:8001::2c, 91.189.88.181, ...
Connecting to canonical.com (canonical.com)|2001:67c:1360:8001::2b|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30933 (30K) [text/html]
Saving to: '/dev/null'

/dev/null 100%[============================>] 30.21K --.-KB/s in 0.001s

2021-07-13 11:54:20 (22.3 MB/s) - '/dev/null' saved [30933/30933]

# Jump to october to experience failure
$ faketime '2021-10-01' wget -O /dev/null https://canonical.com
--2021-10-01 00:00:00-- https://canonical.com/
Resolving canonical.com (canonical.com)... 2001:67c:1360:8001::2b, 2001:67c:1360:8001::2c, 91.189.88.181, ...
Connecting to canonical.com (canonical.com)|2001:67c:1360:8001::2b|:443... connected.
ERROR: cannot verify canonical.com's certificate, issued by 'CN=R3,O=Let\'s Encrypt,C=US':
  Issued certificate has expired.
To connect to canonical.com insecurely, use `--no-check-certificate'.

# upgrade to new openssl, to see that connectivity is restored, even in october
$ dpkg-query -W libssl1.0.0
libssl1.0.0:amd64 1.0.2g-1ubuntu4.20

$ faketime '2021-10-01' wget -O /dev/null https://canonical.com
--2021-10-01 00:00:00-- https://canonical.com/
Resolving canonical.com (canonical.com)... 2001:67c:1360:8001::2c, 2001:67c:1360:8001::2b, 91.189.88.180, ...
Connecting to canonical.com (canonical.com)|2001:67c:1360:8001::2c|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 30933 (30K) [text/html]
Saving to: '/dev/null'

/dev/null 100%[============================>] 30.21K --.-KB/s in 0.001s

2021-10-01 00:00:00 (21.9 MB/s) - '/dev/null' saved [30933/30933]

[Where problems could occur]

 * Changes as to how the trust paths are built in TLS connection may result in introducing bugs (failure to connect to valid sites) and/or security vulnerabilities (connecting to invalid sites successfully).

[Other Info]

 * Background info
 * The current chain from letsencrypt is expiring, they are adding a new chain, but also keeping the expiring one. This will result in connectivity issues when using old gnutls/openssl against websites using the default letsencrypt configuration after September 2021.

https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816
https://community.letsencrypt.org/t/questions-re-openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143817

Currently openssl in xenial and earlier will not establish a connection, if any parts of the trust chain have expired, even though alternative non-expired chains are available.

This has been fixed in OpenSSL 1.1.0+ by setting X509_V_FLAG_TRUSTED_FIRST by default see https://github.com/openssl/openssl/commit/0daccd4dc1f1ac62181738a91714f35472e50f3c

Note only the new default flag is backported on its own. The other changes to more strongly verify additional auxiliary trust OIDs and key usage are not backported.

gnutls bug for this is https://bugs.launchpad.net/ubuntu/bionic/+source/gnutls28/+bug/1928648

Changed in openssl (Ubuntu):
status: New → Fix Released
information type: Public → Public Security
tags: added: letsencryptexpiry
removed: letsencrypt
description: updated
description: updated
Revision history for this message
Dimitri John Ledkov (xnox) wrote :
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

python3.5 has stopped passing its testsuite due to expried test certs. Thus upload of openssl has triggered regression in python3.5

I've cherrypicked updated test certs and keys, but to cherry-pick those cleanly, I also had to cherrypick an earlier bug fix. All of these are unmodified from 3.5.10 release. With these patches in place python3.5 testsuite passes with both current and proposed update of openssl.

Changed in openssl (Ubuntu Xenial):
status: New → Confirmed
Revision history for this message
Dimitri John Ledkov (xnox) wrote :
Revision history for this message
Dimitri John Ledkov (xnox) wrote :
description: updated
Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

I think the patch in comment #1 looks reasonable.

description: updated
Changed in openssl (Ubuntu Xenial):
importance: Undecided → Critical
Changed in openssl (Ubuntu Trusty):
importance: Undecided → Critical
Revision history for this message
Steve Beattie (sbeattie) wrote :

Assigning the verification and publication to xenial-security to myself. Thanks.

Changed in openssl (Ubuntu Xenial):
assignee: nobody → Steve Beattie (sbeattie)
Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello Dimitri, or anyone else affected,

Accepted openssl into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openssl/1.0.2g-1ubuntu4.20 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in openssl (Ubuntu Xenial):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-xenial
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (openssl/1.0.2g-1ubuntu4.20)

All autopkgtests for the newly accepted openssl (1.0.2g-1ubuntu4.20) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

psqlodbc/1:09.03.0300-1 (armhf)
ruby2.3/2.3.1-2~ubuntu16.04.16 (s390x)
python3.5/3.5.2-2ubuntu0~16.04.13 (i386, amd64, ppc64el, s390x, armhf, arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#openssl

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Reproduced the bug with:

# dpkg-query -W libssl1.0.0 openssl
libssl1.0.0:amd64 1.0.2g-1ubuntu4.19
openssl 1.0.2g-1ubuntu4.19

# openssl s_client -connect expired-root-ca-test.germancoding.com:443 -servername expired-root-ca-test.germancoding.com -verify 1 -verifyCAfile ca.pem
verify depth is 1
CONNECTED(00000003)
depth=3 C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Doctored Durian Root CA X3
verify error:num=10:certificate has expired
notAfter=Jan 30 14:01:15 2021 GMT
140540576667288:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:s3_clnt.c:1264:

# upgrading

# dpkg-query -W libssl1.0.0 openssl
libssl1.0.0:amd64 1.0.2g-1ubuntu4.20
openssl 1.0.2g-1ubuntu4.20

# # openssl s_client -connect expired-root-ca-test.germancoding.com:443 -servername expired-root-ca-test.germancoding.com -verify 1 -verifyCAfile ca.pem
verify depth is 1
CONNECTED(00000003)
depth=2 C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
verify return:1
depth=1 C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
verify return:1
depth=0 CN = expired-root-ca-test.germancoding.com
verify return:1
---
Certificate chain
 0 s:/CN=expired-root-ca-test.germancoding.com
   i:/C=US/O=(STAGING) Let's Encrypt/CN=(STAGING) Artificial Apricot R3
 1 s:/C=US/O=(STAGING) Let's Encrypt/CN=(STAGING) Artificial Apricot R3
   i:/C=US/O=(STAGING) Internet Security Research Group/CN=(STAGING) Pretend Pear X1
 2 s:/C=US/O=(STAGING) Internet Security Research Group/CN=(STAGING) Pretend Pear X1
   i:/C=US/O=(STAGING) Internet Security Research Group/CN=(STAGING) Doctored Durian Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGgTCCBWmgAwIBAgITAPqeXD5BcpT3tXI8aoDSYano7DANBgkqhkiG9w0BAQsF

....

connection is successful.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

python3.5 ADT regression is in xenial-updates regression, because the test certificates it uses have expired.

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Download of canonical.com with faketime 2021-10-01 also works.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

ruby2.3 is not a regression on all other arches, not sure why s390x is the only "working" arch with failing test.

retried psqlodbc

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

psqlodbc confuses me, as if clusters fail to create. Seems unrelated to openssl changes.

Revision history for this message
Steve Beattie (sbeattie) wrote :

The Ubuntu Security Team is okay with publishig the xenial openssl in proposed (1.0.2g-1ubuntu4.20) to xenial-security and updates. I didn't see any symbol changes or dependency changes in the binaries that would have indicated that building against xenial-updates was a problem.

Thanks!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openssl - 1.0.2g-1ubuntu4.20

---------------
openssl (1.0.2g-1ubuntu4.20) xenial-security; urgency=medium

  * Enable X509_V_FLAG_TRUSTED_FIRST by default, such that letsencrypt
    connection with the default chain remains trusted even after the
    expiry of the redundant CA certificate. LP: #1928989

 -- Dimitri John Ledkov <email address hidden> Mon, 28 Jun 2021 14:05:36 +0100

Changed in openssl (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Steve Langasek (vorlon) wrote : Update Released

The verification of the Stable Release Update for openssl has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Attempted trusty backport, but failing at making it pass all the existing unit tests. Asking for help. At the moment it seems to me that trusty will remain unfixed.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openssl (Ubuntu Trusty):
status: New → Confirmed
Revision history for this message
Sylvain Beucler (beuc) wrote :

@xnox I'm working on an update for Debian Jessie (1.0.1t) as part of Debian ELTS.
I got one test suite failure in 'verify_extra_test' that I fixed by partially reverting https://github.com/openssl/openssl/commit/cb22d2ae5a5b6069dbf66dbcce07223ac15a16de (hence aligning the test with later OpenSSL versions).

Attached is my current debdiff, I'm open to exchanging code reviews :)

Revision history for this message
Romain Couturat (romaincout) wrote :

Note: for trusty the issue will be worked around by distrusting "DST Root CA X3", see https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+bug/1944481

Revision history for this message
Anonymous (reason) wrote :

Help!

I'm on Trusty and Launchpad PPAs no longer work for me!

sudo add-apt-repository ppa:oibaf/graphics-drivers

Cannot add PPA: '"Error reading https://launchpad.net/api/1.0/~oibaf/+archive/graphics-drivers: (60, 'server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none')"'.

What do I do? How do I fix this???

Revision history for this message
Haw Loeung (hloeung) wrote :

I think you want package updates from Ubuntu ESM, in particular ca-certificates 20190110~14.04.1~esm2.

To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Bug attachments