expiring trust anchor compatibility issue

[Dimitri John Ledkov]

[Impact]

 * gnutls28 fails to talk to letsencrypt website past September 2021, despite trusting the letsencrypt root certificate.

[Test Plan]

 * Import staging cert equivalent to ISRG Root X1 https://letsencrypt.org/certs/staging/letsencrypt-stg-root-x1.pem

 * Import expired staging cert equivalen tto DST Root CA X3
https://letsencrypt.org/certs/staging/letsencrypt-stg-root-dst.pem

 * Test connectivity to the expired-root-ca test website
https://expired-root-ca-test.germancoding.com

setup:

apt install wget gnutls-bin
wget https://letsencrypt.org/certs/staging/letsencrypt-stg-root-x1.pem
wget https://letsencrypt.org/certs/staging/letsencrypt-stg-root-dst.pem
cat letsencrypt-stg-root-x1.pem letsencrypt-stg-root-dst.pem >> ca.pem

test case:
gnutls-cli --x509cafile=ca.pem expired-root-ca-test.germancoding.com

bad result:
- Status: The certificate is NOT trusted. The certificate chain uses expired certificate.
*** PKI verification of server certificate failed...
*** Fatal error: Error in the certificate.
*** handshake has failed: Error in the certificate.

good result:
- Status: The certificate is trusted.
- Description: (TLS1.3-X.509)-(ECDHE-SECP256R1)-(RSA-PSS-RSAE-SHA256)-(AES-256-GCM)
- Session ID: A8:2B:AF:85:54:64:3A:79:81:99:16:D4:6D:9A:FC:30:F1:EC:49:A4:09:A9:0C:31:37:38:C2:0E:73:C7:C9:04
- Options: OCSP status request,
- Handshake was completed

Connection should be successful and trusted with correctly working gnutls client that can manage to ignore expired CA, and build a valid trust path using non-expired CA in the chain.

[Where problems could occur]

 * Changes as to how the trust paths are built in TLS connection may result in introducing bugs (failure to connect to valid sites) and/or security vulnerabilities (connecting to invalid sites successfully).

[Other Info]

 * Background info
 * The current chain from letsencrypt is expiring, they are adding a new chain, but also keeping the expiring one. This will result in connectivity issues when using old gnutls/openssl against websites using the default letsencrypt configuration after September 2021.

https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816
https://community.letsencrypt.org/t/questions-re-openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143817

Currently gnutls28 in bionic and earlier will not establish a connection, if any parts of the trust chain have expired, even though alternative non-expired chains are available.

This has been fixed in GnuTLS 3.6.14, but probably should be backported to bionic and earlier if it was not already been done so.

https://gitlab.com/gnutls/gnutls/-/issues/1008

https://gitlab.com/gnutls/gnutls/-/merge_requests/1271

Openssl bug report for this issue is https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1928989

Bionic packages available from https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4661

Xenial packages available from https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4663

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Marc Deslauriers (mdeslaur)]

The backported patches in comments #1 and #2 look reasonable to me. +1 from the security team.

[Stefan Huehner (stefan-huehner)]

Hi Dmitry/Marc,
thanks for working on this and the related openssl bug, very appreciated trying avoiding the rapidly upcoming problem.

I think this gnutls could get be extra annoying (or very noisy for support) as bionic is both still active LTS and also apt itself uses gnutls backend. ESM maybe even worse (see end of this comment).

While Ubuntu repos itself seems to not have Let's Encrypt certificates a couple of 3rd party repos have and some maybe quite common for developers.

2 examples using Let's encrypt
a.) apt.postgresql.org
To get any still postgresql version for various ubuntu,debian releases
Note: They don't specifically use https:// url in their docs
b.) deb.nodesource.com
To get update node.js via an apt repo.
Their setup instructions specifically use https:// url's

While not having fix should not prevent apt from installing it (giving canonical repos seems to not be using Let's Encrypt) but:
- Lots of support question
- Not sure about unattended-upgrades, custom automation for package updates etc..

On top for ESM (i.e. xenial))
https://esm.ubuntu.com seems to be using Let's Encrypt
I did not check it specifically if it has the Android compatible chain triggering the openssl/gnutls bug or you are using the alternative chain.

If ESM is affected here that could be bigger issue as it prevents people from installing the fix (if they don't get it before 2021-10-01)

[Steve Langasek (vorlon)]

Hello Dimitri, or anyone else affected,

Accepted gnutls28 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/gnutls28/3.5.18-1ubuntu1.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Dimitri, or anyone else affected,

Accepted gnutls28 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/gnutls28/3.4.10-4ubuntu1.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted gnutls28 (3.4.10-4ubuntu1.9) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

network-manager/1.2.6-0ubuntu0.16.04.3 (arm64, ppc64el)
systemd/229-4ubuntu21.31 (amd64)
ffmpeg/7:2.8.17-0ubuntu0.1 (armhf)
lxc/2.0.11-0ubuntu1~16.04.3 (i386, s390x, amd64, arm64, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#gnutls28

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted gnutls28 (3.5.18-1ubuntu1.5) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

lxc/3.0.3-0ubuntu1~18.04.1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#gnutls28

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Dimitri John Ledkov (xnox)]

# gnutls-cli --x509cafile=ca.pem expired-root-ca-test.germancoding.com
Processed 2 CA certificate(s).
Resolving 'expired-root-ca-test.germancoding.com'...
Connecting to '2a01:4f8:151:506c::2:443'...
...
- Status: The certificate is NOT trusted. The certificate chain uses expired certificate.
*** PKI verification of server certificate failed...
*** Fatal error: Error in the certificate.
*** Handshake has failed
GnuTLS error: Error in the certificate.

# faketime 2021-10-01 gnutls-cli canonical.com
Processed 129 CA certificate(s).
Resolving 'canonical.com'...
Connecting to '2001:67c:1360:8001::2c:443'...
...
- Status: The certificate is NOT trusted. The certificate chain uses expired certificate.
*** PKI verification of server certificate failed...
*** Fatal error: Error in the certificate.
*** Handshake has failed
GnuTLS error: Error in the certificate.

Reproduced original problem, upgraded libgnutls30 and gnutls-bin

# dpkg-query -W libgnutls30 gnutls-bin
gnutls-bin 3.4.10-4ubuntu1.9
libgnutls30:amd64 3.4.10-4ubuntu1.9

Things are good now:

# gnutls-cli --x509cafile=ca.pem expired-root-ca-test.germancoding.com
Processed 2 CA certificate(s).
Resolving 'expired-root-ca-test.germancoding.com'...
Connecting to '2a01:4f8:151:506c::2:443'...
...
- Status: The certificate is trusted.
- Description: (TLS1.2)-(ECDHE-RSA-SECP256R1)-(AES-256-GCM)
- Session ID: 53:9C:C7:86:1B:69:79:FC:37:AD:AD:A5:82:11:46:84:4F:B4:46:DC:C1:E7:2E:A9:40:18:6C:8A:B9:4C:B9:7E
- Ephemeral EC Diffie-Hellman parameters
 - Using curve: SECP256R1
 - Curve size: 256 bits
- Version: TLS1.2
- Key Exchange: ECDHE-RSA
- Server Signature: RSA-SHA256
- Cipher: AES-256-GCM
- MAC: AEAD
- Compression: NULL
- Options: extended master secret, safe renegotiation, OCSP status request,
- Handshake was completed

# faketime 2021-10-01 gnutls-cli canonical.com
Processed 129 CA certificate(s).
Resolving 'canonical.com'...
Connecting to '2001:67c:1360:8001::2b:443'...
- Certificate type: X.509
- Got a certificate list of 3 certificates.
...
- Status: The certificate is trusted.
- Description: (TLS1.2)-(ECDHE-RSA-SECP256R1)-(AES-256-GCM)
- Session ID: 71:90:E2:22:86:03:D2:13:17:6E:F1:20:8A:57:B8:E9:FF:0E:07:AB:1E:61:F6:7F:56:43:EF:BF:7A:F3:EF:56
- Ephemeral EC Diffie-Hellman parameters
 - Using curve: SECP256R1
 - Curve size: 256 bits
- Version: TLS1.2
- Key Exchange: ECDHE-RSA
- Server Signature: RSA-SHA256
- Cipher: AES-256-GCM
- MAC: AEAD
- Compression: NULL
- Options: extended master secret, safe renegotiation,
- Handshake was completed

All is good. Test website connectivity works, and future connectivity to canonical.com also works.

xenial is verified.

[Dimitri John Ledkov (xnox)]

# dpkg-query -W gnutls-bin libgnutls30
gnutls-bin 3.5.18-1ubuntu1.4
libgnutls30:amd64 3.5.18-1ubuntu1.4

# gnutls-cli --x509cafile=ca.pem expired-root-ca-test.germancoding.com
Processed 2 CA certificate(s).
Resolving 'expired-root-ca-test.germancoding.com:443'...
Connecting to '2a01:4f8:151:506c::2:443'...
...
*** PKI verification of server certificate failed...
*** Fatal error: Error in the certificate.
*** handshake has failed: Error in the certificate.

# faketime 2021-10-01 gnutls-cli canonical.com
Processed 129 CA certificate(s).
Resolving 'canonical.com:443'...
Connecting to '2001:67c:1360:8001::2b:443'...
...
- Status: The certificate is NOT trusted. The certificate chain uses expired certificate.
*** PKI verification of server certificate failed...
*** Fatal error: Error in the certificate.
*** handshake has failed: Error in the certificate.

Upgrading gnutls
# dpkg-query -W gnutls-bin libgnutls30
gnutls-bin 3.5.18-1ubuntu1.5
libgnutls30:amd64 3.5.18-1ubuntu1.5

# gnutls-cli --x509cafile=ca.pem expired-root-ca-test.germancoding.com </dev/null
Processed 2 CA certificate(s).
Resolving 'expired-root-ca-test.germancoding.com:443'...
Connecting to '2a01:4f8:151:506c::2:443'...
...
- Status: The certificate is trusted.
- Description: (TLS1.2)-(ECDHE-RSA-SECP256R1)-(AES-256-GCM)
- Session ID: 04:95:FF:FD:DF:83:B3:E2:3B:00:83:B7:FA:8B:4C:7D:CB:7A:CE:F4:ED:C6:50:62:A7:EF:07:4C:56:FA:91:A9
- Ephemeral EC Diffie-Hellman parameters
 - Using curve: SECP256R1
 - Curve size: 256 bits
- Version: TLS1.2
- Key Exchange: ECDHE-RSA
- Server Signature: RSA-SHA256
- Cipher: AES-256-GCM
- MAC: AEAD
- Compression: NULL
- Options: extended master secret, safe renegotiation, OCSP status request,
- Handshake was completed

- Simple Client Mode:

# faketime 2021-10-01 gnutls-cli canonical.com
Processed 129 CA certificate(s).
Resolving 'canonical.com:443'...
Connecting to '2001:67c:1360:8001::2c:443'...
...
- Status: The certificate is trusted.
- Description: (TLS1.2)-(ECDHE-RSA-SECP256R1)-(AES-256-GCM)
- Session ID: 6D:77:C1:D6:9B:F3:5F:97:19:D2:AF:AD:8E:8A:1C:7F:9E:2F:9E:D2:80:77:EE:82:D1:F7:1F:F4:F2:1D:50:E5
- Ephemeral EC Diffie-Hellman parameters
 - Using curve: SECP256R1
 - Curve size: 256 bits
- Version: TLS1.2
- Key Exchange: ECDHE-RSA
- Server Signature: RSA-SHA256
- Cipher: AES-256-GCM
- MAC: AEAD
- Compression: NULL
- Options: extended master secret, safe renegotiation,
- Handshake was completed

- Simple Client Mode:

All is good on bionic.

[Dimitri John Ledkov (xnox)]

On xenial lxc autopkgtest fails with "ERROR: Unable to fetch GPG key from keyserver." due to using keyserver that is no longer available on the internet.

[Dimitri John Ledkov (xnox)]

In xenial systemd autopkgtest only fails with boot-smoke

FAIL: expected: '' actual: ' 1 graphical.target start waiting
 92 rng-tools.service start running
101 systemd-update-utmp-runlevel.service start waiting
  2 multi-user.target start waiting'
autopkgtest [09:24:51]: test boot-smoke: -----------------------]
boot-smoke FAIL non-zero exit status 1

as if timeout is too low / cloud is busy, hence the nested KVM VM boot is taking longer than expected.

[Dimitri John Ledkov (xnox)]

bionic autopkgtests are all clean

[Launchpad Janitor (janitor)]

This bug was fixed in the package gnutls28 - 3.5.18-1ubuntu1.5

---------------
gnutls28 (3.5.18-1ubuntu1.5) bionic; urgency=medium

  * Backport patches from Upstream/Debian to check validity against system
    certs. This is to allow correctly validating default letsencrypt
    chains that now also include a redundant expired certficate. LP:
    #1928648

 -- Dimitri John Ledkov <email address hidden> Wed, 25 Aug 2021 19:11:11 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for gnutls28 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Dimitri John Ledkov (xnox)]

xenial autopkgtest regressions explained in
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1928648/comments/13
https://bugs.launchpad.net/ubuntu/+source/gnutls28/+bug/1928648/comments/14

[Launchpad Janitor (janitor)]

This bug was fixed in the package gnutls28 - 3.4.10-4ubuntu1.9

---------------
gnutls28 (3.4.10-4ubuntu1.9) xenial; urgency=medium

  * Backport patches from Upstream/Debian to check validity against system
    certs. This is to allow correctly validating default letsencrypt
    chains that now also include a redundant expired certficate. LP:
    #1928648

 -- Dimitri John Ledkov <email address hidden> Fri, 27 Aug 2021 14:19:17 +0100

[Steve Beattie (sbeattie)]

Ack from the Ubuntu Security team for both gnutls28 3.5.18-1ubuntu1.5 and 3.4.10-4ubuntu1.9 to go to bionic-security and xenial-security respectively.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.