openssl 1.1.1q-0ubuntu1 - c_rehash script broken ("update-ca-certificates -f -v" fails)

[OwN]

I just installed openssl 1.1.1q as part of the apt-get update and apt-get upgrade process in Ubuntu 16.04 Xenial (it's a server using ffmpeg4), and now when I try to connect to any https website, I get the following error:

Verification error: unable to get local issuer certificate

update-ca-certificates -f -v produces the following output errors:

Doing .
x509: Unknown parameter cert
x509: Use -help for summary.
x509: Unknown parameter cert
x509: Use -help for summary.
x509: Unknown parameter cert
x509: Use -help for summary.
x509: Unknown parameter cert
x509: Use -help for summary.
x509: Unknown parameter cert
x509: Use -help for summary.
x509: Unknown parameter cert
x509: Use -help for summary.
x509: Unknown parameter cert

Any idea what's going on?

openssl s_client -CApath /etc/ssl/certs/ -showcerts -connect github.com:443 </dev/null

CONNECTED(00000003)
---
Certificate chain
 0 s:C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = github.com
   i:C = US, O = DigiCert Inc, CN = DigiCert TLS Hybrid ECC SHA384 2020 CA1
-----BEGIN CERTIFICATE-----
MIIFajCCBPCgAwIBAgIQBRiaVOvox+kD4KsNklVF3jAKBggqhkjOPQQDAzBWMQsw
CQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMTAwLgYDVQQDEydEaWdp
Q2VydCBUTFMgSHlicmlkIEVDQyBTSEEzODQgMjAyMCBDQTEwHhcNMjIwMzE1MDAw
MDAwWhcNMjMwMzE1MjM1OTU5WjBmMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2Fs
aWZvcm5pYTEWMBQGA1UEBxMNU2FuIEZyYW5jaXNjbzEVMBMGA1UEChMMR2l0SHVi
LCBJbmMuMRMwEQYDVQQDEwpnaXRodWIuY29tMFkwEwYHKoZIzj0CAQYIKoZIzj0D
AQcDQgAESrCTcYUh7GI/y3TARsjnANwnSjJLitVRgwgRI1JlxZ1kdZQQn5ltP3v7
KTtYuDdUeEu3PRx3fpDdu2cjMlyA0aOCA44wggOKMB8GA1UdIwQYMBaAFAq8CCkX
jKU5bXoOzjPHLrPt+8N6MB0GA1UdDgQWBBR4qnLGcWloFLVZsZ6LbitAh0I7HjAl
BgNVHREEHjAcggpnaXRodWIuY29tgg53d3cuZ2l0aHViLmNvbTAOBgNVHQ8BAf8E
BAMCB4AwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMIGbBgNVHR8EgZMw
gZAwRqBEoEKGQGh0dHA6Ly9jcmwzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydFRMU0h5
YnJpZEVDQ1NIQTM4NDIwMjBDQTEtMS5jcmwwRqBEoEKGQGh0dHA6Ly9jcmw0LmRp
Z2ljZXJ0LmNvbS9EaWdpQ2VydFRMU0h5YnJpZEVDQ1NIQTM4NDIwMjBDQTEtMS5j
cmwwPgYDVR0gBDcwNTAzBgZngQwBAgIwKTAnBggrBgEFBQcCARYbaHR0cDovL3d3
dy5kaWdpY2VydC5jb20vQ1BTMIGFBggrBgEFBQcBAQR5MHcwJAYIKwYBBQUHMAGG
GGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBPBggrBgEFBQcwAoZDaHR0cDovL2Nh
Y2VydHMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0VExTSHlicmlkRUNDU0hBMzg0MjAy
MENBMS0xLmNydDAJBgNVHRMEAjAAMIIBfwYKKwYBBAHWeQIEAgSCAW8EggFrAWkA
dgCt9776fP8QyIudPZwePhhqtGcpXc+xDCTKhYY069yCigAAAX+Oi8SRAAAEAwBH
MEUCIAR9cNnvYkZeKs9JElpeXwztYB2yLhtc8bB0rY2ke98nAiEAjiML8HZ7aeVE
P/DkUltwIS4c73VVrG9JguoRrII7gWMAdwA1zxkbv7FsV78PrUxtQsu7ticgJlHq
P+Eq76gDwzvWTAAAAX+Oi8R7AAAEAwBIMEYCIQDNckqvBhup7GpANMf0WPueytL8
u/PBaIAObzNZeNMpOgIhAMjfEtE6AJ2fTjYCFh/BNVKk1mkTwBTavJlGmWomQyaB
AHYAs3N3B+GEUPhjhtYFqdwRCUp5LbFnDAuH3PADDnk2pZoAAAF/jovErAAABAMA
RzBFAiEA9Uj5Ed/XjQpj/MxQRQjzG0UFQLmgWlc73nnt3CJ7vskCICqHfBKlDz7R
EHdV5Vk8bLMBW1Q6S7Ga2SbFuoVXs6zFMAoGCCqGSM49BAMDA2gAMGUCMCiVhqft
7L/stBmv1XqSRNfE/jG/AqKIbmjGTocNbuQ7kt1Cs7kRg+b3b3C9Ipu5FQIxAM7c
tGKrYDGt0pH8iF6rzbp9Q4HQXMZXkNxg+brjWxnaOVGTDNwNH7048+s/hT9bUQ==
-----END CERTIFICATE-----
 1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS Hybrid ECC SHA384 2020 CA1
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
-----BEGIN CERTIFICATE-----
MIIEFzCCAv+gAwIBAgIQB/LzXIeod6967+lHmTUlvTANBgkqhkiG9w0BAQwFADBh
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBD
QTAeFw0yMTA0MTQwMDAwMDBaFw0zMTA0MTMyMzU5NTlaMFYxCzAJBgNVBAYTAlVT
MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxMDAuBgNVBAMTJ0RpZ2lDZXJ0IFRMUyBI
eWJyaWQgRUNDIFNIQTM4NCAyMDIwIENBMTB2MBAGByqGSM49AgEGBSuBBAAiA2IA
BMEbxppbmNmkKaDp1AS12+umsmxVwP/tmMZJLwYnUcu/cMEFesOxnYeJuq20ExfJ
qLSDyLiQ0cx0NTY8g3KwtdD3ImnI8YDEe0CPz2iHJlw5ifFNkU3aiYvkA8ND5b8v
c6OCAYIwggF+MBIGA1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFAq8CCkXjKU5
bXoOzjPHLrPt+8N6MB8GA1UdIwQYMBaAFAPeUDVW0Uy7ZvCj4hsbw5eyPdFVMA4G
A1UdDwEB/wQEAwIBhjAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwdgYI
KwYBBQUHAQEEajBoMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5kaWdpY2VydC5j
b20wQAYIKwYBBQUHMAKGNGh0dHA6Ly9jYWNlcnRzLmRpZ2ljZXJ0LmNvbS9EaWdp
Q2VydEdsb2JhbFJvb3RDQS5jcnQwQgYDVR0fBDswOTA3oDWgM4YxaHR0cDovL2Ny
bDMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0R2xvYmFsUm9vdENBLmNybDA9BgNVHSAE
NjA0MAsGCWCGSAGG/WwCATAHBgVngQwBATAIBgZngQwBAgEwCAYGZ4EMAQICMAgG
BmeBDAECAzANBgkqhkiG9w0BAQwFAAOCAQEAR1mBf9QbH7Bx9phdGLqYR5iwfnYr
6v8ai6wms0KNMeZK6BnQ79oU59cUkqGS8qcuLa/7Hfb7U7CKP/zYFgrpsC62pQsY
kDUmotr2qLcy/JUjS8ZFucTP5Hzu5sn4kL1y45nDHQsFfGqXbbKrAjbYwrwsAZI/
BKOLdRHHuSm8EdCGupK8JvllyDfNJvaGEwwEqonleLHBTnm8dqMLUeTF0J5q/hos
Vq4GNiejcxwIfZMy0MJEGdqN9A57HSgDKwmKdsp33Id6rHtSJlWncg+d0ohP/rEh
xRqhqjn1VtvChMQ1H3Dau0bwhr9kAMQ+959GG50jBbl9s08PqUU643QwmA==
-----END CERTIFICATE-----
---
Server certificate
subject=C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = github.com

issuer=C = US, O = DigiCert Inc, CN = DigiCert TLS Hybrid ECC SHA384 2020 CA1

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2805 bytes and written 366 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 20 (unable to get local issuer certificate)
---

apt-cache policy openssl
openssl:
  Installed: 1.1.1q-0ubuntu1~16.04.sav0
  Candidate: 1.1.1q-0ubuntu1~16.04.sav0
  Version table:
 *** 1.1.1q-0ubuntu1~16.04.sav0 500
        500 http://ppa.launchpad.net/savoury1/ffmpeg4/ubuntu xenial/main amd64 Packages
        500 http://ppa.launchpad.net/savoury1/graphics/ubuntu xenial/main amd64 Packages
        500 http://ppa.launchpad.net/savoury1/multimedia/ubuntu xenial/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.2g-1ubuntu4.20+esm5 500
        500 https://esm.ubuntu.com/infra/ubuntu xenial-infra-security/main amd64 Packages
     1.0.2g-1ubuntu4.20 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
     1.0.2g-1ubuntu4 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

[OwN (own3mall)]

Running the below command as root fixed my issues:

openssl rehash /etc/ssl/certs/

It appears that c_rehash isn't working which is what update-ca-certificates -f calls. Did c_rehash get updated with this latest release of openssl?

[Rob Savoury (savoury1)]

Thanks for reporting this bug. The c_rehash script has indeed had changes made in two recent OpenSSL 1.1.1 versions:

* c_rehash was changed between 1.1.1n and 1.1.1o to address CVE-2022-1292 [1]
* c_rehash was changed again between 1.1.1o and 1.1.1p to address CVE-2022-2068 [2]

Changes to c_rehash for 1.1.1o were insufficient to fully address the issue of shell metacharacters not being properly sanitised per CVE-2022-1292. So more significant changes were made to c_rehash in 1.1.1p to cover all possibilities per CVE-2022-2068.

The changes in 1.1.1p required reworking of a Debian patch (c_rehash-compat.patch) that modifies c_rehash to always create old hashes for compatibility. This Debian patch was created back in 2010 and has persisted in all Debian OpenSSL versions through 1.1.1n-0+deb11u3 (backport of 1.1.1n to Debian stable, which is currently Bullseye).

A lack of Perl programming experience on my part combined with me not ever having used or tested c_rehash myself resulted in my initial rework of c_rehash-compat.patch for OpenSSL 1.1.1p (carried over to 1.1.1q) being incorrect for the new code changes. As confirmed by your bug report.

Doing some investigation this morning relative to this issue has revealed the simple error that I made. It was a missing parameter when calling the new link_hash subroutine, which was added in 1.1.1p and is now called by the link_hash_cert and link_hash_crl subroutines (these were essentially merged, due being mostly common code, into the new link_hash subroutine).

Adding the missing parameter for the new link_hash subroutine calls and bumping the position of the -subject_hash or -subject_hash_old (the one added by the Debian patch) parameter fixes the issue. A corrected version of c_rehash on my own system now works as expected.

Updated OpenSSL 1.1.1q packages will be uploaded to ppa:savoury1/encryption shortly, and once built and published the revised packages will then be copied to all other SavOS PPAs (ppa:savoury1/*) where this version of OpenSSL is also published (including ppa:savoury1/{ffmpeg4,graphics,multimedia} as on your affected server).

Also note that upstream do now recommend using rehash, which you did to resolve the issues you were having, rather than c_rehash that is described as obsolete by upstream. This detail is mentioned in the notes linked below.

[1] https://www.openssl.org/news/secadv/20220503.txt
[2] https://www.openssl.org/news/secadv/20220621.txt

[Rob Savoury (savoury1)]

Updated openssl 1.1.1q-0ubuntu1~XX.04.sav1 (where XX is year of particular Ubuntu release, ie. 16 for Xenial, 18 for Bionic, and 20 for Focal) packages are now available at ppa:savoury1/encryption and all other SavOS PPAs (ppa:savoury1/*) where src:openssl is published (including for Ubuntu 16.04 Xenial at ppa:savoury1/{ffmpeg4,graphics,multimedia} as used by OP). The new package reworks c_rehash-compat.patch (Debian patch to create old hashes for compatibility) so that patched c_rehash works again with c_rehash code changes implemented in OpenSSL 1.1.1p (to address CVE-2022-2068).

[OwN (own3mall)]

Thank you so much for fixing it and explaining what happened! I really appreciate it!

[Rob Savoury (savoury1)]

You are welcome! :-)