Missing Verisign certs due to broken extract script

[Marc Deslauriers]

Verisign shipped G1 PCA Roots with md2 signatures on them. At some point, they resigned those roots using SHA1, but requested that the original certs keep shipping in Mozilla's cert list as they had issued intermediates with AKIs that point to the
MD2 versions.

See discussion here:
https://groups.google.com/forum/?fromgroups#!msg/mozilla.dev.security.policy/I6bUbW3WkBU/lRxqGv6vYHYJ

Now, ca-certificates uses a script called "certdata2pem.py" to extract the certificates from the certdata.txt file provided by Mozilla into individual files. Unfortunately, the script names the certificate file using the CKA_LABEL. In two instances, the verisign md2 and sha1 certs have the same CKA_LABEL, so the script is overwriting the first one (md2) with the second one (sha1).

This results in the Verisign md2 certs being missing from the system ca certs.
This usually isn't a problem except in the case where a website is handing out a complete cert chain, including the md2 root cert. When that happens, webkit is unable to verify the md2 root cert, and the connection fails.

[Philip (philip-redhat-bugs)]

Description of problem:
This certificate is missing. "/C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)06/CN=VeriSign Class 3 Extended Validation SSL SGC CA"

Version-Release number of selected component (if applicable):
ca-certificates-2011.70-2.fc15.noarch

How reproducible:
Always

Steps to Reproduce:
1. wget https://secure.vonage.com/

Actual results:
wget returns error because of missing certificate.

Expected results:

Additional info:

[Brad (brad-redhat-bugs)]

I believe The actual problem is *not* the lack of the intermediate certificate "CN=VeriSign Class 3 Extended Validation SSL SGC CA", it is in fact that a root certificate

Subject: C=US, O=VeriSign, Inc., OU=Class 3 Public Primary Certification Authority
Serial: 70:ba:e4:1d:10:d9:29:34:b6:38:ca:7b:03:cc:ba:bf

was present in ca-certificates 2010.63-3, but missing in the recently released ca-certificates-2011.78-1.fc14.noarch. The missing CA certificate is still valid according to VeriSign and Mozilla.

This is a bit confusing, although the root certificate is valid, VeriSign stopped using it for signing in 5/2009, replacing it with another certificate with the same subject and keyid, but a different serial number (3c:91:31:cb:1f:f6:d0:1b:0e:9a:b8:d0:44:bf:12:be), as part of their move away from MD2 signatures.

My workaround: Add the dropped certificate manually back into /etc/pki/tls/certs/ca-bundle.crt

I notice that big sites such as vonage, paypal, optionsxpress still deliver certificates whose trust is ultimately established by the now missing root certificate.

[Brad (brad-redhat-bugs)]

Created attachment 531861
The missing certificate

Appending this root certificate to /etc/pki/tls/certs/ca-bundle.crt restores validation of the sites mentioned herein. But it's never a good idea to trust root CA's from strangers. So consider this as posted just for reference. :)

[Joe (joe-redhat-bugs)]

The list of trusted CAs is inherited from upstream (Mozilla) and we are not going to change it ourselves within Fedora - sorry.

[Brad (brad-redhat-bugs)]

(In reply to comment #3)
> The list of trusted CAs is inherited from upstream (Mozilla) and we are not
> going to change it ourselves within Fedora - sorry.

Just a few more notes.

Both the SH1 and MD2 certificates *do* appear to be included in Mozilla's certdata.txt r1.78 (at lines 1010 and 17805), yet the MD2 certficate is not in ca-bundle.crt in package ca-certificates.

I believe the bug is in certdata2pem.py, which does not handle the case where two certificates have the same CKA_LABEL (as is the case in certdata.txt r1.78), since it tries to output both certificates to the same file (one overwrites the other).

I'll add for the google that the Class 3 certificate is not the only one that gets dropped from ca-bundle.pem by the certdata2pem.py script. "Verisign Class 1 Public Primary Certification Authority" also appears as the label of two different certs (same underlying key, signed in two different ways).

Regards,
Brad

[Joe (joe-redhat-bugs)]

Ah, good catch Brad, thanks for checking!

[Brad (brad-redhat-bugs)]

Created attachment 533817
For your review, I'd propose this patch to fix this.

[Brad (brad-redhat-bugs)]

Created attachment 533818
For your review, I'd propose this patch to fix this.

Oops, RHBZ didn't like my patch format. Try this.

[Marc Deslauriers (mdeslaur)]

Here is a small reproducer that shows the issue with a website that needs the md2 Verisign cert.

[Philipp Kern (pkern)]

I think it would be irresponsible to provide MD2-signed certificates. The discussion is dated 2009. I think ca-certificates should provide neither MD2 nor MD5 root certificates. And MD2 verification should be unsupported in the crypto lib anyway (see CVE-2009-2409).

[Marc Deslauriers (mdeslaur)]

These are _root_ certs, the crypto library doesn't verify the signatures on root certs, since they are self-signed.

If we really don't want to ship md2 root certs, we need to make sure ca-certificates deliberately disables them, instead of overwriting them by coincidence just because they are listed first in Mozilla's cert file.

In theory, the sha1 cert should be sufficient, and earlier versions of libsoup accepted that one without an issue. I'm currently investigating whether this is a regression in libsoup or not.

[Marc Deslauriers (mdeslaur)]

OK, I am now convinced that we don't need the md2 certs, applications
should be able to validate using the sha1 certs. I believe a bug in
libsoup/glib-networking is causing the sha1 certs to not be used.

We still should improve ca-certificates to make _sure_ that we're
shipping the sha1 certs instead of the md2 certs, as it currently ships
the sha1 certs by coincidence as they are listed later in Mozilla's
file. If they ever change the order of their file, we'll be shipping the
md2 ones by mistake.

[Michael Vogt (mvo)]

I looked a bit at the gio code this morning and it appears the problem with the site in question is that gtlsdatabase-gnutls.c:build_certificate_chain does not find a "anchor" and therefore passes NULL as the anchors to gnutls_x509_crt_list_verify() which always fails with " *output |= GNUTLS_CERT_SIGNER_NOT_FOUND | GNUTLS_CERT_INVALID;" in lib/x509/verify.c:_gnutls_verify_certificate2. The cli version of gnutls seems to simply pass the list of all trusted CAs to gnutls_x509_crt_list_verify() instead of trying to find the right trusted CA itself (which looks like a more sensible approach to me).

[Michael Vogt (mvo)]

It seems like the problem is the following:

GNUTLS:
- gnutls passes all certificates in /etc/ssl/certs/ca-certificates.crt
- the server secure-test.streamline-esolutions.com returns a certificate that is signed with the Verisign_Class_3_Public_Primary_Certification_Authority.pem certificate with the fingerprint "openssl x509 -in Verisign_Class_3_Public_Primary_Certification_Authority.pem -noout -fingerprint
SHA1 Fingerprint=A1:DB:63:93:91:6F:17:E4:18:55:09:40:04:15:C7:02:40:B0:AE:6B"
- the server does *not* return this certificate though, it returns a weaker md2 certificate
- the "A1:DB:63:93:91:6F:17:E4:18:55:09:40:04:15:C7:02:40:B0:AE:6B"" certificate is part of the trusted certs so gnutls is happy

GIO:
- gio-network is looking at each of the certificates in the certification chain retuned by the server
- it will *not* load all certificates from /etc/ssl/certs/ca-certificates.crt for verification, but *only* those that are also returned by the server as part of the verification chain
- the server only returns the weak md2 ceritificate and not the stronger A1:DB:63:93:91:6F:17:E4:18:55:09:40:04:15:C7:02:40:B0:AE:6B certificate
- gio can not find the md2 certificate as we do not ship that
- gio fails with a error as it does not find root certificate from the server that is also in the local /etc/ssl/certs/ca-certificates.crt

This indicates server misconfiguration IMO plus a it seems that gio is overly restrictive here. It could simply pass all data to gnutls for the verification.

[Marc Deslauriers (mdeslaur)]

I've opened LP: #1033516 for the bug that glib-networking (and libsoup >= 2.37) won't validate properly using the sha1 cert.

[Michael Vogt (mvo)]

Fwiw, when inspecting the site with mozilla and chromeium I see the md2 cert in the root of the chain.

And openssl returns:
$ openssl s_client -connect secure-test.streamline-esolutions.com:443 ; openssl s_client -connect secure-test.streamline-esolutions.com
    Verify return code: 19 (self signed certificate in certificate chain)

Which makes me wonder if adding the md2 certs back is not the right option as that is apparently what mozilla and chrome(ium) are doing. Plus openssl fails.

Technically I think (but I have to admit a certain ignorance about the standard) the verification chain is invalid because the server sends that the certificate issuer of the cert in the middle is the md2 cert. It just happens that gnutls implements the verification by trying to find a issuer from the list of trusted certificates and does not rely on the issuer set in the cert itself.

[Marc Deslauriers (mdeslaur)]

mozilla and Chromium still have the md2 cert, because VeriSign had issued intermediates with AKIs that point to the
MD2 versions. I'm not sure there are any left though.

If you remove the md2 cert from firefox, and restart it, it will still validate the site correctly.

You need to tell openssl where the CA cert bundle is:

openssl s_client -CAfile /etc/ssl/certs/ca-certificates.crt -connect secure-test.streamline-esolutions.com:443

Doing that results in a successful verification, even though the md2 cert isn't in the system CA bundle:
Verify return code: 0 (ok)

[Fedora (fedora-redhat-bugs)]

This message is a notice that Fedora 15 is now at end of life. Fedora
has stopped maintaining and issuing updates for Fedora 15. It is
Fedora's policy to close all bug reports from releases that are no
longer maintained. At this time, all open bugs with a Fedora 'version'
of '15' have been closed as WONTFIX.

(Please note: Our normal process is to give advanced warning of this
occurring, but we forgot to do that. A thousand apologies.)

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, feel free to reopen
this bug and simply change the 'version' to a later Fedora version.

Bug Reporter: Thank you for reporting this issue and we are sorry that
we were unable to fix it before Fedora 15 reached end of life. If you
would still like to see this bug fixed and are able to reproduce it
against a later version of Fedora, you are encouraged to click on
"Clone This Bug" (top right of this page) and open it against that
version of Fedora.

Although we aim to fix as many bugs as possible during every release's
lifetime, sometimes those efforts are overtaken by events. Often a
more recent Fedora release includes newer upstream software that fixes
bugs or makes them obsolete.

The process we are following is described here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

[Michael Vogt (mvo)]

Thanks Marc, indeed you are right :)

[Roger Crew (crew-4)]

> I'm not sure there are any left though.

There are. As of today 5/11/2013, Verisign's md2 certificate *is* still in use,
i.e., still being sent out as part of the cert chain at actual sites,
see e.g., pip.verisignlabs.com -- which is Verisign/Symantec's openid provider
and thus may be particularly widely used.

and causing connections to fail. My own particular trouble is with Perl's IO::Socket::SSL but
openssl fails as well, even with the certificate path/file explicitly specified:

openssl s_client -showcerts -tls1 -CApath /etc/ssl/certs -connect pip.verisignlabs.com:443 </dev/null
openssl s_client -showcerts -tls1 -CAfile /etc/ssl/certs/ca-certificates.crt -connect pip.verisignlabs.com:443 </dev/null

both result in

    Verify return code: 20 (unable to get local issuer certificate)

Adding in the md2 certificate as a locally trusted certificate fixed things.
This being on a fresh upgrade of Debian 7.0 (wheezy) ca-certficates version 20130119

Also, there's another certificate from "Startcom Certification Authority" that's in a similar boat
 i.e., two versions -- SHA1 and SHA256 --- of the same certificate in the Mozilla bundle, identically named

According to openssl's "verify" manpage, verification is SUPPOSED to fail in situations like these

       The lookup first looks in the list of untrusted certificates and if no match is found the remaining lookups are from the
       trusted certificates.

from which I infer that if a match IS found in the supplied chain then there won't BE a lookup in the certificate store and the actual bona fide issuer for Verisign's "G5" subject (the root certificate self-signed by "G5") will never actually be found. I.e., there's no way to know a priori that the supposedly intermediate "G5" actually *is* backed by a known root certificate unless we're going to look up *every* certificate we get in the root store.

So I don't see how this can be an openssl bug unless openssl is interpreting the spec wrong (and I have no idea about that other than that it seems silly to require looking up every certificate received in the root store on the off-chance that it might be a root and the remote server maintainer who is the primary user of the certificate somehow didn't realize it...)

... which would then mean there needs to be a way to include the m2 certificate in the root store somehow

[Launchpad Janitor (janitor)]

This bug was fixed in the package ca-certificates - 20130906ubuntu0.13.10.1

---------------
ca-certificates (20130906ubuntu0.13.10.1) saucy-security; urgency=medium

  * Update ca-certificates database to 20130906 (LP: #1257265):
    - backport changes from the Ubuntu 14.04 20130906ubuntu1 package
    - No longer ship cacert.org certificates (LP: #1258286)
    - mozilla/certdata2pem.py: Work around openssl issue by shipping both
      versions of the same signed roots. Previously, the script would
      simply overwrite the first one found in the certdata.txt with the
      later one since they both have the same CKA_LABEL, resulting in
      identical filenames. (LP: #1014640, LP: #1031333)
 -- Marc Deslauriers <email address hidden> Thu, 06 Feb 2014 17:04:56 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package ca-certificates - 20130906ubuntu0.12.04.1

---------------
ca-certificates (20130906ubuntu0.12.04.1) precise-security; urgency=medium

  * Update ca-certificates database to 20130906 (LP: #1257265):
    - backport changes from the Ubuntu 14.04 20130906ubuntu1 package
    - No longer ship cacert.org certificates (LP: #1258286)
    - No longer ship obsolete debconf.org certificates
    - mozilla/certdata2pem.py: Work around openssl issue by shipping both
      versions of the same signed roots. Previously, the script would
      simply overwrite the first one found in the certdata.txt with the
      later one since they both have the same CKA_LABEL, resulting in
      identical filenames. (LP: #1014640, LP: #1031333)
 -- Marc Deslauriers <email address hidden> Thu, 06 Feb 2014 17:39:43 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package ca-certificates - 20130906ubuntu0.10.04.1

---------------
ca-certificates (20130906ubuntu0.10.04.1) lucid-security; urgency=medium

  * Update ca-certificates database to 20130906 (LP: #1257265, LP: #1271357):
    - backport changes from the Ubuntu 14.04 20130906ubuntu1 package
    - No longer ship cacert.org certificates (LP: #1258286)
    - No longer ship obsolete debconf.org certificates
    - No longer ship expired brasil.gov.br certificates
    - No longer ship expired signet.pl certificates
    - No longer ship gouv.fr certificates, now part of mozilla bundle
    - No longer ship telesec.de certificates, now part of mozilla bundle
    - mozilla/certdata2pem.py: Work around openssl issue by shipping both
      versions of the same signed roots. Previously, the script would
      simply overwrite the first one found in the certdata.txt with the
      later one since they both have the same CKA_LABEL, resulting in
      identical filenames. (LP: #1014640, LP: #1031333)
 -- Marc Deslauriers <email address hidden> Fri, 07 Feb 2014 13:58:53 -0500