c_rehash creating bogus links to ca-certificates.crt

Bug #854927 reported by Scott Moser on 2011-09-20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
ca-certificates (Debian)
Fix Released
ca-certificates (Ubuntu)
Loïc Minier
Loïc Minier
openssl (Ubuntu)
Loïc Minier
Loïc Minier

Bug Description

$ wget https://www.google.com
--2011-09-20 18:12:46-- https://www.google.com/
Resolving www.google.com...,,, ...
Connecting to www.google.com||:443... connected.
ERROR: cannot verify www.google.com's certificate, issued by `/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA':
  Unable to locally verify the issuer's authority.
To connect to www.google.com insecurely, use `--no-check-certificate'.

$ curl -sS https://launchpad.net
curl: (35) error:0B07C065:x509 certificate routines:X509_STORE_add_cert:cert already in hash table

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: openssl 1.0.0e-2ubuntu1
ProcVersionSignature: User Name 3.0.0-11.18-virtual 3.0.4
Uname: Linux 3.0.0-11-virtual i686
ApportVersion: 1.23-0ubuntu1
Architecture: i386
Date: Tue Sep 20 18:11:11 2011
Ec2AMI: ami-00000090
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
SourcePackage: openssl
UpgradeStatus: No upgrade log present (probably fresh install)

Scott Moser (smoser) wrote :

Marking as confirmed.

Changed in openssl (Ubuntu):
status: New → Confirmed
Changed in openssl (Ubuntu):
importance: Undecided → High
tags: added: rls-mgr-o-tracking
Changed in openssl (Ubuntu Oneiric):
milestone: none → ubuntu-11.10-beta-2
Changed in openssl (Ubuntu Oneiric):
status: Confirmed → In Progress
Steve Langasek (vorlon) wrote :

This has been traced to a broken hash directory:

11:40 < kirkland> lrwxrwxrwx 1 root root 19 2011-09-20 01:34 /usr/lib/ssl/certs/55a10908.0 -> ca-certificates.crt
11:40 < kirkland> -rw-r--r-- 1 root root 240312 2011-09-20 01:32 /usr/lib/ssl/certs/ca-certificates.crt

This is expected to point to the specific certificate file, ValiCert_Class_2_VA.pem, instead; but on new installs since the latest upload of the new upstream version of openssl, c_rehash is giving preference to the ca-certificates bundle file over the individual cert files, and libssl subsequently is unable to use ca-certificates.crt for certificate validation.

I would definitely say there's a bug in openssl here, since c_rehash shouldn't create symlinks that the library will be subsequently unable to use; but I think we can work around it in ca-certificates by just making sure the bundle file is moved out of the way at the time we're calling c_rehash - since any time we call c_rehash we're regenerating that bundle file anyway.

Changed in ca-certificates (Ubuntu Oneiric):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Steve Langasek (vorlon)
milestone: none → ubuntu-11.10-beta-2
Changed in openssl (Ubuntu Oneiric):
milestone: ubuntu-11.10-beta-2 → ubuntu-11.10
status: In Progress → Triaged
assignee: nobody → Colin Watson (cjwatson)
Steve Langasek (vorlon) wrote :

12:41 < sbeattie> slangasek: behavior change in c_rehash is likely due to the patch that fixed

summary: - wget, curl can't verify certificates
+ c_rehash creating bogus links to ca-certificates.crt
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ca-certificates - 20110502+nmu1ubuntu2

ca-certificates (20110502+nmu1ubuntu2) oneiric; urgency=low

  * Really only call --fresh on upgrade, instead of all the time; thanks to
    Adam Conrad for catching this in the queue.

ca-certificates (20110502+nmu1ubuntu1) oneiric; urgency=low

  * sbin/update-ca-certificates: move the ca-certificates.crt bundle out of
    the way before calling c_rehash, so that symlinks don't accidentally get
    pointed here, breaking openssl certificate verification. LP: #854927.
  * debian/postinst: kludge in support for running
    update-ca-certificates --fresh on upgrade, to ensure we fix up the hash
    for anyone who happened to install from a daily.
 -- Steve Langasek <email address hidden> Tue, 20 Sep 2011 12:49:57 -0700

Changed in ca-certificates (Ubuntu Oneiric):
status: In Progress → Fix Released
Steve Beattie (sbeattie) wrote :

Following up on the irc comment that Steve Langasek pasted, I can confirm that reverting the patch http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/oneiric/openssl/oneiric/revision/58#debian/patches/c_rehash-multi.patch followed by update-ca-certificates --fresh (without the workaround Steve added) also corrects the hashing/verification issue. However, it does seem like the c_rehash patch is correcting undesirable behavior on its part.

Colin Watson (cjwatson) wrote :

Is this really the entirety of the bug? With the new openssl but the old ca-certificates, I ran:

  $ sudo update-ca-certificates --fresh
  $ ls -l /usr/lib/ssl/certs/55a10908.0
lrwxrwxrwx 1 root root 19 2011-09-21 13:27 /usr/lib/ssl/certs/55a10908.0 -> ca-certificates.crt
  $ curl -sS http://launchpad.net
  <title>301 Moved Permanently</title>
  <h1>Moved Permanently</h1>
  <p>The document has moved <a href="https://launchpad.net/">here</a>.</p>
  <address>Apache/2.2.14 (Ubuntu) Server at launchpad.net Port 80</address>

What am I missing? While we could certainly change c_rehash to make sure it always prefers .pem files over .crt (and that might be preferable anyway), I wonder why libssl is unable to deal with the .crt files ...

Marc Deslauriers (mdeslaur) wrote :

What exactly are you trying to show Colin? You're connecting to http...

Colin Watson (cjwatson) wrote :

Whoops, I'm also unwell today and not thinking clearly. But in any case HTTPS works too:

$ wget https://www.google.com
--2011-09-21 14:52:14-- https://www.google.com/
Resolving www.google.com...,,, ...
Connecting to www.google.com||:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://encrypted.google.com/ [following]
--2011-09-21 14:52:14-- https://encrypted.google.com/
Resolving encrypted.google.com...,,, ...
Connecting to encrypted.google.com||:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html'

    [ <=> ] 11,434 --.-K/s in 0.07s

2011-09-21 14:52:15 (165 KB/s) - `index.html' saved [11434]

$ curl -sS https://launchpad.net
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

Colin Watson (cjwatson) wrote :

Should be fixed by Loïc's recent change:

openssl (1.0.0e-2ubuntu2) oneiric; urgency=low

  * Unapply patch c_rehash-multi and comment it out in the series as it breaks
    parsing of certificates with CRLF line endings and other cases (see
    Debian #642314 for discussion), it also changes the semantics of c_rehash
    directories by requiring applications to parse hash link targets as files
    containing potentially *multiple* certificates rather than exactly one.
    LP: #855454.

 -- Loïc Minier <email address hidden> Tue, 27 Sep 2011 18:13:07 +0200

Changed in openssl (Ubuntu Oneiric):
assignee: Colin Watson (cjwatson) → Loïc Minier (lool)
status: Triaged → Fix Released
Loïc Minier (lool) wrote :

While this wont happen with current ca-certificates, I think we should revert the changes which caused this bug:
in Debian's 20110421 QA upload, a c_rehash call was added to postinst for upgrades from versions <= 20090814+nmu3, this was an attempt to rebuild the symlinks in /etc/ssl/certs, but because update-ca-certificates wasn't removing /etc/ssl/cert/ca-certificates.crt, it did generate one symlink to this file for the first certificate. With the Debian change from openssl 1.0.0e-1 to support multiple certificates in one file, this probably took even worse proportions. However this probably depended on the order in which c_rehash processed files; it just does readdir() and generates links for the first certificate of each .pem and .crt file it finds.

Now in 20110502+nmu1ubuntu1/20110502+nmu1ubuntu2, a call was added to properly regenerate the links, but kept the plain c_rehash call *after* it in the postinst, so that it might trigger when upgrading from <= 20090814+nmu3 (so upgrades from natty or lucid will cause this).

Because of the new call I've added in20110502+nmu1ubuntu4 to regenerates certs when upgrading from <= 20110502+nmu1ubuntu4, this should be fixed for oneiric users.

Now, what needs to be fixed:
* plain c_rehash is wrong in any case; also an issue in Debian (and the rm needs to be copied there too)
* postinst has tons of update-ca-certificates calls, mine is the strongest one as it affects all updates (from natty); all of these should be dropped after oneiric

Now this could be fixed in oneiric + 1, but it would be clearer to remove these now to prevent any regression when removing the postinst snippets (e.g. leaving the plain c_rehash call alone after oneiric would be wrong).

Changed in ca-certificates (Ubuntu Oneiric):
assignee: Steve Langasek (vorlon) → Loïc Minier (lool)
milestone: ubuntu-11.10-beta-2 → none
status: Fix Released → Triaged
Loïc Minier (lool) wrote :

I've sent a patch to Debian including Steve's changes to remove ca-certificates.crt before running c_rehash in update-ca-certificates; will set bug id once I have it.

Changed in ca-certificates (Ubuntu Oneiric):
status: Triaged → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ca-certificates - 20110502+nmu1ubuntu5

ca-certificates (20110502+nmu1ubuntu5) oneiric; urgency=low

  * Tweak postinst to not run update-ca-certificates multiple times and remove
    dangerous plain c_rehash snippet; LP: #854927.
 -- Loic Minier <email address hidden> Wed, 28 Sep 2011 15:49:34 +0200

Changed in ca-certificates (Ubuntu Oneiric):
status: Fix Committed → Fix Released
Loïc Minier (lool) on 2011-09-28
Changed in ca-certificates (Debian):
importance: Undecided → Unknown
status: New → Unknown
Changed in ca-certificates (Debian):
status: Unknown → New
Changed in ca-certificates (Debian):
status: New → Fix Committed
Changed in ca-certificates (Debian):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.