During update-status hook: "Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ssl/certs/ca-certificates.crt" can happen

Bug #1762431 reported by Nobuto Murata on 2018-04-09
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Charm Helpers
High
David Ames
OpenStack cinder charm
High
David Ames
OpenStack glance charm
High
David Ames
OpenStack heat charm
High
David Ames
OpenStack keystone charm
High
David Ames
OpenStack neutron-api charm
High
David Ames
OpenStack nova-cloud-controller charm
High
David Ames
OpenStack openstack-dashboard charm
High
David Ames

Bug Description

Charm revision: latest stable = #258
https://api.jujucharms.com/charmstore/v5/neutron-api-258/archive/repo-info

I'm seeing the following error sometimes when running a continuous series of tests against Neutron services. e.g. creating/deleting multiple networks continuously. Failure rate is around 2%.

"InternalServerError: The server has either erred or is incapable of performing the requested operation."

From neutron-server.log, it looks like the internal error was caused by IOError to /etc/ssl/certs/ca-certificates.crt, which is weird because the OpenStack is already deployed and the SSL related configurations haven't been changed sine then.

====
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 407, in send
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors self.cert_verify(conn, request.url, verify, cert)
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 226, in cert_verify
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors "invalid path: {0}".format(cert_loc))
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors IOError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ssl/certs/ca-certificates.crt
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors
2018-04-09 13:27:21.311 1216754 INFO neutron.wsgi [req-ceadfa7e-7d9c-417a-b5c6-1ccddabd2699 a0cff814cd2f4256b4de5677c5815baf 36a28cee612e4f00b53842403552f0ce - a9b936a8334a4d9699ba9c26c0c20668 a9b936a8334a4d9699ba9c26c0c20668] 10.X.Y.Z,127.0.0.1 "DELETE /v2.0/security-groups/87b70810-52bb-46d1-af7e-f9e3ea40f606 HTTP/1.1" status: 500 len: 404 time: 0.0044849
====

When the IOError happens, Juju/Charms are touching /etc/ssl/certs with update-status hook.

====
2018-04-09 13:27:18 INFO juju-log Installing new CA cert
2018-04-09 13:27:18 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:27:19 DEBUG update-status done.
2018-04-09 13:27:19 DEBUG update-status Updating certificates in /etc/ssl/certs...
2018-04-09 13:27:22 DEBUG update-status 149 added, 0 removed; done.
2018-04-09 13:27:22 DEBUG update-status Running hooks in /etc/ca-certificates/update.d...
2018-04-09 13:27:22 DEBUG update-status done.
2018-04-09 13:27:22 DEBUG update-status Considering dependency setenvif for ssl:
2018-04-09 13:27:22 DEBUG update-status Module setenvif already enabled
2018-04-09 13:27:22 DEBUG update-status Considering dependency mime for ssl:
2018-04-09 13:27:22 DEBUG update-status Module mime already enabled
2018-04-09 13:27:22 DEBUG update-status Considering dependency socache_shmcb for ssl:
2018-04-09 13:27:22 DEBUG update-status Module socache_shmcb already enabled
2018-04-09 13:27:22 DEBUG update-status Module ssl already enabled
2018-04-09 13:27:22 DEBUG update-status Module proxy already enabled
2018-04-09 13:27:22 DEBUG update-status Considering dependency proxy for proxy_http:
2018-04-09 13:27:22 DEBUG update-status Module proxy already enabled
2018-04-09 13:27:22 DEBUG update-status Module proxy_http already enabled
2018-04-09 13:27:22 DEBUG update-status Module headers already enabled
====

1. update-status hook shouldn't change the files. It should be read-only.
2. Even with other hooks, the charm shouldn't touch /etc/ssl/certs or /etc/ssl/certs/ca-certificates.crt if SSL related configurations are unchanged. Touching /etc/ssl/certs causes the service disruption as you see above.

Nobuto Murata (nobuto) wrote :

The API service is affected every 5 minutes.

$ grep 'Clearing symlinks in /etc/ssl/certs' /var/log/juju/unit-neutron-api-0.log
2018-04-09 11:57:11 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 11:57:49 DEBUG config-changed Clearing symlinks in /etc/ssl/certs...
2018-04-09 11:57:58 DEBUG config-changed Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:01:33 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:06:50 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:11:48 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:15:54 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:19:58 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:26:06 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:31:14 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:35:53 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:41:54 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:46:50 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:51:18 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:56:25 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:00:25 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:05:47 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:11:18 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:17:03 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:21:22 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:27:18 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:32:47 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:37:43 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:43:23 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:47:21 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:52:34 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:56:42 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 14:00:52 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 14:05:54 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 14:10:37 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 14:14:39 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 14:20:28 DEBUG update-status Clearing symlinks in /etc/ssl/certs...

description: updated
Nobuto Murata (nobuto) wrote :

Subscribing ~field-high since the API disruption in every 5 minutes is hard to be justified.

Nobuto Murata (nobuto) wrote :

https://github.com/openstack/charm-neutron-api/blob/stable/18.02/hooks/charmhelpers/contrib/hahelpers/apache.py#L84-L95

$ sha256sum /usr/local/share/ca-certificates/keystone_juju_ca_cert.crt
f2d19eb104885f6911cae938a34a10f5466df4f6ffef64b0871800ef1f65b0ff /usr/local/share/ca-certificates/keystone_juju_ca_cert.crt

$ head -n1 /usr/local/share/ca-certificates/keystone_juju_ca_cert.crt
-----BEGIN CERTIFICATE-----

^^^ plain text.

https://github.com/openstack/charm-neutron-api/blob/stable/18.02/hooks/charmhelpers/contrib/openstack/context.py#L803-L806
https://github.com/openstack/charm-neutron-api/blob/stable/18.02/hooks/charmhelpers/contrib/hahelpers/apache.py#L63-L73

$ juju run --unit neutron-api/0 'config-get ssl_ca | base64 -d | sha256sum'
f2d19eb104885f6911cae938a34a10f5466df4f6ffef64b0871800ef1f65b0ff -

^^^ with base64 decode, hashsum matches.

> if old_cert and old_cert == ca_cert:
> log("CA cert is the same as installed version", level=INFO)

Wondering why those do not match for 2. below.

> 1. update-status hook shouldn't change the files. It should be read-only.
> 2. Even with other hooks, the charm shouldn't touch /etc/ssl/certs or /etc/ssl/certs/ca-certificates.crt if SSL related configurations are unchanged. Touching /etc/ssl/certs causes the service disruption as you see above.

1. is still valid though.

Nobuto Murata (nobuto) wrote :

The reason behind 2. seems as follows.

(Pdb) l
 85 import pdb; pdb.set_trace()
 86 if ca_cert:
 87 cert_file = ('/usr/local/share/ca-certificates/'
 88 'keystone_juju_ca_cert.crt')
 89 old_cert = retrieve_ca_cert(cert_file)
 90 -> if old_cert and old_cert == ca_cert:
 91 log("CA cert is the same as installed version", level=INFO)
 92 else:
 93 log("Installing new CA cert", level=INFO)
 94 with open(cert_file, 'wb') as crt:
 95 crt.write(ca_cert)
(Pdb) old_cert
'-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n'
(Pdb) ca_cert
b'-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n'

^^^ certs are identical, but the diff if plain text vs binary.

Nobuto Murata (nobuto) wrote :

Ok, according to Alex Kavanagh, it's a known issue in charm-helpers.
https://github.com/juju/charm-helpers/issues/86

Alex Kavanagh (ajkavanagh) wrote :

I think this bug is responsible for the issue: https://github.com/juju/charm-helpers/issues/86

tags: added: 4010
David Ames (thedac) wrote :

The charm-helpers change is near trivial as suggested by Tilman:
https://github.com/juju/charm-helpers/pull/146

I will also add the other charms potentially affected. They will all require at minimum a charm-helpers sync.

We should look at keystone's handling of certificates as well. Although, the keystone charm is currently python 2 it is meant to go to python 3 soon.

Changed in charm-helpers:
status: New → Triaged
status: Triaged → In Progress
importance: Undecided → High
assignee: nobody → David Ames (thedac)
milestone: none → 1.0
milestone: 1.0 → none
Changed in charm-neutron-api:
importance: Undecided → High
assignee: nobody → David Ames (thedac)
milestone: none → 18.05

Fix proposed to branch: master
Review: https://review.openstack.org/559814

Changed in charm-neutron-api:
status: New → In Progress
David Ames (thedac) wrote :

@Nobutu Can you please test cs:~thedac/neutron-api-4 and see if that fixes things or if we have other problems lurking?

Reviewed: https://review.openstack.org/559814
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=522951870272f6b9036b2549f189851de11b2fb4
Submitter: Zuul
Branch: master

commit 522951870272f6b9036b2549f189851de11b2fb4
Author: David Ames <email address hidden>
Date: Mon Apr 9 11:43:24 2018 -0700

    Charm-helpers sync to fix CA cert comparison

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Change-Id: Icaf99f6a069c152f901210c38b26e71622560693
    Closes-Bug: #1762431

Changed in charm-neutron-api:
status: In Progress → Fix Committed
Nobuto Murata (nobuto) wrote :

With ~openstack-charmers-next/neutron-api-337, deployment succeeds and the CA is considered the same as existing one properly.
https://api.jujucharms.com/charmstore/v5/~openstack-charmers-next/neutron-api-337/archive/repo-info

# grep 'CA cert' /var/log/juju/unit-neutron-api-0.log
2018-04-11 05:53:14 INFO juju-log Installing new CA cert
...
2018-04-11 13:10:42 INFO juju-log CA cert is the same as installed version
2018-04-11 13:16:18 INFO juju-log CA cert is the same as installed version
2018-04-11 13:21:10 INFO juju-log CA cert is the same as installed version
2018-04-11 13:25:46 INFO juju-log CA cert is the same as installed version
2018-04-11 13:30:16 INFO juju-log CA cert is the same as installed version
2018-04-11 13:36:07 INFO juju-log CA cert is the same as installed version

David Ames (thedac) on 2018-04-11
Changed in charm-helpers:
status: In Progress → Fix Committed
David Ames (thedac) on 2018-04-11
Changed in charm-cinder:
status: New → Triaged
importance: Undecided → High
assignee: nobody → David Ames (thedac)
milestone: none → 18.05
Changed in charm-glance:
assignee: nobody → David Ames (thedac)
milestone: none → 18.05
Changed in charm-heat:
assignee: nobody → David Ames (thedac)
milestone: none → 18.05
Changed in charm-keystone:
assignee: nobody → David Ames (thedac)
milestone: none → 18.05
Changed in charm-neutron-api:
assignee: David Ames (thedac) → nobody
Changed in charm-nova-cloud-controller:
assignee: nobody → David Ames (thedac)
Changed in charm-neutron-api:
assignee: nobody → David Ames (thedac)
Changed in charm-openstack-dashboard:
assignee: nobody → David Ames (thedac)
Changed in charm-nova-cloud-controller:
milestone: none → 18.05
Changed in charm-openstack-dashboard:
milestone: none → 18.05
Changed in charm-nova-cloud-controller:
importance: Undecided → High
Changed in charm-openstack-dashboard:
importance: Undecided → High
Changed in charm-keystone:
importance: Undecided → High
Changed in charm-heat:
importance: Undecided → High
Changed in charm-glance:
importance: Undecided → High

Reviewed: https://review.openstack.org/560655
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=92f5248a072505315546a9d28f0c0739adb03e00
Submitter: Zuul
Branch: master

commit 92f5248a072505315546a9d28f0c0739adb03e00
Author: David Ames <email address hidden>
Date: Wed Apr 11 14:19:29 2018 -0700

    Charm-helpers sync to fix CA cert comparison

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Closes-Bug: #1762431

    Change-Id: Ic226149cc124ac5b84ab30d95a590f08489c67f2

Changed in charm-keystone:
status: New → Fix Committed

Reviewed: https://review.openstack.org/560657
Committed: https://git.openstack.org/cgit/openstack/charm-glance/commit/?id=e581cbafc95670653cc8102edd2bf38a0d9b4822
Submitter: Zuul
Branch: master

commit e581cbafc95670653cc8102edd2bf38a0d9b4822
Author: David Ames <email address hidden>
Date: Wed Apr 11 14:22:15 2018 -0700

    Charm-helpers sync to fix CA cert comparison

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Closes-Bug: #1762431

    Change-Id: I65f57573f0b24094b5db8b6854e26146e1b78455

Changed in charm-glance:
status: New → Fix Committed

Reviewed: https://review.openstack.org/560660
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=794b8ed0bb9f908ca7c12362cdcba6c3b24f5dc8
Submitter: Zuul
Branch: master

commit 794b8ed0bb9f908ca7c12362cdcba6c3b24f5dc8
Author: David Ames <email address hidden>
Date: Wed Apr 11 14:26:49 2018 -0700

    Charm-helpers sync to fix CA cert comparison

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Closes-Bug: #1762431

    Change-Id: Ie2348a83671b9636bd94227e903b1a50bff7aecc

Changed in charm-nova-cloud-controller:
status: New → Fix Committed

Reviewed: https://review.openstack.org/560656
Committed: https://git.openstack.org/cgit/openstack/charm-cinder/commit/?id=af394fc9e5560bc91201ba1bd0fbb1a74c7d214b
Submitter: Zuul
Branch: master

commit af394fc9e5560bc91201ba1bd0fbb1a74c7d214b
Author: David Ames <email address hidden>
Date: Wed Apr 11 14:21:04 2018 -0700

    Charm-helpers sync to fix CA cert comparison

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Change-Id: Ie591ab8c958b59ef190a2f255eee8db0338d5ee3
    Closes-Bug: #1762431

Changed in charm-cinder:
status: Triaged → Fix Committed

Reviewed: https://review.openstack.org/560661
Committed: https://git.openstack.org/cgit/openstack/charm-openstack-dashboard/commit/?id=0707590487376b8c45d3a9f056b427081f2b138a
Submitter: Zuul
Branch: master

commit 0707590487376b8c45d3a9f056b427081f2b138a
Author: David Ames <email address hidden>
Date: Wed Apr 11 14:28:26 2018 -0700

    Charm-helpers sync to fix CA cert comparison

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Change-Id: I3b233a79689858db0962bafe71eba0a0667c2bf0
    Closes-Bug: #1762431

Changed in charm-openstack-dashboard:
status: New → Fix Committed

Reviewed: https://review.openstack.org/560659
Committed: https://git.openstack.org/cgit/openstack/charm-heat/commit/?id=5f8d3317e6dd9b67861bab10258f0290281339fc
Submitter: Zuul
Branch: master

commit 5f8d3317e6dd9b67861bab10258f0290281339fc
Author: David Ames <email address hidden>
Date: Wed Apr 11 14:23:17 2018 -0700

    Charm-helpers sync to fix CA cert comparison

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Change-Id: I1271ff7dc7fd2ebc0327801cd97d62971ff82e94
    Closes-Bug: #1762431

Changed in charm-heat:
status: New → Fix Committed

Reviewed: https://review.openstack.org/560421
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=6620eeb340a66bec842813add8e31ce9a33527d7
Submitter: Zuul
Branch: stable/18.02

commit 6620eeb340a66bec842813add8e31ce9a33527d7
Author: David Ames <email address hidden>
Date: Mon Apr 9 11:43:24 2018 -0700

    Charm-helpers sync to fix CA cert comparison

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Change-Id: Icaf99f6a069c152f901210c38b26e71622560693
    Closes-Bug: #1762431
    (cherry picked from commit 522951870272f6b9036b2549f189851de11b2fb4)

Frode Nordahl (fnordahl) on 2018-04-20
Changed in charm-neutron-api:
milestone: 18.05 → 18.02
status: Fix Committed → Fix Released
tags: added: canonical-bootstack
Drew Freiberger (afreiberger) wrote :

This is field-critical for production live bootstack customers. Valid L2 issues on heat stacks as heat is constantly trying to manage stacks and will hit this bug with worse consequences than other apis that are user or program retry-able.

Felipe Reyes (freyes) on 2018-05-07
tags: added: sts
Felipe Reyes (freyes) on 2018-05-08
tags: added: backport-potential stable-backport

Reviewed: https://review.openstack.org/566734
Committed: https://git.openstack.org/cgit/openstack/charm-heat/commit/?id=c3fcab00c93f9f180b0859a590e9d73f82bf66e0
Submitter: Zuul
Branch: stable/18.02

commit c3fcab00c93f9f180b0859a590e9d73f82bf66e0
Author: David Ames <email address hidden>
Date: Mon May 7 17:42:20 2018 -0300

    Read in ca certificate as binary for PY3

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services. The writing of the file was changed to bytes
    in https://github.com/juju/charm-helpers/pull/58/.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Change-Id: Iedcf0b40004c387384a071c5ad4116e6eb33fd90
    Closes-Bug: #1762431

Ryan Beisner (1chb1n) on 2018-05-10
Changed in charm-heat:
status: Fix Committed → Fix Released

Reviewed: https://review.openstack.org/566763
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=9053e73d209387f6f90603694246df5d6d3dee76
Submitter: Zuul
Branch: stable/18.02

commit 9053e73d209387f6f90603694246df5d6d3dee76
Author: David Ames <email address hidden>
Date: Mon May 7 20:59:46 2018 -0300

    Read in ca certificate as binary for PY3

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services. The writing of the file was changed to bytes
    in https://github.com/juju/charm-helpers/pull/58/.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Change-Id: I64eb2b90776bb1e37de009208f016f176bd6a82a
    Closes-Bug: #1762431

Felipe Reyes (freyes) on 2018-05-16
Changed in charm-keystone:
status: Fix Committed → Fix Released
Jason Hobbs (jason-hobbs) wrote :

What is the plan for releasing this? Some of the charms are targetted for 18.05 and some are for 18.02.

David Ames (thedac) on 2018-06-11
Changed in charm-cinder:
status: Fix Committed → Fix Released
Changed in charm-glance:
status: Fix Committed → Fix Released
Changed in charm-nova-cloud-controller:
status: Fix Committed → Fix Released
Changed in charm-openstack-dashboard:
status: Fix Committed → Fix Released
David Ames (thedac) on 2018-06-14
Changed in charm-helpers:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.