During update-status hook: "Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ssl/certs/ca-certificates.crt" can happen
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Charm Helpers |
Fix Released
|
High
|
David Ames | ||
OpenStack Cinder Charm |
Fix Released
|
High
|
David Ames | ||
OpenStack Dashboard Charm |
Fix Released
|
High
|
David Ames | ||
OpenStack Glance Charm |
Fix Released
|
High
|
David Ames | ||
OpenStack Heat Charm |
Fix Released
|
High
|
David Ames | ||
OpenStack Keystone Charm |
Fix Released
|
High
|
David Ames | ||
OpenStack Neutron API Charm |
Fix Released
|
High
|
David Ames | ||
OpenStack Nova Cloud Controller Charm |
Fix Released
|
High
|
David Ames |
Bug Description
Charm revision: latest stable = #258
https:/
I'm seeing the following error sometimes when running a continuous series of tests against Neutron services. e.g. creating/deleting multiple networks continuously. Failure rate is around 2%.
"InternalServer
From neutron-server.log, it looks like the internal error was caused by IOError to /etc/ssl/
====
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware
2018-04-09 13:27:21.311 1216754 INFO neutron.wsgi [req-ceadfa7e-
====
When the IOError happens, Juju/Charms are touching /etc/ssl/certs with update-status hook.
====
2018-04-09 13:27:18 INFO juju-log Installing new CA cert
2018-04-09 13:27:18 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:27:19 DEBUG update-status done.
2018-04-09 13:27:19 DEBUG update-status Updating certificates in /etc/ssl/certs...
2018-04-09 13:27:22 DEBUG update-status 149 added, 0 removed; done.
2018-04-09 13:27:22 DEBUG update-status Running hooks in /etc/ca-
2018-04-09 13:27:22 DEBUG update-status done.
2018-04-09 13:27:22 DEBUG update-status Considering dependency setenvif for ssl:
2018-04-09 13:27:22 DEBUG update-status Module setenvif already enabled
2018-04-09 13:27:22 DEBUG update-status Considering dependency mime for ssl:
2018-04-09 13:27:22 DEBUG update-status Module mime already enabled
2018-04-09 13:27:22 DEBUG update-status Considering dependency socache_shmcb for ssl:
2018-04-09 13:27:22 DEBUG update-status Module socache_shmcb already enabled
2018-04-09 13:27:22 DEBUG update-status Module ssl already enabled
2018-04-09 13:27:22 DEBUG update-status Module proxy already enabled
2018-04-09 13:27:22 DEBUG update-status Considering dependency proxy for proxy_http:
2018-04-09 13:27:22 DEBUG update-status Module proxy already enabled
2018-04-09 13:27:22 DEBUG update-status Module proxy_http already enabled
2018-04-09 13:27:22 DEBUG update-status Module headers already enabled
====
1. update-status hook shouldn't change the files. It should be read-only.
2. Even with other hooks, the charm shouldn't touch /etc/ssl/certs or /etc/ssl/
tags: | added: 4010 |
Changed in charm-helpers: | |
status: | In Progress → Fix Committed |
Changed in charm-cinder: | |
status: | New → Triaged |
importance: | Undecided → High |
assignee: | nobody → David Ames (thedac) |
milestone: | none → 18.05 |
Changed in charm-glance: | |
assignee: | nobody → David Ames (thedac) |
milestone: | none → 18.05 |
Changed in charm-heat: | |
assignee: | nobody → David Ames (thedac) |
milestone: | none → 18.05 |
Changed in charm-keystone: | |
assignee: | nobody → David Ames (thedac) |
milestone: | none → 18.05 |
Changed in charm-neutron-api: | |
assignee: | David Ames (thedac) → nobody |
Changed in charm-nova-cloud-controller: | |
assignee: | nobody → David Ames (thedac) |
Changed in charm-neutron-api: | |
assignee: | nobody → David Ames (thedac) |
Changed in charm-openstack-dashboard: | |
assignee: | nobody → David Ames (thedac) |
Changed in charm-nova-cloud-controller: | |
milestone: | none → 18.05 |
Changed in charm-openstack-dashboard: | |
milestone: | none → 18.05 |
Changed in charm-nova-cloud-controller: | |
importance: | Undecided → High |
Changed in charm-openstack-dashboard: | |
importance: | Undecided → High |
Changed in charm-keystone: | |
importance: | Undecided → High |
Changed in charm-heat: | |
importance: | Undecided → High |
Changed in charm-glance: | |
importance: | Undecided → High |
Changed in charm-neutron-api: | |
milestone: | 18.05 → 18.02 |
status: | Fix Committed → Fix Released |
tags: | added: canonical-bootstack |
tags: | added: sts |
tags: | added: backport-potential stable-backport |
Changed in charm-heat: | |
status: | Fix Committed → Fix Released |
Changed in charm-keystone: | |
status: | Fix Committed → Fix Released |
Changed in charm-cinder: | |
status: | Fix Committed → Fix Released |
Changed in charm-glance: | |
status: | Fix Committed → Fix Released |
Changed in charm-nova-cloud-controller: | |
status: | Fix Committed → Fix Released |
Changed in charm-openstack-dashboard: | |
status: | Fix Committed → Fix Released |
Changed in charm-helpers: | |
status: | Fix Committed → Fix Released |
The API service is affected every 5 minutes.
$ grep 'Clearing symlinks in /etc/ssl/certs' /var/log/ juju/unit- neutron- api-0.log
2018-04-09 11:57:11 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 11:57:49 DEBUG config-changed Clearing symlinks in /etc/ssl/certs...
2018-04-09 11:57:58 DEBUG config-changed Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:01:33 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:06:50 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:11:48 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:15:54 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:19:58 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:26:06 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:31:14 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:35:53 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:41:54 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:46:50 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:51:18 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:56:25 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:00:25 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:05:47 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:11:18 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:17:03 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:21:22 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:27:18 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:32:47 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:37:43 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:43:23 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:47:21 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:52:34 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:56:42 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 14:00:52 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 14:05:54 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 14:10:37 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 14:14:39 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 14:20:28 DEBUG update-status Clearing symlinks in /etc/ssl/certs...