Comment 0 for bug 1762431

I'm seeing the following error sometimes when running a continuous series of tests against Neutron services. e.g. creating/deleting multiple networks continuously. Failure rate is around 2%.

"InternalServerError: The server has either erred or is incapable of performing the requested operation."

From neutron-server.log, it looks like the internal error was caused by IOError to /etc/ssl/certs/ca-certificates.crt, which is weird because the OpenStack is already deployed and the SSL related configurations haven't been changed sine then.

====
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 407, in send
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors self.cert_verify(conn, request.url, verify, cert)
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 226, in cert_verify
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors "invalid path: {0}".format(cert_loc))
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors IOError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ssl/certs/ca-certificates.crt
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors
2018-04-09 13:27:21.311 1216754 INFO neutron.wsgi [req-ceadfa7e-7d9c-417a-b5c6-1ccddabd2699 a0cff814cd2f4256b4de5677c5815baf 36a28cee612e4f00b53842403552f0ce - a9b936a8334a4d9699ba9c26c0c20668 a9b936a8334a4d9699ba9c26c0c20668] 10.X.Y.Z,127.0.0.1 "DELETE /v2.0/security-groups/87b70810-52bb-46d1-af7e-f9e3ea40f606 HTTP/1.1" status: 500 len: 404 time: 0.0044849
====

When the IOError happens, Juju/Charms are touching /etc/ssl/certs with update-status hook.

====
2018-04-09 13:27:18 INFO juju-log Installing new CA cert
2018-04-09 13:27:18 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:27:19 DEBUG update-status done.
2018-04-09 13:27:19 DEBUG update-status Updating certificates in /etc/ssl/certs...
2018-04-09 13:27:22 DEBUG update-status 149 added, 0 removed; done.
2018-04-09 13:27:22 DEBUG update-status Running hooks in /etc/ca-certificates/update.d...
2018-04-09 13:27:22 DEBUG update-status done.
2018-04-09 13:27:22 DEBUG update-status Considering dependency setenvif for ssl:
2018-04-09 13:27:22 DEBUG update-status Module setenvif already enabled
2018-04-09 13:27:22 DEBUG update-status Considering dependency mime for ssl:
2018-04-09 13:27:22 DEBUG update-status Module mime already enabled
2018-04-09 13:27:22 DEBUG update-status Considering dependency socache_shmcb for ssl:
2018-04-09 13:27:22 DEBUG update-status Module socache_shmcb already enabled
2018-04-09 13:27:22 DEBUG update-status Module ssl already enabled
2018-04-09 13:27:22 DEBUG update-status Module proxy already enabled
2018-04-09 13:27:22 DEBUG update-status Considering dependency proxy for proxy_http:
2018-04-09 13:27:22 DEBUG update-status Module proxy already enabled
2018-04-09 13:27:22 DEBUG update-status Module proxy_http already enabled
2018-04-09 13:27:22 DEBUG update-status Module headers already enabled
====

1. update-status hook shouldn't change the files. It should be read-only.
2. Even with other hooks, the charm shouldn't touch /etc/ssl/certs or /etc/ssl/certs/ca-certificates.crt if SSL related configurations are unchanged. Touching /etc/ssl/certs causes the service disruption as you see above.