| 2018-04-09 14:17:51 |
Nobuto Murata |
bug |
|
|
added bug |
| 2018-04-09 14:20:28 |
Nobuto Murata |
bug |
|
|
added subscriber Canonical Field High |
| 2018-04-09 14:23:24 |
Nobuto Murata |
description |
I'm seeing the following error sometimes when running a continuous series of tests against Neutron services. e.g. creating/deleting multiple networks continuously. Failure rate is around 2%.
"InternalServerError: The server has either erred or is incapable of performing the requested operation."
From neutron-server.log, it looks like the internal error was caused by IOError to /etc/ssl/certs/ca-certificates.crt, which is weird because the OpenStack is already deployed and the SSL related configurations haven't been changed sine then.
====
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 407, in send
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors self.cert_verify(conn, request.url, verify, cert)
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 226, in cert_verify
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors "invalid path: {0}".format(cert_loc))
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors IOError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ssl/certs/ca-certificates.crt
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors
2018-04-09 13:27:21.311 1216754 INFO neutron.wsgi [req-ceadfa7e-7d9c-417a-b5c6-1ccddabd2699 a0cff814cd2f4256b4de5677c5815baf 36a28cee612e4f00b53842403552f0ce - a9b936a8334a4d9699ba9c26c0c20668 a9b936a8334a4d9699ba9c26c0c20668] 10.X.Y.Z,127.0.0.1 "DELETE /v2.0/security-groups/87b70810-52bb-46d1-af7e-f9e3ea40f606 HTTP/1.1" status: 500 len: 404 time: 0.0044849
====
When the IOError happens, Juju/Charms are touching /etc/ssl/certs with update-status hook.
====
2018-04-09 13:27:18 INFO juju-log Installing new CA cert
2018-04-09 13:27:18 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:27:19 DEBUG update-status done.
2018-04-09 13:27:19 DEBUG update-status Updating certificates in /etc/ssl/certs...
2018-04-09 13:27:22 DEBUG update-status 149 added, 0 removed; done.
2018-04-09 13:27:22 DEBUG update-status Running hooks in /etc/ca-certificates/update.d...
2018-04-09 13:27:22 DEBUG update-status done.
2018-04-09 13:27:22 DEBUG update-status Considering dependency setenvif for ssl:
2018-04-09 13:27:22 DEBUG update-status Module setenvif already enabled
2018-04-09 13:27:22 DEBUG update-status Considering dependency mime for ssl:
2018-04-09 13:27:22 DEBUG update-status Module mime already enabled
2018-04-09 13:27:22 DEBUG update-status Considering dependency socache_shmcb for ssl:
2018-04-09 13:27:22 DEBUG update-status Module socache_shmcb already enabled
2018-04-09 13:27:22 DEBUG update-status Module ssl already enabled
2018-04-09 13:27:22 DEBUG update-status Module proxy already enabled
2018-04-09 13:27:22 DEBUG update-status Considering dependency proxy for proxy_http:
2018-04-09 13:27:22 DEBUG update-status Module proxy already enabled
2018-04-09 13:27:22 DEBUG update-status Module proxy_http already enabled
2018-04-09 13:27:22 DEBUG update-status Module headers already enabled
====
1. update-status hook shouldn't change the files. It should be read-only.
2. Even with other hooks, the charm shouldn't touch /etc/ssl/certs or /etc/ssl/certs/ca-certificates.crt if SSL related configurations are unchanged. Touching /etc/ssl/certs causes the service disruption as you see above. |
Charm revision: latest stable = #258
https://api.jujucharms.com/charmstore/v5/neutron-api-258/archive/repo-info
I'm seeing the following error sometimes when running a continuous series of tests against Neutron services. e.g. creating/deleting multiple networks continuously. Failure rate is around 2%.
"InternalServerError: The server has either erred or is incapable of performing the requested operation."
From neutron-server.log, it looks like the internal error was caused by IOError to /etc/ssl/certs/ca-certificates.crt, which is weird because the OpenStack is already deployed and the SSL related configurations haven't been changed sine then.
====
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 407, in send
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors self.cert_verify(conn, request.url, verify, cert)
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 226, in cert_verify
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors "invalid path: {0}".format(cert_loc))
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors IOError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ssl/certs/ca-certificates.crt
2018-04-09 13:27:21.309 1216754 ERROR oslo_middleware.catch_errors
2018-04-09 13:27:21.311 1216754 INFO neutron.wsgi [req-ceadfa7e-7d9c-417a-b5c6-1ccddabd2699 a0cff814cd2f4256b4de5677c5815baf 36a28cee612e4f00b53842403552f0ce - a9b936a8334a4d9699ba9c26c0c20668 a9b936a8334a4d9699ba9c26c0c20668] 10.X.Y.Z,127.0.0.1 "DELETE /v2.0/security-groups/87b70810-52bb-46d1-af7e-f9e3ea40f606 HTTP/1.1" status: 500 len: 404 time: 0.0044849
====
When the IOError happens, Juju/Charms are touching /etc/ssl/certs with update-status hook.
====
2018-04-09 13:27:18 INFO juju-log Installing new CA cert
2018-04-09 13:27:18 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:27:19 DEBUG update-status done.
2018-04-09 13:27:19 DEBUG update-status Updating certificates in /etc/ssl/certs...
2018-04-09 13:27:22 DEBUG update-status 149 added, 0 removed; done.
2018-04-09 13:27:22 DEBUG update-status Running hooks in /etc/ca-certificates/update.d...
2018-04-09 13:27:22 DEBUG update-status done.
2018-04-09 13:27:22 DEBUG update-status Considering dependency setenvif for ssl:
2018-04-09 13:27:22 DEBUG update-status Module setenvif already enabled
2018-04-09 13:27:22 DEBUG update-status Considering dependency mime for ssl:
2018-04-09 13:27:22 DEBUG update-status Module mime already enabled
2018-04-09 13:27:22 DEBUG update-status Considering dependency socache_shmcb for ssl:
2018-04-09 13:27:22 DEBUG update-status Module socache_shmcb already enabled
2018-04-09 13:27:22 DEBUG update-status Module ssl already enabled
2018-04-09 13:27:22 DEBUG update-status Module proxy already enabled
2018-04-09 13:27:22 DEBUG update-status Considering dependency proxy for proxy_http:
2018-04-09 13:27:22 DEBUG update-status Module proxy already enabled
2018-04-09 13:27:22 DEBUG update-status Module proxy_http already enabled
2018-04-09 13:27:22 DEBUG update-status Module headers already enabled
====
1. update-status hook shouldn't change the files. It should be read-only.
2. Even with other hooks, the charm shouldn't touch /etc/ssl/certs or /etc/ssl/certs/ca-certificates.crt if SSL related configurations are unchanged. Touching /etc/ssl/certs causes the service disruption as you see above. |
|
| 2018-04-09 15:20:25 |
Nobuto Murata |
bug task added |
|
charm-helpers |
|
| 2018-04-09 15:21:33 |
Nobuto Murata |
bug watch added |
|
https://github.com/juju/charm-helpers/issues/86 |
|
| 2018-04-09 15:44:55 |
Tilman Baumann |
tags |
cpe-onsite |
4010 cpe-onsite |
|
| 2018-04-09 18:33:02 |
David Ames |
charm-helpers: status |
New |
Triaged |
|
| 2018-04-09 18:33:06 |
David Ames |
charm-helpers: status |
Triaged |
In Progress |
|
| 2018-04-09 18:33:09 |
David Ames |
charm-helpers: importance |
Undecided |
High |
|
| 2018-04-09 18:33:12 |
David Ames |
charm-helpers: assignee |
|
David Ames (thedac) |
|
| 2018-04-09 18:33:13 |
David Ames |
charm-helpers: milestone |
|
1.0 |
|
| 2018-04-09 18:33:17 |
David Ames |
charm-helpers: milestone |
1.0 |
|
|
| 2018-04-09 18:33:24 |
David Ames |
charm-neutron-api: importance |
Undecided |
High |
|
| 2018-04-09 18:33:27 |
David Ames |
charm-neutron-api: assignee |
|
David Ames (thedac) |
|
| 2018-04-09 18:33:30 |
David Ames |
charm-neutron-api: milestone |
|
18.05 |
|
| 2018-04-09 18:36:51 |
David Ames |
bug task added |
|
charm-keystone |
|
| 2018-04-09 18:37:13 |
David Ames |
bug task added |
|
charm-cinder |
|
| 2018-04-09 18:37:34 |
David Ames |
bug task added |
|
charm-glance |
|
| 2018-04-09 18:37:47 |
David Ames |
bug task added |
|
charm-nova-cloud-controller |
|
| 2018-04-09 18:38:03 |
David Ames |
bug task added |
|
charm-openstack-dashboard |
|
| 2018-04-09 18:38:17 |
David Ames |
bug task added |
|
charm-heat |
|
| 2018-04-09 18:44:55 |
OpenStack Infra |
charm-neutron-api: status |
New |
In Progress |
|
| 2018-04-10 20:28:53 |
OpenStack Infra |
charm-neutron-api: status |
In Progress |
Fix Committed |
|
| 2018-04-11 21:18:01 |
David Ames |
charm-helpers: status |
In Progress |
Fix Committed |
|
| 2018-04-11 21:29:25 |
David Ames |
charm-cinder: status |
New |
Triaged |
|
| 2018-04-11 21:29:28 |
David Ames |
charm-cinder: importance |
Undecided |
High |
|
| 2018-04-11 21:29:31 |
David Ames |
charm-cinder: assignee |
|
David Ames (thedac) |
|
| 2018-04-11 21:29:34 |
David Ames |
charm-cinder: milestone |
|
18.05 |
|
| 2018-04-11 21:29:36 |
David Ames |
charm-glance: assignee |
|
David Ames (thedac) |
|
| 2018-04-11 21:29:39 |
David Ames |
charm-glance: milestone |
|
18.05 |
|
| 2018-04-11 21:29:41 |
David Ames |
charm-heat: assignee |
|
David Ames (thedac) |
|
| 2018-04-11 21:29:44 |
David Ames |
charm-heat: milestone |
|
18.05 |
|
| 2018-04-11 21:29:46 |
David Ames |
charm-keystone: assignee |
|
David Ames (thedac) |
|
| 2018-04-11 21:29:48 |
David Ames |
charm-keystone: milestone |
|
18.05 |
|
| 2018-04-11 21:29:51 |
David Ames |
charm-neutron-api: assignee |
David Ames (thedac) |
|
|
| 2018-04-11 21:29:55 |
David Ames |
charm-nova-cloud-controller: assignee |
|
David Ames (thedac) |
|
| 2018-04-11 21:29:57 |
David Ames |
charm-neutron-api: assignee |
|
David Ames (thedac) |
|
| 2018-04-11 21:29:59 |
David Ames |
charm-openstack-dashboard: assignee |
|
David Ames (thedac) |
|
| 2018-04-11 21:30:01 |
David Ames |
charm-nova-cloud-controller: milestone |
|
18.05 |
|
| 2018-04-11 21:30:06 |
David Ames |
charm-openstack-dashboard: milestone |
|
18.05 |
|
| 2018-04-11 21:30:10 |
David Ames |
charm-nova-cloud-controller: importance |
Undecided |
High |
|
| 2018-04-11 21:30:12 |
David Ames |
charm-openstack-dashboard: importance |
Undecided |
High |
|
| 2018-04-11 21:30:14 |
David Ames |
charm-keystone: importance |
Undecided |
High |
|
| 2018-04-11 21:30:15 |
David Ames |
charm-heat: importance |
Undecided |
High |
|
| 2018-04-11 21:30:17 |
David Ames |
charm-glance: importance |
Undecided |
High |
|
| 2018-04-12 08:16:25 |
OpenStack Infra |
charm-keystone: status |
New |
Fix Committed |
|
| 2018-04-12 08:35:37 |
OpenStack Infra |
charm-glance: status |
New |
Fix Committed |
|
| 2018-04-12 08:55:06 |
OpenStack Infra |
charm-nova-cloud-controller: status |
New |
Fix Committed |
|
| 2018-04-12 09:16:56 |
OpenStack Infra |
charm-cinder: status |
Triaged |
Fix Committed |
|
| 2018-04-19 06:55:03 |
OpenStack Infra |
charm-openstack-dashboard: status |
New |
Fix Committed |
|
| 2018-04-19 15:46:34 |
OpenStack Infra |
charm-heat: status |
New |
Fix Committed |
|
| 2018-04-20 05:35:53 |
Frode Nordahl |
charm-neutron-api: milestone |
18.05 |
18.02 |
|
| 2018-04-20 05:35:59 |
Frode Nordahl |
charm-neutron-api: status |
Fix Committed |
Fix Released |
|
| 2018-05-07 20:12:11 |
Drew Freiberger |
tags |
4010 cpe-onsite |
4010 canonical-bootstack cpe-onsite |
|
| 2018-05-07 20:24:43 |
Drew Freiberger |
bug |
|
|
added subscriber Canonical Field Critical |
| 2018-05-07 20:36:02 |
Felipe Reyes |
tags |
4010 canonical-bootstack cpe-onsite |
4010 canonical-bootstack cpe-onsite sts |
|
| 2018-05-08 14:19:24 |
Felipe Reyes |
tags |
4010 canonical-bootstack cpe-onsite sts |
4010 backport-potential canonical-bootstack cpe-onsite stable-backport sts |
|
| 2018-05-10 17:05:40 |
Ryan Beisner |
charm-heat: status |
Fix Committed |
Fix Released |
|
| 2018-05-16 17:53:54 |
Felipe Reyes |
charm-keystone: status |
Fix Committed |
Fix Released |
|
| 2018-06-11 22:12:30 |
David Ames |
charm-cinder: status |
Fix Committed |
Fix Released |
|
| 2018-06-11 22:12:32 |
David Ames |
charm-glance: status |
Fix Committed |
Fix Released |
|
| 2018-06-11 22:12:34 |
David Ames |
charm-nova-cloud-controller: status |
Fix Committed |
Fix Released |
|
| 2018-06-11 22:12:36 |
David Ames |
charm-openstack-dashboard: status |
Fix Committed |
Fix Released |
|
| 2018-06-14 17:01:20 |
David Ames |
charm-helpers: status |
Fix Committed |
Fix Released |
|
