Swift backend can not use custom CA bundle to verify server SSL certs when those are not added to global system certs

Bug #1820817 reported by Pavlo Shchelokovskyy on 2019-03-19
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
glance_store
Undecided
Pavlo Shchelokovskyy

Bug Description

Some installations have security policies that require custom SSL CA bundles not added to global system CA certificates - and Glance + Swift backend can not work in that mode, producing the following trace:

Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.api.v2.image_data [None req-0668a2ba-9860-4510-a016-d9de931aa1c7 demo demo] Failed to upload image data due to internal error: Bac[11/5436]
tion: Cannot find swift service endpoint : Unable to establish connection to https://192.168.100.12/identity/v3/auth/tokens: HTTPSConnectionPool(host='192.168.100.12', port=443): Max retries exceeded with url: /id
entity/v3/auth/tokens (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi [None req-0668a2ba-9860-4510-a016-d9de931aa1c7 demo demo] Caught error: Cannot find swift service endpoint : Unable to establis
h connection to https://192.168.100.12/identity/v3/auth/tokens: HTTPSConnectionPool(host='192.168.100.12', port=443): Max retries exceeded with url: /identity/v3/auth/tokens (Caused by SSLError(SSLError("bad hands
hake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),)): BackendException: Cannot find swift service endpoint : Unable to establish connection to https://192.168.100.1
2/identity/v3/auth/tokens: HTTPSConnectionPool(host='192.168.100.12', port=443): Max retries exceeded with url: /identity/v3/auth/tokens (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_pr
ocess_server_certificate', 'certificate verify failed')],)",),))
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi Traceback (most recent call last):
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/wsgi.py", line 1176, in __call__
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi request, **action_args)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/wsgi.py", line 1215, in dispatch
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi return method(*args, **kwargs)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/utils.py", line 363, in wrapped
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/api/v2/image_data.py", line 269, in upload
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi self._restore(image_repo, image)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi self.force_reraise()
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/api/v2/image_data.py", line 134, in upload
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi image.set_data(data, size)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/domain/proxy.py", line 195, in set_data
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi self.base.set_data(data, size)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/notifier.py", line 480, in set_data
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi self.force_reraise()
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/notifier.py", line 427, in set_data
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi self.repo.set_data(data, size)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/api/policy.py", line 194, in set_data
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/quota/__init__.py", line 304, in set_data
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi self.image.set_data(data, size=size)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/location.py", line 439, in set_data
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi verifier=verifier)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 453, in add_to_backend
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi verifier)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 426, in store_add_to_backend
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi verifier=verifier)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/usr/local/lib/python2.7/dist-packages/glance_store/capabilities.py", line 225, in op_checker
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/usr/local/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py", line 852, in add
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi allow_reauth=need_chunks) as manager:
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/usr/local/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py", line 1304, in get_manager
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi allow_reauth)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/usr/local/lib/python2.7/dist-packages/glance_store/_drivers/swift/connection_manager.py", line 62, in __init__
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi self.storage_url = self._get_storage_url()
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi File "/usr/local/lib/python2.7/dist-packages/glance_store/_drivers/swift/connection_manager.py", line 158, in _get_storage_ur
l
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi raise exceptions.BackendException(msg)
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi BackendException: Cannot find swift service endpoint : Unable to establish connection to https://192.168.100.12/identity/v3/aut
h/tokens: HTTPSConnectionPool(host='192.168.100.12', port=443): Max retries exceeded with url: /identity/v3/auth/tokens (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_cert
ificate', 'certificate verify failed')],)",),))
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: ERROR glance.common.wsgi
Mar 18 16:51:58 dsvm2-kvm-work <email address hidden>[5114]: [pid: 5118|app: 0|req: 12/25] 127.0.0.1 () {40 vars in 860 bytes} [Mon Mar 18 16:51:58 2019] PUT /v2/images/9ff9a37d-49a7-442f-b778-0bfba7469b48/file =>
 generated 228 bytes in 150 msecs (HTTP/1.1 500) 4 headers in 184 bytes (1 switches on core 0)

The reason seems to be the keystone client session being used to fetch the token to pass to swiftclient is not using the swift_store_cacert option (which is being passed to the swiftclient though).
Using it to verify connection to Keystone would in fact be in line with how any other openstack clint is configured and what is supported by keystoneauth's session - which is the same CA bundle is used to verify both Keystone connection to fetch the token and the connection to actual service, so that CA bundle must verify both.

Changed in glance-store:
status: New → In Progress
assignee: nobody → Pavlo Shchelokovskyy (pshchelo)
Pavlo Shchelokovskyy (pshchelo) wrote :

Somehow Gerrit/Launchpad integration did not work, so here is the patch to fix this:

https://review.openstack.org/#/c/644390

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers