keystoneauth doesn't use a default cafile

[Sean McCully]

KeystoneAuth doesn't use a default cafile, this causes problems when generating a local CA or self signed CA with HTTPS enabled endpoints. Even though the CA can be installed locally, keystone auth will still fail ssl verification.

=================
2017-04-03 00:54:49.305 545 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: bb9ce702f5864adf8e4720d2304fcb2a __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:346
2017-04-03 00:54:49.337 545 DEBUG cinderclient.v2.client [req-7cb00c0e-be3d-4e25-b369-fd8aecbae803 7106629bf3b440a79030d327abd0747e 2aeed525cd4e4f329b0567be30d3aa6c - default default] REQ: curl -g -i -X GET https://openstack.local.net:8776/v2/2aeed525cd4e4f329b0567be30d3aa6c/volumes/ef828539-027c-4daa-9c96-19d2f3cd51e3 -H "X-Service-Token: {SHA1}77aedd00ae7642ecf44c452749b8b3ed6f45330d" -H "User-Agent: python-cinderclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}a91d7c21ef9f2401ffbe691355000e7bcc9d390c" _http_log_request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:347
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions [req-7cb00c0e-be3d-4e25-b369-fd8aecbae803 7106629bf3b440a79030d327abd0747e 2aeed525cd4e4f329b0567be30d3aa6c - default default] Unexpected exception in API method
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 108, in wrapper
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/volumes.py", line 338, in create
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions volume_id, device)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 204, in inner
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions return function(self, context, instance, *args, **kwargs)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 152, in inner
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions return f(self, context, instance, *args, **kw)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3772, in attach_volume
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions disk_bus, device_type)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3715, in _attach_volume
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions volume_bdm.destroy()
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions self.force_reraise()
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3711, in _attach_volume
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions self._check_attach_and_reserve_volume(context, volume_id, instance)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3693, in _check_attach_and_reserve_volume
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions volume = self.volume_api.get(context, volume_id)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 177, in wrapper
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions _reraise(exception.CinderConnectionFailed(reason=err_msg))
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 231, in _reraise
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions six.reraise(type(desired_exc), desired_exc, sys.exc_info()[2])
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 173, in wrapper
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions res = method(self, ctx, *args, **kwargs)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 195, in wrapper
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions res = method(self, ctx, volume_id, *args, **kwargs)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 239, in get
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions item = cinderclient(context).volumes.get(volume_id)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 277, in get
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions return self._get("/volumes/%s" % volume_id, "volume")
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/cinderclient/base.py", line 314, in _get
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions resp, body = self.api.client.get(url)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 171, in get
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions return self._cs_request(url, 'GET', **kwargs)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 162, in _cs_request
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions return self.request(url, method, **kwargs)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 148, in request
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions **kwargs)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 380, in request
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 148, in request
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions return self.session.request(url, method, **kwargs)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions return wrapped(*args, **kwargs)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 616, in request
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions resp = send(**kwargs)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 678, in _send_request
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions raise exceptions.SSLError(msg)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions CinderConnectionFailed: Connection to cinder host failed: SSL exception connecting to https://openstack.local.nunet.net:8776/v2/2aeed525cd4e4f329b0567be30d3aa6c/volumes/ef828539-027c-4daa-9c96-19d2f3cd51e3: ("bad handshake: Error([('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')],)",)
2017-04-03 00:54:49.442 545 ERROR nova.api.openstack.extensions

====================

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/452585

[Sean McCully (sean-mccully)]

Is it worthwhile to consider finding openssl's default paths if the current proposed method not available?

Or just not provide this if it isn't available?

https://docs.python.org/2/library/ssl.html#ssl.SSLContext.set_default_verify_paths

[Juan Antonio Osorio Robles (juan-osorio-robles)]

Sean, well, you could get them with:

import ssl
ssl.get_default_verify_paths().openssl_cafile

as opposed to using the set_.* function.

Currently, the default CA bundle is the same as the one for python-requests, since it's what keystoneauth's session uses. And it's using some default already: http://docs.python-requests.org/en/master/user/advanced/#ca-certificates

So you might want to discuss this in the mailing list, since this might change the behavior of a LOT of OpenStack services, since most use keystoneauth's sessions.

[Sean McCully (sean-mccully)]

Juan et al,

Right so "python-requests" bundles Mozilla CA Bundle, but this creates a problem when working with a CA not in Mozilla's default Bundle (i.e. self created CA).

The current method of passing the cabundle via config options works well until it's not being used. Which is the result of this bug, proposed solution changes the CA Bundle to be used by default to the system CA bundle which is where any CA not in Mozilla's default CA would be stored. This alleviates the need for requiring os-cacert to be passed via every requests. Which is troublesome since this auth library is used so frequently. Which the problem as I see it, is that this alleviates the need to constantly fight these types of bugs every time they creep up.

Python's SSL library calls libssl to get the default path (at compile time) for the CA bundle. If this is not available or not accurate it will search through several well known locations for a CABundle cert or file system path. This covers the major distro and uses cases where this software is intended to run.

Finally it will fallback to the Mozilla CA Bundle used by requests when being run on Client Machines (OS X, Windows, etc.) or some unknown OS.

[Lance Bragstad (lbragstad)]

Automatically unassigning due to inactivity.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Monty Taylor (<email address hidden>) on branch: master
Review: https://review.openstack.org/452585
Reason: This is fairly old at this point. Feel free to restore if it's still relevant.