_determine_version doesn't consider provided CA cert

[Brian Holmes]

If openstack-integrator's _determine_version function is unable to parse the Keystone endpoint API version from the provided URL (for example, due to the separate issue I reported in LP #1964544), it resorts to the fallback strategy of connecting to the endpoint and reading its version from the JSON response. This connection attempt (at lib/charms/layer/openstack.py:328) is made via urllib.request.urlopen without considering that a CA certificate bundle may have been passed to the charm via its 'endpoint-tls-ca' configuration option. In scenarios where that CA bundle is required to verify the Keystone endpoint's certificate, an ssl.SSLCertVerificationError exception is thrown and the "install" hook aborts:

unit-openstack-integrator-0: 00:45:59 INFO unit.openstack-integrator/0.juju-log Reactive main running for hook install
unit-openstack-integrator-0: 00:45:59 INFO unit.openstack-integrator/0.juju-log Initializing Snap Layer
unit-openstack-integrator-0: 00:46:00 INFO unit.openstack-integrator/0.juju-log Installing openstackclients from store
unit-openstack-integrator-0: 00:46:00 INFO unit.openstack-integrator/0.juju-log Invoking reactive handler: reactive/openstack.py:18:set_app_ver
unit-openstack-integrator-0: 00:46:00 INFO unit.openstack-integrator/0.juju-log Get installed key for snap openstackclients
unit-openstack-integrator-0: 00:46:00 INFO unit.openstack-integrator/0.juju-log Invoking reactive handler: reactive/openstack.py:24:update_creds
unit-openstack-integrator-0: 00:46:00 INFO unit.openstack-integrator/0.juju-log Invoking reactive handler: reactive/openstack.py:55:get_creds
unit-openstack-integrator-0: 00:46:00 INFO unit.openstack-integrator/0.juju-log Checking credentials-get for credentials
unit-openstack-integrator-0: 00:46:00 ERROR unit.openstack-integrator/0.juju-log Hook error:
Traceback (most recent call last):
  File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.8/http/client.py", line 1256, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 951, in send
    self.connect()
  File "/usr/lib/python3.8/http/client.py", line 1425, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-openstack-integrator-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-openstack-integrator-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-openstack-integrator-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-openstack-integrator-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-openstack-integrator-0/charm/reactive/openstack.py", line 58, in get_creds
    credentials_exist = layer.openstack.update_credentials()
  File "/var/lib/juju/agents/unit-openstack-integrator-0/charm/lib/charms/layer/openstack.py", line 74, in update_credentials
    _merge_if_set(creds_data, _normalize_creds(_creds_data))
  File "/var/lib/juju/agents/unit-openstack-integrator-0/charm/lib/charms/layer/openstack.py", line 248, in _normalize_creds
    version=_determine_version(attrs, endpoint),
  File "/var/lib/juju/agents/unit-openstack-integrator-0/charm/lib/charms/layer/openstack.py", line 328, in _determine_version
    with urlopen(endpoint) as fp:
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 1397, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/lib/python3.8/urllib/request.py", line 1357, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>

[Adam Dyess (addyess)]

Fortunately the openstack-integration has access to the ca-cert for that endpoint when _determine_version is called.

We can simply pass that ca_cert through to the call of _determine_version like this:

def _determine_version(attrs, endpoint, endpoint_tls_ca):
   ...

and when urlopen is called, ensure that it's called setting the argument cafile to point to a temporary file holding the cert value.

I'm working on a solution for this but i've got to get access to an openstack cluster that has an https keystone endpoint

[Adam Dyess (addyess)]

Creating PR to address this issue:
https://github.com/juju-solutions/charm-openstack-integrator/pull/69