OpenStack Nova Cloud Controller Charm

nova-compute service stops working in TLS-enabled deployment

Bug #1911902 reported by Frode Nordahl
28
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Nova Cloud Controller Charm
Fix Released
Critical
Frode Nordahl
OpenStack Nova Cloud Controller Charm 21.01
OpenStack Nova Compute Charm
Invalid
Undecided
Unassigned

Bug Description

In a TLS-enabled deployment where certificates are either provided by Vault's self signed root CA or through the ssl_* configuration options using self-signed certificates, the nova-compute service will stop working. If the deployment has TLS enabled with certificates signed by a CA authority shipped with the Ubuntu distribution it will work fine.

Historically the nova-compute service has primarily communicated with the world around it through the message queue and as such we have to date not provided it with means to communicate with TLS-enabled services.

However, it appears something has changed which makes this a requirement. If you deploy a focal-ussuri bundle with Vault and certificate relations to all services today, the nova-compute services will start and register before TLS is enabled and look OK for a little bit. If you leave it up the nova-compute services will eventually all stop and you will find log messages like this in it's log:

2021-01-15 11:14:12.457 60502 CRITICAL nova [req-97b3da1c-dc86-4754-92a7-b04da4cf8de4 - - - - -] Unhandled error: keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. SSL exception connecting to https://10.5.0.9:35357: HTTPSConnectionPool(host='10.5.0.9', port=35357): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
2021-01-15 11:14:12.457 60502 ERROR nova Traceback (most recent call last):
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 485, in wrap_socket
2021-01-15 11:14:12.457 60502 ERROR nova cnx.do_handshake()
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1915, in do_handshake
2021-01-15 11:14:12.457 60502 ERROR nova self._raise_ssl_error(self._ssl, result)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1647, in _raise_ssl_error
2021-01-15 11:14:12.457 60502 ERROR nova _raise_current_error()
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
2021-01-15 11:14:12.457 60502 ERROR nova raise exception_type(errors)
2021-01-15 11:14:12.457 60502 ERROR nova OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
2021-01-15 11:14:12.457 60502 ERROR nova
2021-01-15 11:14:12.457 60502 ERROR nova During handling of the above exception, another exception occurred:
2021-01-15 11:14:12.457 60502 ERROR nova
2021-01-15 11:14:12.457 60502 ERROR nova Traceback (most recent call last):
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
2021-01-15 11:14:12.457 60502 ERROR nova httplib_response = self._make_request(
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 376, in _make_request
2021-01-15 11:14:12.457 60502 ERROR nova self._validate_conn(conn)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 996, in _validate_conn
2021-01-15 11:14:12.457 60502 ERROR nova conn.connect()
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in connect
2021-01-15 11:14:12.457 60502 ERROR nova self.sock = ssl_wrap_socket(
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 383, in ssl_wrap_socket
2021-01-15 11:14:12.457 60502 ERROR nova return context.wrap_socket(sock)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 491, in wrap_socket
2021-01-15 11:14:12.457 60502 ERROR nova raise ssl.SSLError("bad handshake: %r" % e)
2021-01-15 11:14:12.457 60502 ERROR nova ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)
2021-01-15 11:14:12.457 60502 ERROR nova
2021-01-15 11:14:12.457 60502 ERROR nova During handling of the above exception, another exception occurred:
2021-01-15 11:14:12.457 60502 ERROR nova
2021-01-15 11:14:12.457 60502 ERROR nova Traceback (most recent call last):
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
2021-01-15 11:14:12.457 60502 ERROR nova resp = conn.urlopen(
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
2021-01-15 11:14:12.457 60502 ERROR nova retries = retries.increment(
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment
2021-01-15 11:14:12.457 60502 ERROR nova raise MaxRetryError(_pool, url, error or ResponseError(cause))
2021-01-15 11:14:12.457 60502 ERROR nova urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.5.0.9', port=35357): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
2021-01-15 11:14:12.457 60502 ERROR nova
2021-01-15 11:14:12.457 60502 ERROR nova During handling of the above exception, another exception occurred:
2021-01-15 11:14:12.457 60502 ERROR nova
2021-01-15 11:14:12.457 60502 ERROR nova Traceback (most recent call last):
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1004, in _send_request
2021-01-15 11:14:12.457 60502 ERROR nova resp = self.session.request(method, url, **kwargs)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
2021-01-15 11:14:12.457 60502 ERROR nova resp = self.send(prep, **send_kwargs)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
2021-01-15 11:14:12.457 60502 ERROR nova r = adapter.send(request, **kwargs)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/requests/adapters.py", line 514, in send
2021-01-15 11:14:12.457 60502 ERROR nova raise SSLError(e, request=request)
2021-01-15 11:14:12.457 60502 ERROR nova requests.exceptions.SSLError: HTTPSConnectionPool(host='10.5.0.9', port=35357): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
2021-01-15 11:14:12.457 60502 ERROR nova
2021-01-15 11:14:12.457 60502 ERROR nova During handling of the above exception, another exception occurred:
2021-01-15 11:14:12.457 60502 ERROR nova
2021-01-15 11:14:12.457 60502 ERROR nova Traceback (most recent call last):
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 136, in _do_create_plugin
2021-01-15 11:14:12.457 60502 ERROR nova disc = self.get_discovery(session,
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 608, in get_discovery
2021-01-15 11:14:12.457 60502 ERROR nova return discover.get_discovery(session=session, url=url,
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 1452, in get_discovery
2021-01-15 11:14:12.457 60502 ERROR nova disc = Discover(session, url, authenticated=authenticated)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 535, in __init__
2021-01-15 11:14:12.457 60502 ERROR nova self._data = get_version_data(session, url,
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/discover.py", line 102, in get_version_data
2021-01-15 11:14:12.457 60502 ERROR nova resp = session.get(url, headers=headers, authenticated=authenticated)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1123, in get
2021-01-15 11:14:12.457 60502 ERROR nova return self.request(url, 'GET', **kwargs)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 913, in request
2021-01-15 11:14:12.457 60502 ERROR nova resp = send(**kwargs)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1008, in _send_request
2021-01-15 11:14:12.457 60502 ERROR nova raise exceptions.SSLError(msg)
2021-01-15 11:14:12.457 60502 ERROR nova keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://10.5.0.9:35357: HTTPSConnectionPool(host='10.5.0.9', port=35357): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
2021-01-15 11:14:12.457 60502 ERROR nova
2021-01-15 11:14:12.457 60502 ERROR nova During handling of the above exception, another exception occurred:
2021-01-15 11:14:12.457 60502 ERROR nova
2021-01-15 11:14:12.457 60502 ERROR nova Traceback (most recent call last):
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/bin/nova-compute", line 10, in <module>
2021-01-15 11:14:12.457 60502 ERROR nova sys.exit(main())
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/nova/cmd/compute.py", line 56, in main
2021-01-15 11:14:12.457 60502 ERROR nova server = service.Service.create(binary='nova-compute',
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/nova/service.py", line 265, in create
2021-01-15 11:14:12.457 60502 ERROR nova service_obj = cls(host, binary, topic, manager,
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/nova/service.py", line 127, in __init__
2021-01-15 11:14:12.457 60502 ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 569, in __init__
2021-01-15 11:14:12.457 60502 ERROR nova self.reportclient = report.SchedulerReportClient()
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 186, in __init__
2021-01-15 11:14:12.457 60502 ERROR nova self._client = self._create_client()
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/nova/scheduler/client/report.py", line 229, in _create_client
2021-01-15 11:14:12.457 60502 ERROR nova client = self._adapter or utils.get_sdk_adapter('placement')
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/nova/utils.py", line 1079, in get_sdk_adapter
2021-01-15 11:14:12.457 60502 ERROR nova return getattr(conn, service_type)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/openstack/service_description.py", line 93, in __get__
2021-01-15 11:14:12.457 60502 ERROR nova endpoint = proxy_mod.Proxy.get_endpoint(proxy)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 282, in get_endpoint
2021-01-15 11:14:12.457 60502 ERROR nova return self.session.get_endpoint(auth or self.auth, **kwargs)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1225, in get_endpoint
2021-01-15 11:14:12.457 60502 ERROR nova return auth.get_endpoint(self, **kwargs)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 375, in get_endpoint
2021-01-15 11:14:12.457 60502 ERROR nova endpoint_data = self.get_endpoint_data(
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2021-01-15 11:14:12.457 60502 ERROR nova service_catalog = self.get_access(session).service_catalog
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2021-01-15 11:14:12.457 60502 ERROR nova self.auth_ref = self.get_auth_ref(session)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
2021-01-15 11:14:12.457 60502 ERROR nova self._plugin = self._do_create_plugin(session)
2021-01-15 11:14:12.457 60502 ERROR nova File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 158, in _do_create_plugin
2021-01-15 11:14:12.457 60502 ERROR nova raise exceptions.DiscoveryFailure(
2021-01-15 11:14:12.457 60502 ERROR nova keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. SSL exception connecting to https://10.5.0.9:35357: HTTPSConnectionPool(host='10.5.0.9', port=35357): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
2021-01-15 11:14:12.457 60502 ERROR nova

And surely enough, the nova-compute charm does not have any ssl_ca configuration option nor certificates relation that would allow it to install a CA certificate Nova could use to authenticate the API endpoints it speaks to.

Revision history for this message
Frode Nordahl (fnordahl) wrote :

Apparently the nova-cloud-controller charm seeks to share it's CA certificate with the nova-compute charms over its cloud-compute relation.

However, the code looks for /usr/local/share/ca-certificates/keystone_juju_ca_cert.crt but vault_juju_ca_cert.crt is the file on disk.

CA_CERT_PATH = '/usr/local/share/ca-certificates/keystone_juju_ca_cert.crt'

def keystone_ca_cert_b64():
    '''Returns the local Keystone-provided CA cert if it exists, or None.'''
    if not os.path.isfile(CA_CERT_PATH):
        return None
    with open(CA_CERT_PATH, 'rb') as _in:
        return base64.b64encode(_in.read()).decode('utf-8')

Changed in charm-nova-compute:
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → Frode Nordahl (fnordahl)
milestone: none → 21.01
Frode Nordahl (fnordahl)
Changed in charm-nova-cloud-controller:
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → Frode Nordahl (fnordahl)
milestone: none → 21.01
Changed in charm-nova-compute:
status: In Progress → Invalid
Frode Nordahl (fnordahl)
Changed in charm-nova-compute:
importance: Critical → Undecided
assignee: Frode Nordahl (fnordahl) → nobody
milestone: 21.01 → none
Revision history for this message
Frode Nordahl (fnordahl) wrote :

https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/770938

Aurelien Lourot (aurelien-lourot)
Changed in charm-nova-cloud-controller:
status: In Progress → Fix Committed
David Ames (thedac)
Changed in charm-nova-cloud-controller:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.