keystone auth url not updated when switching from http to https

[Narinder Gupta]

openstack-service-checks does not handle the case where keystone updates it's configuration. If keystone changes from using http to https (e.g. it adds vault certificates after deployment), openstack-service-checks will continue using the http endpoint in it's nagios.novarc file, thus using the wrong endpoint.

Steps to reproduce:
Deploy openstack-service-checks and keystone.
Ensure keystone is configured as http.
Add relation keystone:identity-notifications openstack-service-checks:notifications and keystone:identity-credentials openstack-service-checks:identity-credentials.
Reconfigure keystone to use ssl.
Cause an Openstack Service endpoint change. That will trigger identity-notifications relation change and cause a request using the stored auth url.

Todo: update nagios.novarc file upon keystone auth url change.

[Ashley Lai (alai)]

It looks like there are two issues here:

1. save_creds() does not save cacert.
http://marker.to/oTj3Ay

2. The auth_url is pointing to http instead of https.
020-06-10 14:34:03 DEBUG identity-notifications-relation-changed {'username': 'nagios', 'password': 'WyTyJSFs8m6qcT4B3wC7hx8ZMTk3C776356ZmjW6sHPBBZzzTsRwzksMMyNKLkB6', 'region': 'RegionOne', 'project_name': 'services', 'auth_version': '3', 'user_domain_name': 'service_domain', 'project_domain_name': 'service_domain', 'auth_url': 'http://keystoneinternal.production.solutionsqa:35357/v3'}

[Ashley Lai (alai)]

Adding intermediate CA support as mentioned in bug 1845039 would be useful here.

https://bugs.launchpad.net/charm-openstack-service-checks/+bug/1845039

[Ashley Lai (alai)]

This bug is seen after the vault is initialized.

openstack-service-checks/0* error idle 21/lxd/8 10.244.40.237 hook failed: "identity-notifications-relation-changed"

[Ashley Lai (alai)]

2020-06-10 04:47:05 INFO juju-log identity-notifications:313: Invoking reactive handler: reactive/openstack_service_checks.py:190:configure_nrpe_endpoints
2020-06-10 04:47:06 ERROR juju-log identity-notifications:313: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-openstack-service-checks-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-openstack-service-checks-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-openstack-service-checks-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-openstack-service-checks-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-openstack-service-checks-0/charm/reactive/openstack_service_checks.py", line 207, in configure_nrpe_endpoints
    helper.create_endpoint_checks(creds)
  File "lib/lib_openstack_service_checks.py", line 272, in create_endpoint_checks
    endpoints = self.keystone_endpoints
  File "lib/lib_openstack_service_checks.py", line 363, in keystone_endpoints
    return self._keystone_client.endpoints.list()
  File "/usr/lib/python3/dist-packages/keystoneclient/v3/endpoints.py", line 125, in list
    **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneclient/base.py", line 75, in func
    return f(*args, **new_kwargs)
  File "/usr/lib/python3/dist-packages/keystoneclient/base.py", line 397, in list
    self.collection_key)
  File "/usr/lib/python3/dist-packages/keystoneclient/base.py", line 125, in _list
    resp, body = self.client.get(url, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 304, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 463, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 189, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneclient/session.py", line 343, in request
    auth_headers = self.get_auth_headers(auth)
  File "/usr/lib/python3/dist-packages/keystoneclient/session.py", line 660, in get_auth_headers
    return auth.get_headers(self, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneclient/auth/base.py", line 159, in get_headers
    token = self.get_token(session)
  File "/usr/lib/python3/dist-packages/keystoneclient/auth/identity/base.py", line 206, in get_token
    return self.get_access(session).auth_token
  File "/usr/lib/python3/dist-packages/keystoneclient/auth/identity/base.py", line 252, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/usr/lib/python3/dist-packages/keystoneclient/auth/identity/v3/base.py", line 189, in get_auth_ref
    authenticated=False, log=False, **rkwargs)
  File "/usr/lib/python3/dist-packages/keystoneclient/session.py", line 543,...

[Ashley Lai (alai)]

Subscribed to field high as it is blocking field deployment.

[Zachary Zehring (zzehring)]

Hello,

To address the original description of this bug, the openstack-service-checks charm handles TLS by placing it in /usr/local/share/ca-certificates/ and then running /usr/sbin/update-ca-certificates instead of placing it in the novarc [0].

To address Ashley's issues:

1. Since the TLS cert is handled elsewhere, it is not involved in save_creds() as that function is just collecting and saving the credentials/data keystone passes in through relation and stores it in kv().

2. openstack-service-checks is only storing what the keystone charm provides. The keystone charm dictates what protocol should be used for auth_url [1] [2]. Can you check the juju config for keystone and make sure that keystone's https() function returns True based on the criteria [1].

If the problem persists after keystone config is evaluated/updated, please upload keystone charm logs, openstack-service-checks charm logs, and the juju config for both.

Thanks,
Zack

[Zachary Zehring (zzehring)]

Adding sources for above comment:

[0] https://git.launchpad.net/charm-openstack-service-checks/tree/reactive/openstack_service_checks.py#n157
[1] https://github.com/openstack/charm-keystone/blob/4eb640ab56a48f49f354fd4c4ae929c763e72f61/charmhelpers/contrib/hahelpers/cluster.py#L215
[2] https://github.com/openstack/charm-keystone/blob/4eb640ab56a48f49f354fd4c4ae929c763e72f61/hooks/keystone_utils.py#L1985

[Ashley Lai (alai)]

The work around for the issue is to deploy a new openstack-service-checks unit after the vault is initialized.

For new deployment, it needs to point to cs:openstack-service-checks and not the bootstack namespace.

[Zachary Zehring (zzehring)]

Does this bug still constitute a field high or can we downgrade/remove?

[Ashley Lai (alai)]

It is still a field high as it is blocking our SSL deployment.

[Ashley Lai (alai)]

We hit the same issue using this charm also cs:openstack-service-checks.

[Ashley Lai (alai)]

We have two runs did not hit this issue. Both are pointing to Zach's fix.
cs:~zzehring/charm-openstack-service-checks-0

[Adam Dyess (addyess)]

Released here: https://jaas.ai/u/llama-charmers-next/openstack-service-checks/9