keystone auth url not updated when switching from http to https

Bug #1882822 reported by Narinder Gupta on 2020-06-09
This bug affects 2 people
Affects Status Importance Assigned to Milestone

Bug Description

openstack-service-checks does not handle the case where keystone updates it's configuration. If keystone changes from using http to https (e.g. it adds vault certificates after deployment), openstack-service-checks will continue using the http endpoint in it's nagios.novarc file, thus using the wrong endpoint.

Steps to reproduce:
Deploy openstack-service-checks and keystone.
Ensure keystone is configured as http.
Add relation keystone:identity-notifications openstack-service-checks:notifications and keystone:identity-credentials openstack-service-checks:identity-credentials.
Reconfigure keystone to use ssl.
Cause an Openstack Service endpoint change. That will trigger identity-notifications relation change and cause a request using the stored auth url.

Todo: update nagios.novarc file upon keystone auth url change.

Related branches

Ashley Lai (alai) wrote :

It looks like there are two issues here:

1. save_creds() does not save cacert.

2. The auth_url is pointing to http instead of https.
020-06-10 14:34:03 DEBUG identity-notifications-relation-changed {'username': 'nagios', 'password': 'WyTyJSFs8m6qcT4B3wC7hx8ZMTk3C776356ZmjW6sHPBBZzzTsRwzksMMyNKLkB6', 'region': 'RegionOne', 'project_name': 'services', 'auth_version': '3', 'user_domain_name': 'service_domain', 'project_domain_name': 'service_domain', 'auth_url': 'http://keystoneinternal.production.solutionsqa:35357/v3'}

Ashley Lai (alai) wrote :

Adding intermediate CA support as mentioned in bug 1845039 would be useful here.

Ashley Lai (alai) wrote :

This bug is seen after the vault is initialized.

openstack-service-checks/0* error idle 21/lxd/8 hook failed: "identity-notifications-relation-changed"

Ashley Lai (alai) wrote :
Download full text (3.4 KiB)

2020-06-10 04:47:05 INFO juju-log identity-notifications:313: Invoking reactive handler: reactive/
2020-06-10 04:47:06 ERROR juju-log identity-notifications:313: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-openstack-service-checks-0/.venv/lib/python3.6/site-packages/charms/reactive/", line 74, in main
  File "/var/lib/juju/agents/unit-openstack-service-checks-0/.venv/lib/python3.6/site-packages/charms/reactive/", line 390, in dispatch
  File "/var/lib/juju/agents/unit-openstack-service-checks-0/.venv/lib/python3.6/site-packages/charms/reactive/", line 359, in _invoke
  File "/var/lib/juju/agents/unit-openstack-service-checks-0/.venv/lib/python3.6/site-packages/charms/reactive/", line 181, in invoke
  File "/var/lib/juju/agents/unit-openstack-service-checks-0/charm/reactive/", line 207, in configure_nrpe_endpoints
  File "lib/", line 272, in create_endpoint_checks
    endpoints = self.keystone_endpoints
  File "lib/", line 363, in keystone_endpoints
    return self._keystone_client.endpoints.list()
  File "/usr/lib/python3/dist-packages/keystoneclient/v3/", line 125, in list
  File "/usr/lib/python3/dist-packages/keystoneclient/", line 75, in func
    return f(*args, **new_kwargs)
  File "/usr/lib/python3/dist-packages/keystoneclient/", line 397, in list
  File "/usr/lib/python3/dist-packages/keystoneclient/", line 125, in _list
    resp, body = self.client.get(url, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/", line 304, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/", line 463, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/", line 189, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneclient/", line 343, in request
    auth_headers = self.get_auth_headers(auth)
  File "/usr/lib/python3/dist-packages/keystoneclient/", line 660, in get_auth_headers
    return auth.get_headers(self, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneclient/auth/", line 159, in get_headers
    token = self.get_token(session)
  File "/usr/lib/python3/dist-packages/keystoneclient/auth/identity/", line 206, in get_token
    return self.get_access(session).auth_token
  File "/usr/lib/python3/dist-packages/keystoneclient/auth/identity/", line 252, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/usr/lib/python3/dist-packages/keystoneclient/auth/identity/v3/", line 189, in get_auth_ref
    authenticated=False, log=False, **rkwargs)
  File "/usr/lib/python3/dist-packages/keystoneclient/", line 543,...


tags: added: cdo-qa foundations-engine
tags: added: field-high
Ashley Lai (alai) wrote :

Subscribed to field high as it is blocking field deployment.

Changed in charm-openstack-service-checks:
importance: Undecided → High
assignee: nobody → Zachary Zehring (zzehring)
status: New → In Progress
Zachary Zehring (zzehring) wrote :


To address the original description of this bug, the openstack-service-checks charm handles TLS by placing it in /usr/local/share/ca-certificates/ and then running /usr/sbin/update-ca-certificates instead of placing it in the novarc [0].

To address Ashley's issues:

1. Since the TLS cert is handled elsewhere, it is not involved in save_creds() as that function is just collecting and saving the credentials/data keystone passes in through relation and stores it in kv().

2. openstack-service-checks is only storing what the keystone charm provides. The keystone charm dictates what protocol should be used for auth_url [1] [2]. Can you check the juju config for keystone and make sure that keystone's https() function returns True based on the criteria [1].

If the problem persists after keystone config is evaluated/updated, please upload keystone charm logs, openstack-service-checks charm logs, and the juju config for both.


Changed in charm-openstack-service-checks:
status: In Progress → Incomplete
Ashley Lai (alai) wrote :

The work around for the issue is to deploy a new openstack-service-checks unit after the vault is initialized.

For new deployment, it needs to point to cs:openstack-service-checks and not the bootstack namespace.

summary: - adding ca-cert does not render new nagios-novarc template
+ keystone auth url not updated when switching from http to https
description: updated
Zachary Zehring (zzehring) wrote :

Does this bug still constitute a field high or can we downgrade/remove?

Changed in charm-openstack-service-checks:
status: Incomplete → Triaged
assignee: Zachary Zehring (zzehring) → nobody
Ashley Lai (alai) wrote :

It is still a field high as it is blocking our SSL deployment.

Ashley Lai (alai) wrote :

We hit the same issue using this charm also cs:openstack-service-checks.

Changed in charm-openstack-service-checks:
status: Triaged → In Progress
Adam Dyess (addyess) on 2020-07-07
Changed in charm-openstack-service-checks:
assignee: nobody → Adam Dyess (addyess)
Ashley Lai (alai) wrote :

We have two runs did not hit this issue. Both are pointing to Zach's fix.

Adam Dyess (addyess) on 2020-07-09
Changed in charm-openstack-service-checks:
status: In Progress → Fix Committed
Adam Dyess (addyess) wrote :
Changed in charm-openstack-service-checks:
status: Fix Committed → Fix Released
Adam Dyess (addyess) on 2020-07-16
Changed in charm-openstack-service-checks:
milestone: none → 20.08
Adam Dyess (addyess) on 2020-07-16
Changed in charm-openstack-service-checks:
assignee: Adam Dyess (addyess) → nobody
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers