hook failed: "certificates-relation-changed" for vault:certificates

Bug #1810910 reported by Frode Nordahl
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Keystone Charm
Fix Released
High
Frode Nordahl

Bug Description

2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/keystone/keystone.conf
2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/keystone/logging.conf
2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/haproxy/haproxy.cfg
2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/apache2/sites-available/openstack_https_frontend.conf
2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/keystone/policy.json
2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/cron.d/keystone-token-flush
2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/cron.d/keystone-fernet-rotate-sync
2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/apache2/sites-enabled/wsgi-openstack-api.conf
2019-01-08 09:10:11 INFO juju-log certificates:5: Registered config file: /etc/memcached.conf
2019-01-08 09:10:12 INFO juju-log certificates:5: Making dir /etc/apache2/ssl/keystone root:root 555
2019-01-08 09:10:12 INFO juju-log certificates:5: Installing new CA cert at: /usr/local/share/ca-certificates/keystone_juju_ca_cert.crt
2019-01-08 09:10:12 DEBUG juju-log certificates:5: Writing file /usr/local/share/ca-certificates/keystone_juju_ca_cert.crt root:root 444
2019-01-08 09:10:12 DEBUG certificates-relation-changed Clearing symlinks in /etc/ssl/certs...
2019-01-08 09:10:13 DEBUG certificates-relation-changed done.
2019-01-08 09:10:13 DEBUG certificates-relation-changed Updating certificates in /etc/ssl/certs...
2019-01-08 09:10:14 DEBUG certificates-relation-changed 149 added, 0 removed; done.
2019-01-08 09:10:14 DEBUG certificates-relation-changed Running hooks in /etc/ca-certificates/update.d...
2019-01-08 09:10:14 DEBUG certificates-relation-changed done.
2019-01-08 09:10:14 DEBUG juju-log certificates:5: Writing file /etc/apache2/ssl/keystone/cert_juju-d9e982-0.lxd root:root 640
2019-01-08 09:10:14 DEBUG juju-log certificates:5: Writing file /etc/apache2/ssl/keystone/key_juju-d9e982-0.lxd root:root 640
2019-01-08 09:10:15 INFO juju-log certificates:5: Loaded template from templates/ocata/logging.conf
2019-01-08 09:10:15 INFO juju-log certificates:5: Rendering from template: /etc/keystone/logging.conf
2019-01-08 09:10:15 INFO juju-log certificates:5: Wrote template /etc/keystone/logging.conf.
2019-01-08 09:10:15 INFO juju-log certificates:5: Loaded template from templates/keystone-token-flush
2019-01-08 09:10:15 INFO juju-log certificates:5: Rendering from template: /etc/cron.d/keystone-token-flush
2019-01-08 09:10:15 INFO juju-log certificates:5: Wrote template /etc/cron.d/keystone-token-flush.
2019-01-08 09:10:15 DEBUG certificates-relation-changed lxc
2019-01-08 09:10:15 INFO juju-log certificates:5: Configuring Keystone to use a random admin token.
2019-01-08 09:10:15 INFO juju-log certificates:5: Loading a previously generated admin token from /var/lib/keystone/keystone.token
2019-01-08 09:10:15 INFO juju-log certificates:5: Loaded template from templates/wsgi-openstack-api.conf
2019-01-08 09:10:15 INFO juju-log certificates:5: Rendering from template: /etc/apache2/sites-enabled/wsgi-openstack-api.conf
2019-01-08 09:10:15 INFO juju-log certificates:5: Wrote template /etc/apache2/sites-enabled/wsgi-openstack-api.conf.
2019-01-08 09:10:15 DEBUG juju-log certificates:5: Ensuring haproxy enabled in /etc/default/haproxy.
2019-01-08 09:10:16 INFO juju-log certificates:5: HAProxy context is incomplete, this unit has no peers.
2019-01-08 09:10:16 INFO juju-log certificates:5: Loaded template from /var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/templates/haproxy.cfg
2019-01-08 09:10:16 INFO juju-log certificates:5: Rendering from template: /etc/haproxy/haproxy.cfg
2019-01-08 09:10:16 INFO juju-log certificates:5: Wrote template /etc/haproxy/haproxy.cfg.
2019-01-08 09:10:16 DEBUG certificates-relation-changed Considering dependency setenvif for ssl:
2019-01-08 09:10:16 DEBUG certificates-relation-changed Module setenvif already enabled
2019-01-08 09:10:16 DEBUG certificates-relation-changed Considering dependency mime for ssl:
2019-01-08 09:10:16 DEBUG certificates-relation-changed Module mime already enabled
2019-01-08 09:10:16 DEBUG certificates-relation-changed Considering dependency socache_shmcb for ssl:
2019-01-08 09:10:16 DEBUG certificates-relation-changed Module socache_shmcb already enabled
2019-01-08 09:10:16 DEBUG certificates-relation-changed Module ssl already enabled
2019-01-08 09:10:16 DEBUG certificates-relation-changed Module proxy already enabled
2019-01-08 09:10:16 DEBUG certificates-relation-changed Considering dependency proxy for proxy_http:
2019-01-08 09:10:16 DEBUG certificates-relation-changed Module proxy already enabled
2019-01-08 09:10:16 DEBUG certificates-relation-changed Module proxy_http already enabled
2019-01-08 09:10:16 DEBUG certificates-relation-changed Module headers already enabled
2019-01-08 09:10:16 INFO juju-log certificates:5: Loaded template from templates/openstack_https_frontend.conf
2019-01-08 09:10:16 INFO juju-log certificates:5: Rendering from template: /etc/apache2/sites-available/openstack_https_frontend.conf
2019-01-08 09:10:16 INFO juju-log certificates:5: Wrote template /etc/apache2/sites-available/openstack_https_frontend.conf.
2019-01-08 09:10:16 INFO juju-log certificates:5: Loaded template from /var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/templates/memcached.conf
2019-01-08 09:10:16 INFO juju-log certificates:5: Rendering from template: /etc/memcached.conf
2019-01-08 09:10:16 INFO juju-log certificates:5: Wrote template /etc/memcached.conf.
2019-01-08 09:10:16 INFO juju-log certificates:5: Configuring Keystone to use a random admin token.
2019-01-08 09:10:16 INFO juju-log certificates:5: Loading a previously generated admin token from /var/lib/keystone/keystone.token
2019-01-08 09:10:16 INFO juju-log certificates:5: Loaded template from templates/queens/policy.json
2019-01-08 09:10:16 INFO juju-log certificates:5: Rendering from template: /etc/keystone/policy.json
2019-01-08 09:10:16 INFO juju-log certificates:5: Wrote template /etc/keystone/policy.json.
2019-01-08 09:10:16 INFO juju-log certificates:5: Configuring Keystone to use a random admin token.
2019-01-08 09:10:17 INFO juju-log certificates:5: Loading a previously generated admin token from /var/lib/keystone/keystone.token
2019-01-08 09:10:17 INFO juju-log certificates:5: HAProxy context is incomplete, this unit has no peers.
2019-01-08 09:10:17 DEBUG certificates-relation-changed lxc
2019-01-08 09:10:17 INFO juju-log certificates:5: Loaded template from templates/ocata/keystone.conf
2019-01-08 09:10:17 INFO juju-log certificates:5: Rendering from template: /etc/keystone/keystone.conf
2019-01-08 09:10:17 INFO juju-log certificates:5: Wrote template /etc/keystone/keystone.conf.
2019-01-08 09:10:17 INFO juju-log certificates:5: Loaded template from templates/keystone-fernet-rotate-sync
2019-01-08 09:10:17 INFO juju-log certificates:5: Rendering from template: /etc/cron.d/keystone-fernet-rotate-sync
2019-01-08 09:10:17 INFO juju-log certificates:5: Wrote template /etc/cron.d/keystone-fernet-rotate-sync.
2019-01-08 09:10:17 DEBUG certificates-relation-changed lxc
2019-01-08 09:10:17 INFO juju-log certificates:5: Configuring Keystone to use a random admin token.
2019-01-08 09:10:17 INFO juju-log certificates:5: Loading a previously generated admin token from /var/lib/keystone/keystone.token
2019-01-08 09:10:17 INFO juju-log certificates:5: Configuring Keystone to use a random admin token.
2019-01-08 09:10:17 INFO juju-log certificates:5: Loading a previously generated admin token from /var/lib/keystone/keystone.token
2019-01-08 09:10:18 DEBUG certificates-relation-changed Site openstack_https_frontend already enabled
2019-01-08 09:10:19 DEBUG juju-log certificates:5: Database is initialised
2019-01-08 09:10:19 DEBUG certificates-relation-changed /usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py:179: UserWarning: Using keystoneclient sessions has been deprecated. Please update your software to use keystoneauth1.
2019-01-08 09:10:19 DEBUG certificates-relation-changed warnings.warn('Using keystoneclient sessions has been deprecated. '
2019-01-08 09:11:04 DEBUG certificates-relation-changed Traceback (most recent call last):
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 557, in <module>
2019-01-08 09:11:04 DEBUG certificates-relation-changed admin_token=spec['admin_token'])
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 487, in get_manager
2019-01-08 09:11:04 DEBUG certificates-relation-changed api_local_endpoint, admin_token, api_version)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 102, in _retry_on_exception_inner_2
2019-01-08 09:11:04 DEBUG certificates-relation-changed return f(*args, **kwargs)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 150, in get_keystone_manager
2019-01-08 09:11:04 DEBUG certificates-relation-changed for svc in manager.api.services.list():
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/services.py", line 93, in list
2019-01-08 09:11:04 DEBUG certificates-relation-changed **kwargs)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
2019-01-08 09:11:04 DEBUG certificates-relation-changed return f(*args, **new_kwargs)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 397, in list
2019-01-08 09:11:04 DEBUG certificates-relation-changed self.collection_key)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 125, in _list
2019-01-08 09:11:04 DEBUG certificates-relation-changed resp, body = self.client.get(url, **kwargs)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 304, in get
2019-01-08 09:11:04 DEBUG certificates-relation-changed return self.request(url, 'GET', **kwargs)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
2019-01-08 09:11:04 DEBUG certificates-relation-changed resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
2019-01-08 09:11:04 DEBUG certificates-relation-changed return self.session.request(url, method, **kwargs)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 428, in request
2019-01-08 09:11:04 DEBUG certificates-relation-changed resp = send(**kwargs)
2019-01-08 09:11:04 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 472, in _send_request
2019-01-08 09:11:04 DEBUG certificates-relation-changed raise exceptions.ConnectionRefused(msg)
2019-01-08 09:11:04 DEBUG certificates-relation-changed ConnectFailure: Unable to establish connection to http://localhost:35337/v3/services?
2019-01-08 09:11:04 ERROR juju-log certificates:5: The call within manager.py failed with the error: 'Unable to establish connection to http://localhost:35337/v3/services?'. The call was: path=['resolve_domain_id'], args=('default',), kwargs={}, api_version=None
2019-01-08 09:11:04 INFO juju-log certificates:5: Retrying '_ensure_initial_admin' 3 more times (delay=3)
2019-01-08 09:11:52 DEBUG certificates-relation-changed Traceback (most recent call last):
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 557, in <module>
2019-01-08 09:11:52 DEBUG certificates-relation-changed admin_token=spec['admin_token'])
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 487, in get_manager
2019-01-08 09:11:52 DEBUG certificates-relation-changed api_local_endpoint, admin_token, api_version)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 102, in _retry_on_exception_inner_2
2019-01-08 09:11:52 DEBUG certificates-relation-changed return f(*args, **kwargs)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 150, in get_keystone_manager
2019-01-08 09:11:52 DEBUG certificates-relation-changed for svc in manager.api.services.list():
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/services.py", line 93, in list
2019-01-08 09:11:52 DEBUG certificates-relation-changed **kwargs)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
2019-01-08 09:11:52 DEBUG certificates-relation-changed return f(*args, **new_kwargs)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 397, in list
2019-01-08 09:11:52 DEBUG certificates-relation-changed self.collection_key)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 125, in _list
2019-01-08 09:11:52 DEBUG certificates-relation-changed resp, body = self.client.get(url, **kwargs)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 304, in get
2019-01-08 09:11:52 DEBUG certificates-relation-changed return self.request(url, 'GET', **kwargs)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
2019-01-08 09:11:52 DEBUG certificates-relation-changed resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
2019-01-08 09:11:52 DEBUG certificates-relation-changed return self.session.request(url, method, **kwargs)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 428, in request
2019-01-08 09:11:52 DEBUG certificates-relation-changed resp = send(**kwargs)
2019-01-08 09:11:52 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 472, in _send_request
2019-01-08 09:11:52 DEBUG certificates-relation-changed raise exceptions.ConnectionRefused(msg)
2019-01-08 09:11:52 DEBUG certificates-relation-changed ConnectFailure: Unable to establish connection to http://localhost:35337/v3/services?
2019-01-08 09:11:53 ERROR juju-log certificates:5: The call within manager.py failed with the error: 'Unable to establish connection to http://localhost:35337/v3/services?'. The call was: path=['resolve_domain_id'], args=('default',), kwargs={}, api_version=None
2019-01-08 09:11:53 INFO juju-log certificates:5: Retrying '_ensure_initial_admin' 2 more times (delay=6)
2019-01-08 09:12:44 DEBUG certificates-relation-changed Traceback (most recent call last):
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 557, in <module>
2019-01-08 09:12:44 DEBUG certificates-relation-changed admin_token=spec['admin_token'])
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 487, in get_manager
2019-01-08 09:12:44 DEBUG certificates-relation-changed api_local_endpoint, admin_token, api_version)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 102, in _retry_on_exception_inner_2
2019-01-08 09:12:44 DEBUG certificates-relation-changed return f(*args, **kwargs)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 150, in get_keystone_manager
2019-01-08 09:12:44 DEBUG certificates-relation-changed for svc in manager.api.services.list():
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/services.py", line 93, in list
2019-01-08 09:12:44 DEBUG certificates-relation-changed **kwargs)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
2019-01-08 09:12:44 DEBUG certificates-relation-changed return f(*args, **new_kwargs)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 397, in list
2019-01-08 09:12:44 DEBUG certificates-relation-changed self.collection_key)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 125, in _list
2019-01-08 09:12:44 DEBUG certificates-relation-changed resp, body = self.client.get(url, **kwargs)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 304, in get
2019-01-08 09:12:44 DEBUG certificates-relation-changed return self.request(url, 'GET', **kwargs)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
2019-01-08 09:12:44 DEBUG certificates-relation-changed resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
2019-01-08 09:12:44 DEBUG certificates-relation-changed return self.session.request(url, method, **kwargs)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 428, in request
2019-01-08 09:12:44 DEBUG certificates-relation-changed resp = send(**kwargs)
2019-01-08 09:12:44 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 472, in _send_request
2019-01-08 09:12:44 DEBUG certificates-relation-changed raise exceptions.ConnectionRefused(msg)
2019-01-08 09:12:44 DEBUG certificates-relation-changed ConnectFailure: Unable to establish connection to http://localhost:35337/v3/services?
2019-01-08 09:12:44 ERROR juju-log certificates:5: The call within manager.py failed with the error: 'Unable to establish connection to http://localhost:35337/v3/services?'. The call was: path=['resolve_domain_id'], args=('default',), kwargs={}, api_version=None
2019-01-08 09:12:44 INFO juju-log certificates:5: Retrying '_ensure_initial_admin' 1 more times (delay=9)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Traceback (most recent call last):
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 557, in <module>
2019-01-08 09:13:38 DEBUG certificates-relation-changed admin_token=spec['admin_token'])
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 487, in get_manager
2019-01-08 09:13:38 DEBUG certificates-relation-changed api_local_endpoint, admin_token, api_version)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 102, in _retry_on_exception_inner_2
2019-01-08 09:13:38 DEBUG certificates-relation-changed return f(*args, **kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 150, in get_keystone_manager
2019-01-08 09:13:38 DEBUG certificates-relation-changed for svc in manager.api.services.list():
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/services.py", line 93, in list
2019-01-08 09:13:38 DEBUG certificates-relation-changed **kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
2019-01-08 09:13:38 DEBUG certificates-relation-changed return f(*args, **new_kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 397, in list
2019-01-08 09:13:38 DEBUG certificates-relation-changed self.collection_key)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 125, in _list
2019-01-08 09:13:38 DEBUG certificates-relation-changed resp, body = self.client.get(url, **kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 304, in get
2019-01-08 09:13:38 DEBUG certificates-relation-changed return self.request(url, 'GET', **kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
2019-01-08 09:13:38 DEBUG certificates-relation-changed resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
2019-01-08 09:13:38 DEBUG certificates-relation-changed return self.session.request(url, method, **kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 428, in request
2019-01-08 09:13:38 DEBUG certificates-relation-changed resp = send(**kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 472, in _send_request
2019-01-08 09:13:38 DEBUG certificates-relation-changed raise exceptions.ConnectionRefused(msg)
2019-01-08 09:13:38 DEBUG certificates-relation-changed ConnectFailure: Unable to establish connection to http://localhost:35337/v3/services?
2019-01-08 09:13:38 ERROR juju-log certificates:5: The call within manager.py failed with the error: 'Unable to establish connection to http://localhost:35337/v3/services?'. The call was: path=['resolve_domain_id'], args=('default',), kwargs={}, api_version=None
2019-01-08 09:13:38 DEBUG certificates-relation-changed Traceback (most recent call last):
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/certificates-relation-changed", line 882, in <module>
2019-01-08 09:13:38 DEBUG certificates-relation-changed main()
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/certificates-relation-changed", line 875, in main
2019-01-08 09:13:38 DEBUG certificates-relation-changed hooks.execute(sys.argv)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/hookenv.py", line 909, in execute
2019-01-08 09:13:38 DEBUG certificates-relation-changed self._hooks[hook_name]()
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1479, in wrapped_f
2019-01-08 09:13:38 DEBUG certificates-relation-changed stopstart, restart_functions)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/host.py", line 739, in restart_on_change_helper
2019-01-08 09:13:38 DEBUG certificates-relation-changed r = lambda_f()
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1478, in <lambda>
2019-01-08 09:13:38 DEBUG certificates-relation-changed (lambda: f(*args, **kwargs)), __restart_map_cache['cache'],
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/certificates-relation-changed", line 854, in certs_changed
2019-01-08 09:13:38 DEBUG certificates-relation-changed ensure_initial_admin(config)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1359, in ensure_initial_admin
2019-01-08 09:13:38 DEBUG certificates-relation-changed return _ensure_initial_admin(config)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2019-01-08 09:13:38 DEBUG certificates-relation-changed return f(*args, **kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1303, in _ensure_initial_admin
2019-01-08 09:13:38 DEBUG certificates-relation-changed default_domain_id = create_or_show_domain(DEFAULT_DOMAIN)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 953, in create_or_show_domain
2019-01-08 09:13:38 DEBUG certificates-relation-changed domain_id = manager.resolve_domain_id(name)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1019, in __call__
2019-01-08 09:13:38 DEBUG certificates-relation-changed return _proxy_manager_call(self._path, self.api_version, args, kwargs)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1052, in _proxy_manager_call
2019-01-08 09:13:38 DEBUG certificates-relation-changed raise e
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1049, in _proxy_manager_call
2019-01-08 09:13:38 DEBUG certificates-relation-changed raise RuntimeError(s)
2019-01-08 09:13:38 DEBUG certificates-relation-changed RuntimeError: The call within manager.py failed with the error: 'Unable to establish connection to http://localhost:35337/v3/services?'. The call was: path=['resolve_domain_id'], args=('default',), kwargs={}, api_version=None
2019-01-08 09:13:38 DEBUG certificates-relation-changed Traceback (most recent call last):
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 553, in <module>
2019-01-08 09:13:38 DEBUG certificates-relation-changed spec = json.loads(data)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
2019-01-08 09:13:38 DEBUG certificates-relation-changed return _default_decoder.decode(s)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
2019-01-08 09:13:38 DEBUG certificates-relation-changed obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2019-01-08 09:13:38 DEBUG certificates-relation-changed TypeError: expected string or buffer
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed /var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py: something went wrong: Unable to establish connection to http://localhost:35337/v3/services?
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed /var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py: something went wrong: Unable to establish connection to http://localhost:35337/v3/services?
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed /var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py: something went wrong: Unable to establish connection to http://localhost:35337/v3/services?
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed Retrying '%s' %d more times (delay=get_keystone_manager)
2019-01-08 09:13:38 DEBUG certificates-relation-changed /var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py: something went wrong: Unable to establish connection to http://localhost:35337/v3/services?
2019-01-08 09:13:38 DEBUG certificates-relation-changed /var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py: something went wrong: expected string or buffer
2019-01-08 09:13:38 DEBUG certificates-relation-changed Traceback (most recent call last):
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 585, in <module>
2019-01-08 09:13:38 DEBUG certificates-relation-changed uds_client.send(result_json)
2019-01-08 09:13:38 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/uds_comms.py", line 227, in send
2019-01-08 09:13:38 DEBUG certificates-relation-changed raise UDSException(str(e))
2019-01-08 09:13:38 DEBUG certificates-relation-changed uds_comms.UDSException: [Errno 32] Broken pipe
2019-01-08 09:13:38 ERROR juju.worker.uniter.operation runhook.go:132 hook "certificates-relation-changed" failed: exit status 1

Revision history for this message
Frode Nordahl (fnordahl) wrote :

# cat /var/log/apache2/error.log
[Tue Jan 08 09:10:19.337478 2019] [ssl:emerg] [pid 17268:tid 140175730857856] AH01903: Failed to configure CA certificate chain!
[Tue Jan 08 09:10:19.337549 2019] [ssl:emerg] [pid 17268:tid 140175730857856] AH02312: Fatal error initialising mod_ssl, exiting.
AH00016: Configuration Failed

# cat /etc/apache2/ssl/keystone/cert_10.219.3.172
[ snip ]
w5m5YwDz0BoWXHt3saHMllfd1rDNpu+UvZVrqAEKBVXha+iV4XBZP+1GxjitpclG
ny7J+kEk
-----END CERTIFICATE----------BEGIN CERTIFICATE-----
MIIDADCCAeigAwIBAgIUL1jUPL/4OJ9hGr/9KfSvCXErLncwDQYJKoZIhvcNAQEL
[ snip ]

Looking at the above it appears a separator between the host certificate and the chain certificate is missing.

I confirmed this to be the case by manually editing the file and adding a newline between the end/begin certificate markers.

This occurs with Keystone charm from latest stable and latest master.

Revision history for this message
Frode Nordahl (fnordahl) wrote :

This can be validated using the charm-vault func-smoke test

Changed in charm-keystone:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Frode Nordahl (fnordahl)
milestone: none → 19.04
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/629129

Changed in charm-keystone:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-keystone (master)

Reviewed: https://review.openstack.org/629129
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=8fff100a0da559238563dc1070477f1ebd5a136c
Submitter: Zuul
Branch: master

commit 8fff100a0da559238563dc1070477f1ebd5a136c
Author: Frode Nordahl <email address hidden>
Date: Tue Jan 8 11:00:40 2019 +0100

    Sync charm-helpers to fix certificate separation

    Change-Id: I0af42ea9642f3bd9181d729d73219753621e29be
    Closes-Bug: #1810910

Changed in charm-keystone:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-keystone (stable/18.11)

Fix proposed to branch: stable/18.11
Review: https://review.openstack.org/629254

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-keystone (stable/18.11)

Reviewed: https://review.openstack.org/629254
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=6d4ba63814a0cd3c486702187702230fde6c31d7
Submitter: Zuul
Branch: stable/18.11

commit 6d4ba63814a0cd3c486702187702230fde6c31d7
Author: Frode Nordahl <email address hidden>
Date: Tue Jan 8 11:00:40 2019 +0100

    Sync charm-helpers to fix certificate separation

    Change-Id: I0af42ea9642f3bd9181d729d73219753621e29be
    Closes-Bug: #1810910
    (cherry picked from commit 8fff100a0da559238563dc1070477f1ebd5a136c)

David Ames (thedac)
Changed in charm-keystone:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers