ConnectFailure: Unable to establish connection to http://localhost:35347/v3/domains?

Bug #1754729 reported by Chris Gregan on 2018-03-09
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack keystone charm
Medium
David Ames

Bug Description

Openstack fails to deploy with the keystone charm retrying connections

2018-03-09 04:07:51 DEBUG juju-log cluster:24: User 'ceilometer' already has role 'ResellerAdmin' on tenant 'services' in domain 'service_domain'
2018-03-09 04:07:51 DEBUG cluster-relation-changed Traceback (most recent call last):
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 880, in <module>
2018-03-09 04:07:51 DEBUG cluster-relation-changed main()
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 873, in main
2018-03-09 04:07:51 DEBUG cluster-relation-changed hooks.execute(sys.argv)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-03-09 04:07:51 DEBUG cluster-relation-changed self._hooks[hook_name]()
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1449, in wrapped_f
2018-03-09 04:07:51 DEBUG cluster-relation-changed restart_functions)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 730, in restart_on_change_helper
2018-03-09 04:07:51 DEBUG cluster-relation-changed r = lambda_f()
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1448, in <lambda>
2018-03-09 04:07:51 DEBUG cluster-relation-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1650, in _inner_update_certs_if_available
2018-03-09 04:07:51 DEBUG cluster-relation-changed return f(*args, **kwargs)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 618, in cluster_changed
2018-03-09 04:07:51 DEBUG cluster-relation-changed update_all_identity_relation_units_force_sync()
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1800, in inner_synchronize_ca_if_changed2
2018-03-09 04:07:51 DEBUG cluster-relation-changed return f(*args, **kwargs)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 374, in update_all_identity_relation_units_force_sync
2018-03-09 04:07:51 DEBUG cluster-relation-changed update_all_identity_relation_units()
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 362, in update_all_identity_relation_units
2018-03-09 04:07:51 DEBUG cluster-relation-changed identity_changed(relation_id=rid, remote_unit=unit)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1449, in wrapped_f
2018-03-09 04:07:51 DEBUG cluster-relation-changed restart_functions)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 730, in restart_on_change_helper
2018-03-09 04:07:51 DEBUG cluster-relation-changed r = lambda_f()
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1448, in <lambda>
2018-03-09 04:07:51 DEBUG cluster-relation-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 1800, in inner_synchronize_ca_if_changed2
2018-03-09 04:07:51 DEBUG cluster-relation-changed return f(*args, **kwargs)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/cluster-relation-changed", line 449, in identity_changed
2018-03-09 04:07:51 DEBUG cluster-relation-changed add_service_to_keystone(relation_id, remote_unit)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 2114, in add_service_to_keystone
2018-03-09 04:07:51 DEBUG cluster-relation-changed service_domain_id = manager.resolve_domain_id(SERVICE_DOMAIN)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/manager.py", line 197, in resolve_domain_id
2018-03-09 04:07:51 DEBUG cluster-relation-changed domains = [d._info for d in self.api.domains.list()]
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/domains.py", line 90, in list
2018-03-09 04:07:51 DEBUG cluster-relation-changed return super(DomainManager, self).list(**kwargs)/#/qa/overview?start_date=One%20Year%20Ago&finish_date=Now
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
2018-03-09 04:07:51 DEBUG cluster-relation-changed return f(*args, **new_kwargs)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 390, in list
2018-03-09 04:07:51 DEBUG cluster-relation-changed self.collection_key)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 125, in _list
2018-03-09 04:07:51 DEBUG cluster-relation-changed resp, body = self.client.get(url, **kwargs)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 288, in get
2018-03-09 04:07:51 DEBUG cluster-relation-changed return self.request(url, 'GET', **kwargs)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 447, in request
2018-03-09 04:07:51 DEBUG cluster-relation-changed resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 192, in request
2018-03-09 04:07:51 DEBUG cluster-relation-changed return self.session.request(url, method, **kwargs)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2018-03-09 04:07:51 DEBUG cluster-relation-changed return wrapped(*args, **kwargs)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 430, in request
2018-03-09 04:07:51 DEBUG cluster-relation-changed resp = send(**kwargs)
2018-03-09 04:07:51 DEBUG cluster-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 474, in _send_request
2018-03-09 04:07:51 DEBUG cluster-relation-changed raise exceptions.ConnectionRefused(msg)
2018-03-09 04:07:51 DEBUG cluster-relation-changed keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://localhost:35347/v3/domains?

Chris Gregan (cgregan) wrote :
Chris Gregan (cgregan) wrote :

Escalated to field high for additional attention

Ryan Beisner (1chb1n) wrote :

Please attach the bundles and any other bundle inheritance files.

Ashley Lai (alai) wrote :

The bundle is attached. This deployment does not have any overlay yaml file.

Ryan Beisner (1chb1n) wrote :

https://bugs.launchpad.net/charm-keystone/+bug/1754729/+attachment/5074539/+files/juju-crashdump-9ba9ac17-4eaf-4f4f-a844-18c789659936.tar.gz

In this crashdump, all keystone units are reporting "message: Unit is ready and clustered". In fact, and all other units have a good status. This is at odds with the bug description.

rbeisner@rby:~/Downloads/juju-crashdump-9ba9ac17-4eaf-4f4f-a844-18c789659936⟫ egrep "message:|current:" juju_status.yaml | pastebinit -a beisner
http://paste.ubuntu.com/p/vt4NHX8Fnp/

Ryan Beisner (1chb1n) on 2018-03-22
Changed in charm-keystone:
assignee: nobody → David Ames (thedac)
importance: Undecided → High
Ryan Beisner (1chb1n) wrote :

The deployment eventually succeeds, with what looks like failed hook retries. We generally do not want to lean on that for resolution, so I consider this bug/race valid. A hook should never go into error. We'll triage further and attempt to reproduce. Thank you.

Changed in charm-keystone:
status: New → In Progress
David Ames (thedac) wrote :

@Ashley,

So far my attempts to reproduce have not been productive. Can you tell me how often QA is seeing this error?

Ryan Beisner (1chb1n) wrote :

Given that this race is so far not reproducible in our lab, and the reporting lab as seen it two times, I suggest this is not necessarily a high SLA bug.

Ryan Beisner (1chb1n) wrote :

By measure of our SLA process, it is indeed an "Infrastructure or control plane component crashing," but given the rarity and inability to provide a clear reproducer, I would not consider that "Major" under the same SLA process.

Ryan Beisner (1chb1n) wrote :

We will continue to attempt to reproduce, and if it recurs with increasing impact, please do let us know.

Changed in charm-keystone:
importance: High → Medium
status: In Progress → Incomplete
Chris Gregan (cgregan) wrote :

@Ryan
I was asked to escalate this issue from management to get some additional attention. I could drop the field group, but only until we repro it again. I'll leave it in incomplete state until we see it again.

Ashley Lai (alai) wrote :

Yep we hit it two times. 3/9 and 3/11.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers