OpenStack Octavia Charm

Traceback with keystoneauth1.exceptions.discovery.DiscoveryFailure and keystoneauth1.exceptions.http.ServiceUnavailable

Bug #1809250 reported by Frode Nordahl on 2018-12-20

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Octavia Charm	Fix Released	High	Frode Nordahl	OpenStack Octavia Charm 19.04

Bug Description

In some situations the Octavia charm will end up in error state during deploy.

Log excerpt:
2018-12-20 12:00:01 INFO juju-log identity-service:57: Reactive main running for hook identity-service-relation-changed
2018-12-20 12:00:04 INFO juju-log identity-service:57: Initializing Leadership Layer (is leader)
2018-12-20 12:00:04 INFO juju-log identity-service:57: Invoking reactive handler: hooks/relations/keystone/requires.py:62:changed
2018-12-20 12:00:09 INFO juju-log identity-service:57: Invoking reactive handler: reactive/octavia_handlers.py:48:setup_neutron_lbaas_proxy
2018-12-20 12:00:09 INFO juju-log identity-service:57: Invoking reactive handler: reactive/octavia_handlers.py:69:setup_hm_port
2018-12-20 12:00:11 DEBUG identity-service-relation-changed Failed to discover available identity versions when contacting http://172.16.122.20:35357/. Attempting to parse version from URL.
2018-12-20 12:00:11 ERROR juju-log identity-service:57: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
    authenticated=False)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
    authenticated=authenticated)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery
    disc = Discover(session, url, authenticated=authenticated)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__
    authenticated=authenticated)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
    resp = session.get(url, headers=headers, authenticated=authenticated)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get
    return self.request(url, 'GET', **kwargs)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/session.py", line 869, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.ServiceUnavailable: Service Unavailable (HTTP 503)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-octavia-0/charm/reactive/octavia_handlers.py", line 87, in setup_hm_port
    octavia_charm):
  File "lib/charm/openstack/api_crud.py", line 276, in setup_hm_port
    octavia_charm.local_address)
  File "lib/charm/openstack/api_crud.py", line 154, in get_hm_port
    resp = nc.list_networks(tags='charm-octavia')
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 809, in list_networks
    **_params)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 369, in list
    for r in self._pagination(collection, path, **params):
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
    res = self.get(path, params=params)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 354, in get
    headers=headers, params=params)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
    headers=headers, params=params)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 282, in do_request
    headers=headers)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/client.py", line 342, in do_request
    self._check_uri_length(url)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/client.py", line 335, in _check_uri_length
    uri_len = len(self.endpoint_url) + len(url)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/client.py", line 349, in endpoint_url
    return self.get_endpoint()
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint
    return self.session.get_endpoint(auth or self.auth, **kwargs)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
    return auth.get_endpoint(self, **kwargs)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
    allow_version_hack=allow_version_hack, **kwargs)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
    service_catalog = self.get_access(session).service_catalog
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
    self._plugin = self._do_create_plugin(session)
  File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 161, in _do_create_plugin
    'auth_url is correct. %s' % e)
keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Service Unavailable (HTTP 503)

2018-12-20 12:00:11 DEBUG identity-service-relation-changed Traceback (most recent call last):
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 138, in _do_create_plugin
2018-12-20 12:00:11 DEBUG identity-service-relation-changed authenticated=False)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 610, in get_discovery
2018-12-20 12:00:11 DEBUG identity-service-relation-changed authenticated=authenticated)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/discover.py", line 1442, in get_discovery
2018-12-20 12:00:11 DEBUG identity-service-relation-changed disc = Discover(session, url, authenticated=authenticated)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/discover.py", line 526, in __init__
2018-12-20 12:00:11 DEBUG identity-service-relation-changed authenticated=authenticated)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/discover.py", line 101, in get_version_data
2018-12-20 12:00:11 DEBUG identity-service-relation-changed resp = session.get(url, headers=headers, authenticated=authenticated)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/session.py", line 1011, in get
2018-12-20 12:00:11 DEBUG identity-service-relation-changed return self.request(url, 'GET', **kwargs)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/session.py", line 869, in request
2018-12-20 12:00:11 DEBUG identity-service-relation-changed raise exceptions.from_response(resp, method, url)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed keystoneauth1.exceptions.http.ServiceUnavailable: Service Unavailable (HTTP 503)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed
2018-12-20 12:00:11 DEBUG identity-service-relation-changed During handling of the above exception, another exception occurred:
2018-12-20 12:00:11 DEBUG identity-service-relation-changed
2018-12-20 12:00:11 DEBUG identity-service-relation-changed Traceback (most recent call last):
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/charm/hooks/identity-service-relation-changed", line 22, in <module>
2018-12-20 12:00:11 DEBUG identity-service-relation-changed main()
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main
2018-12-20 12:00:11 DEBUG identity-service-relation-changed bus.dispatch(restricted=restricted_mode)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
2018-12-20 12:00:11 DEBUG identity-service-relation-changed _invoke(other_handlers)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
2018-12-20 12:00:11 DEBUG identity-service-relation-changed handler.invoke()
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
2018-12-20 12:00:11 DEBUG identity-service-relation-changed self._action(*args)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/charm/reactive/octavia_handlers.py", line 87, in setup_hm_port
2018-12-20 12:00:11 DEBUG identity-service-relation-changed octavia_charm):
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "lib/charm/openstack/api_crud.py", line 276, in setup_hm_port
2018-12-20 12:00:11 DEBUG identity-service-relation-changed octavia_charm.local_address)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "lib/charm/openstack/api_crud.py", line 154, in get_hm_port
2018-12-20 12:00:11 DEBUG identity-service-relation-changed resp = nc.list_networks(tags='charm-octavia')
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 809, in list_networks
2018-12-20 12:00:11 DEBUG identity-service-relation-changed **_params)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 369, in list
2018-12-20 12:00:11 DEBUG identity-service-relation-changed for r in self._pagination(collection, path, **params):
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2018-12-20 12:00:11 DEBUG identity-service-relation-changed res = self.get(path, params=params)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 354, in get
2018-12-20 12:00:11 DEBUG identity-service-relation-changed headers=headers, params=params)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2018-12-20 12:00:11 DEBUG identity-service-relation-changed headers=headers, params=params)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 282, in do_request
2018-12-20 12:00:11 DEBUG identity-service-relation-changed headers=headers)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/client.py", line 342, in do_request
2018-12-20 12:00:11 DEBUG identity-service-relation-changed self._check_uri_length(url)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/client.py", line 335, in _check_uri_length
2018-12-20 12:00:11 DEBUG identity-service-relation-changed uri_len = len(self.endpoint_url) + len(url)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/neutronclient/client.py", line 349, in endpoint_url
2018-12-20 12:00:11 DEBUG identity-service-relation-changed return self.get_endpoint()
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 247, in get_endpoint
2018-12-20 12:00:11 DEBUG identity-service-relation-changed return self.session.get_endpoint(auth or self.auth, **kwargs)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/session.py", line 1113, in get_endpoint
2018-12-20 12:00:11 DEBUG identity-service-relation-changed return auth.get_endpoint(self, **kwargs)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 380, in get_endpoint
2018-12-20 12:00:11 DEBUG identity-service-relation-changed allow_version_hack=allow_version_hack, **kwargs)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2018-12-20 12:00:11 DEBUG identity-service-relation-changed service_catalog = self.get_access(session).service_catalog
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2018-12-20 12:00:11 DEBUG identity-service-relation-changed self.auth_ref = self.get_auth_ref(session)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 206, in get_auth_ref
2018-12-20 12:00:11 DEBUG identity-service-relation-changed self._plugin = self._do_create_plugin(session)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-octavia-0/.venv/lib/python3.6/site-packages/keystoneauth1/identity/generic/base.py", line 161, in _do_create_plugin
2018-12-20 12:00:11 DEBUG identity-service-relation-changed 'auth_url is correct. %s' % e)
2018-12-20 12:00:11 DEBUG identity-service-relation-changed keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Service Unavailable (HTTP 503)
2018-12-20 12:00:11 ERROR juju.worker.uniter.operation runhook.go:132 hook "identity-service-relation-changed" failed: exit status 1

Frode Nordahl (fnordahl) on 2018-12-20

Changed in charm-octavia:
status:	New → Triaged
importance:	Undecided → High
assignee:	nobody → Frode Nordahl (fnordahl)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-12-20: Fix proposed to charm-octavia (master)

Fix proposed to branch: master
Review: https://review.openstack.org/626575

Changed in charm-octavia:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-01-10: Fix merged to charm-octavia (master)

Reviewed: https://review.openstack.org/626575
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=2224f4ebf71a739dd2f89f1ae918b53f2e032e21
Submitter: Zuul
Branch: master

commit 2224f4ebf71a739dd2f89f1ae918b53f2e032e21
Author: Frode Nordahl <email address hidden>
Date: Thu Dec 20 14:16:00 2018 +0100

Catch two more exceptions caused by temporary API unavailability

Change-Id: I89ff4effc8da8f5ebe5d9aa2e43c34bbe6ee3df5
Closes-Bug: #1809250

Changed in charm-octavia:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-01-10: Fix proposed to charm-octavia (stable/18.11)

Fix proposed to branch: stable/18.11
Review: https://review.openstack.org/629779

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-01-14: Fix merged to charm-octavia (stable/18.11)

Reviewed: https://review.openstack.org/629779
Committed: https://git.openstack.org/cgit/openstack/charm-octavia/commit/?id=b9abc69a60cd4757e3510b967e983f3b29aedcd9
Submitter: Zuul
Branch: stable/18.11

commit b9abc69a60cd4757e3510b967e983f3b29aedcd9
Author: Frode Nordahl <email address hidden>
Date: Thu Dec 20 14:16:00 2018 +0100

Catch two more exceptions caused by temporary API unavailability

    Change-Id: I89ff4effc8da8f5ebe5d9aa2e43c34bbe6ee3df5
    Closes-Bug: #1809250
    (cherry picked from commit 2224f4ebf71a739dd2f89f1ae918b53f2e032e21)

James Page (james-page) on 2019-04-17

Changed in charm-octavia:
milestone:	none → 19.04

David Ames (thedac) on 2019-04-17

Changed in charm-octavia:
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.