keystoneauth1.exceptions.http.BadGateway: Bad Gateway during identity-service-relation-changed

[Francis Ginther]

Hit this when deploying a swift/iscsi landscape autopilot openstack deployment. It caused a fatal hook error and a failure of the overall cloud deployment.

[From /var/log/juju/unit-ceph-radosgw-1.log (see http://paste.ubuntu.com/16748660/ for better formatting)]
2016-05-26 18:35:01 INFO identity-service-relation-changed Traceback (most recent call last):
2016-05-26 18:35:01 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-1/charm/hooks/identity-service-relation-changed", line 466, in <module>
2016-05-26 18:35:01 INFO identity-service-relation-changed hooks.execute(sys.argv)
2016-05-26 18:35:01 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-1/charm/hooks/charmhelpers/core/hookenv.py", line 717, in execute
2016-05-26 18:35:01 INFO identity-service-relation-changed self._hooks[hook_name]()
2016-05-26 18:35:01 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-1/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1573, in wrapped_f
2016-05-26 18:35:01 INFO identity-service-relation-changed restart_functions)
2016-05-26 18:35:01 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-1/charm/hooks/charmhelpers/core/host.py", line 478, in restart_on_change_helper
2016-05-26 18:35:01 INFO identity-service-relation-changed r = lambda_f()
2016-05-26 18:35:01 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-1/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1572, in <lambda>
2016-05-26 18:35:01 INFO identity-service-relation-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2016-05-26 18:35:01 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-1/charm/hooks/identity-service-relation-changed", line 345, in identity_changed
2016-05-26 18:35:01 INFO identity-service-relation-changed identity_joined(relid)
2016-05-26 18:35:01 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-1/charm/hooks/identity-service-relation-changed", line 339, in identity_joined
2016-05-26 18:35:01 INFO identity-service-relation-changed setup_keystone_certs()
2016-05-26 18:35:01 INFO identity-service-relation-changed File "/var/lib/juju/agents/unit-ceph-radosgw-1/charm/hooks/identity-service-relation-changed", line 191, in setup_keystone_certs
2016-05-26 18:35:01 INFO identity-service-relation-changed ca_cert = keystone.certificates.get_ca_certificate()
2016-05-26 18:35:01 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/certificates.py", line 29, in get_ca_certificate
2016-05-26 18:35:01 INFO identity-service-relation-changed resp, body = self._client.get('/certificates/ca', authenticated=False)
2016-05-26 18:35:01 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 173, in get
2016-05-26 18:35:01 INFO identity-service-relation-changed return self.request(url, 'GET', **kwargs)
2016-05-26 18:35:01 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 331, in request
2016-05-26 18:35:01 INFO identity-service-relation-changed resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2016-05-26 18:35:01 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
2016-05-26 18:35:01 INFO identity-service-relation-changed return self.session.request(url, method, **kwargs)
2016-05-26 18:35:01 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-05-26 18:35:01 INFO identity-service-relation-changed return func(*args, **kwargs)
2016-05-26 18:35:01 INFO identity-service-relation-changed File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 420, in request
2016-05-26 18:35:01 INFO identity-service-relation-changed raise exceptions.from_response(resp, method, url)
2016-05-26 18:35:01 INFO identity-service-relation-changed keystoneauth1.exceptions.http.BadGateway: Bad Gateway (HTTP 502)
2016-05-26 18:35:01 ERROR juju.worker.uniter.operation runhook.go:107 hook "identity-service-relation-changed" failed: exit status 1

[Francis Ginther (fginther)]

Attaching logs from all ceph-radosgw and keystone units.

[James Page (james-page)]

setup_keystone_certs is used to retrieve the CA and signing certs from keystone for processing token revocation lists.

The code deals with errors returned directly from keystone, but does not cover a BadGateway error that may happen in the event that the keystone haproxy service is unable to contact the keystone wsgi backend under Apache.

[James Page (james-page)]

code:

    except (ConnectionRefused, requests.exceptions.ConnectionError, Forbidden):

[James Page (james-page)]

Fixing this by removing all of the revocation list processing code - its only needed for PKI and PKI is no longer supported in the keystone charm

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/636593
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-radosgw/commit/?id=76cddc525e0e1d1f9ef9766b19a87c8a6e6c7aca
Submitter: Zuul
Branch: master

commit 76cddc525e0e1d1f9ef9766b19a87c8a6e6c7aca
Author: James Page <email address hidden>
Date: Wed Feb 13 11:09:00 2019 +0000

    Drop keystone integration for PKI token format

    The PKI token format is no longer supported by the keystone charm;
    drop code, tests and associated template fragments which deal with
    configuration of PKI revocation list processing.

    Change-Id: Ie08779c2aef15589b621c324808bb13089fb4f72
    Closes-Bug: 1586550