Upgrade to pike did not create cinderv3 endpoints

[Xav Paice]

Charm 18.11, Xenial.

I ran an upgrade from Ocata to Pike, was expecting the Cinder v3 endpoints to appear in the Keystone catalog but they did not. This resulted in errors attaching volumes to instances:

From nova-api log:

2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 336, in wrapped
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 108, in wrapper
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 108, in wrapper
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/volumes.py", line 340, in create
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions volume_id, device, tag=tag)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 201, in inner
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions return function(self, context, instance, *args, **kwargs)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 149, in inner
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions return f(self, context, instance, *args, **kw)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3820, in attach_volume
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions disk_bus, device_type, tag=tag)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3756, in _attach_volume
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions volume_bdm.destroy()
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions self.force_reraise()
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3752, in _attach_volume
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions self._check_attach_and_reserve_volume(context, volume_id, instance)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3734, in _check_attach_and_reserve_volume
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions volume = self.volume_api.get(context, volume_id)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 235, in wrapper
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions res = method(self, ctx, *args, **kwargs)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 257, in wrapper
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions res = method(self, ctx, volume_id, *args, **kwargs)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 314, in get
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions item = cinderclient(context).volumes.get(volume_id)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 116, in cinderclient
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions url = _SESSION.get_endpoint(auth, **service_parameters)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 947, in get_endpoint
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions return auth.get_endpoint(self, **kwargs)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 378, in get_endpoint
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions allow_version_hack=allow_version_hack, **kwargs)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 277, in get_endpoint_data
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions service_name=service_name)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in inner
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions return wrapped(*args, **kwargs)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/keystoneauth1/access/service_catalog.py", line 407, in endpoint_data_for
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions raise exceptions.EndpointNotFound(msg)
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions EndpointNotFound: publicURL endpoint for volumev3 service named cinderv3 in RegionOne region not found
2018-12-24 04:37:37.316 7460 ERROR nova.api.openstack.extensions

[James Page (james-page)]

Charm needs to re-execution the identity-service joined hook to get the new endpoints created.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/640726

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/640726
Committed: https://git.openstack.org/cgit/openstack/charm-cinder/commit/?id=8d977e815205cc7410b6d6abd222740809e7c05b
Submitter: Zuul
Branch: master

commit 8d977e815205cc7410b6d6abd222740809e7c05b
Author: James Page <email address hidden>
Date: Mon Mar 4 10:42:27 2019 +0000

    Ensure identity-joined execute post upgrade.

    Re-execute the identity-joined relation to ensure that cinder v3
    API endpoints are registered for OpenStack Pike or later.

    Change-Id: Ia259696b787188218003f63ab510eb143c0ea563
    Closes-Bug: 1809634

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/18.11
Review: https://review.openstack.org/640778

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/640778
Committed: https://git.openstack.org/cgit/openstack/charm-cinder/commit/?id=cfbb085cb36dc9065fc27567f3efafc1bb4e87e3
Submitter: Zuul
Branch: stable/18.11

commit cfbb085cb36dc9065fc27567f3efafc1bb4e87e3
Author: James Page <email address hidden>
Date: Mon Mar 4 10:42:27 2019 +0000

    Ensure identity-joined execute post upgrade.

    Re-execute the identity-joined relation to ensure that cinder v3
    API endpoints are registered for OpenStack Pike or later.

    Change-Id: Ia259696b787188218003f63ab510eb143c0ea563
    Closes-Bug: 1809634
    (cherry picked from commit 8d977e815205cc7410b6d6abd222740809e7c05b)