Keystone relation missing required data: service_port service_host

[Alexandros Soumplis]

Running openstack-train after the upgrade to latest charms version 20.02 openstack-dashboard is broken and it seems that cannot all the necessary data from keystone relation. The relevant log is:
2020-02-20 09:14:33 INFO juju-log Generating template context for identity-service
2020-02-20 09:14:33 INFO juju-log Missing required data: service_port service_host
2020-02-20 09:14:33 INFO juju-log Missing required data: service_port service_host
2020-02-20 09:14:33 INFO juju-log Missing required data: service_port service_host
2020-02-20 09:14:33 INFO juju-log Generating template context for identity-service
2020-02-20 09:14:33 INFO juju-log Missing required data: service_port service_host
2020-02-20 09:14:34 INFO juju-log Missing required data: service_port service_host
2020-02-20 09:14:34 INFO juju-log Missing required data: service_port service_host

The relation seems to be set but not have all required info:
ubuntu@cfp001:~$ juju run --unit openstack-dashboard/11 'relation-get -r identity-service:390 - keystone/12'
egress-subnets: 10.0.253.98/32
ingress-address: 10.0.253.98
private-address: 10.0.253.98

[Alex Kavanagh (ajkavanagh)]

Hi. There's unfortunately not enough information to resolve this bug.

Please could you provide:

- the output of "juju status"
- Log files of the keystone and openstack-dashboard units from before to after the charm upgrade.
- if possible, the versions of the charms prior to the upgrade.
- the output of juju config for the keystone and openstack-dashboard units.

Thanks.

[Alexandros Soumplis (soumplis)]

Log files and status info. Logs prior to upgrade are not available as all units have been rebuild just in case it could resolve the issue. The version prior to upgrade also not available but almost certainly the latest pre-20.20 release.

[Alexandros Soumplis (soumplis)]

OK, seems to have resolved the issue in a very strange way. It seems that if the host cannot resolve the hostname of the remote unit, it just fails (?) in silence. Setting the correct DNS server (also re-established the relation but maybe it wasn't needed) the relation is established with complete data and dashboard works as expected.

I am not sure that this issue can be handled in the charm or the juju level but there should be a clear error message when there are resolving or connectivity errors with remote hosts that affect operation.

[Tom-Erik Røberg (tom-erik-roberg)]

We were hit by a similar issue when migrating our HA keystone service to
a new set of units. We did the move by expanding to a total of six nodes and
then reducing to three nodes. We have done similar moves with cinder and glance.

After removing the old units we got "Incomplete relations: identity" on the
openstack-dashboard units.

With some debugging of the relation between openstack-dashboard and keystone we
found that it's only the leader unit for keystone that provides the keystone
configuration variables. With cinder and glance we see that all units provides
this information. Removing the current leader unit gives a state where none of
the relations provides the necessary configuration.

A workaround is to remove and re-add the relation.

In production we are running older revisions of openstack-dashboard and
keystone (245 and 264). We reproduced the issue with openstack-dashboard
revision 299 and keystone revision 310.

Reproduction with juju commands and output attached in debug.txt.

[Launchpad Janitor (janitor)]

[Expired for OpenStack openstack-dashboard charm because there has been no activity for 60 days.]

[Robert Gildein (rgildein)]

I managed to reproduce it [1].

---
[1]: https://pastebin.ubuntu.com/p/2S3TNrGYV4/

[Robert Gildein (rgildein)]

I find out that from keystone revision 269, this issue should be fixed by
this function [1].
I've tried it with the revision 269 and it works fine [2].

I think this bug should be flagged as "Invalid".

FYI: I tried it also with the revision 538 and it also works fine, but I came
across another issue [3].

---
[1]: https://opendev.org/openstack/charm-keystone/src/commit/68a0c87235b84f382b1168e7694007816c42ced2/hooks/keystone_hooks.py#L676
[2]: https://pastebin.ubuntu.com/p/PY7zd5RkZq/
[3]: https://bugs.launchpad.net/keystone/+bug/1950987