gs3 tried to delete an endpoint that keystone claims does not exist

[Francis Ginther]

This was found with a Landscape autopilot deployment. The result is a failure of the keystone charm's identity-service-relation-changed hook (which fails the entire deploy).

I don't know if this is on gs3 or keystone. Gs3 call DELETE on "4760c1d6154b4b74a7b3c464c554a2af", http://paste.ubuntu.com/16512920/ (from /glance-symplestream-sync-0/var/log/glance-symplestream-sync.log).

Keystone claims this endpoint does not exist and the charm relation fails, http://paste.ubuntu.com/16513011/ (from /keystone-0/var/log/juju/unit-keystone-0.log)

It looks a bit like the "image-stream" endpoint is being updated from within a juju hook context and being deleted via the keystone REST api at the same time. But if this is the case, I can't figure out what triggered the identity-service-relation-changed in keystone. Another charm perhaps?

Attached are the unit logs from keystone and gs3. I have others if the logs from all other units if needed.

[Данило Шеган (danilo)]

FWIW, what the charm is doing is attempting to update the image-streams endpoint by deleting and recreating it first. This allows for a potential race, which is bad in itself. I think it's worth fixing it in the charm as well.

However, we'd have to make it conditional on the returned keystone client API version: 2.0 does not support the update() call, whereas 3.0 does (see http://docs.openstack.org/developer/python-keystoneclient/api/keystoneclient.v2_0.html#module-keystoneclient.v2_0.endpoints, http://docs.openstack.org/developer/python-keystoneclient/api/keystoneclient.v3.html#module-keystoneclient.v3.endpoints).

[Данило Шеган (danilo)]

Unfortunately, in all my tests, we always get version V2.0 with keystone client. Any other ideas, anyone?

[Данило Шеган (danilo)]

Looking at the hook, this all stems from the fact that we are not related to swift or at least glance: instead of setting up the right endpoint there, the charm simply sets it to a bogus value. Thus, we can't rely on juju to ensure consistency since we are not using its features.

I feel like this could only be properly solved with refactoring of the gs3 charm: make it relate to glance (if that ensures that swift is available, if not, we'd have to relate it to either ceph-radosgw or swift, whichever is installed), and then simply set the endpoint in the hook, and drop it from the cronjob.

[Данило Шеган (danilo)]

Also note that our charm contains a fix for endpoints that might affect this, and the upstream charm is behaving slightly differently here (though it does do the same delete-create dance, the values it sets the endpoints to are different).

[Данило Шеган (danilo)]

After discussing with David, we've agreed to make the charm behave slightly better: only go through delete/create dance if the existing "image-stream" endpoint is set to a different URL. Since we expect swift address to remain the same during installation, at least we'll reduce the chance of such raciness.

Still, keystone charm should probably behave more atomically regarding this: once it fetches endpoint data, it should use it both for the check if it's defined and to generate the URL using the template.