gs3 tried to delete an endpoint that keystone claims does not exist

Bug #1583762 reported by Francis Ginther on 2016-05-19
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance - Simplestreams Sync Charm
High
Данило Шеган
Landscape Server
Medium
Данило Шеган
16.06
Undecided
Данило Шеган

Bug Description

This was found with a Landscape autopilot deployment. The result is a failure of the keystone charm's identity-service-relation-changed hook (which fails the entire deploy).

I don't know if this is on gs3 or keystone. Gs3 call DELETE on "4760c1d6154b4b74a7b3c464c554a2af", http://paste.ubuntu.com/16512920/ (from /glance-symplestream-sync-0/var/log/glance-symplestream-sync.log).

Keystone claims this endpoint does not exist and the charm relation fails, http://paste.ubuntu.com/16513011/ (from /keystone-0/var/log/juju/unit-keystone-0.log)

It looks a bit like the "image-stream" endpoint is being updated from within a juju hook context and being deleted via the keystone REST api at the same time. But if this is the case, I can't figure out what triggered the identity-service-relation-changed in keystone. Another charm perhaps?

Attached are the unit logs from keystone and gs3. I have others if the logs from all other units if needed.

Related branches

Revision history for this message
Francis Ginther (fginther) wrote :
description: updated
description: updated
description: updated
description: updated
Revision history for this message
Данило Шеган (danilo) wrote :

FWIW, what the charm is doing is attempting to update the image-streams endpoint by deleting and recreating it first. This allows for a potential race, which is bad in itself. I think it's worth fixing it in the charm as well.

However, we'd have to make it conditional on the returned keystone client API version: 2.0 does not support the update() call, whereas 3.0 does (see http://docs.openstack.org/developer/python-keystoneclient/api/keystoneclient.v2_0.html#module-keystoneclient.v2_0.endpoints, http://docs.openstack.org/developer/python-keystoneclient/api/keystoneclient.v3.html#module-keystoneclient.v3.endpoints).

Changed in glance-simplestreams-sync-charm:
assignee: nobody → Данило Шеган (danilo)
Revision history for this message
Данило Шеган (danilo) wrote :

Unfortunately, in all my tests, we always get version V2.0 with keystone client. Any other ideas, anyone?

Revision history for this message
Данило Шеган (danilo) wrote :

Looking at the hook, this all stems from the fact that we are not related to swift or at least glance: instead of setting up the right endpoint there, the charm simply sets it to a bogus value. Thus, we can't rely on juju to ensure consistency since we are not using its features.

I feel like this could only be properly solved with refactoring of the gs3 charm: make it relate to glance (if that ensures that swift is available, if not, we'd have to relate it to either ceph-radosgw or swift, whichever is installed), and then simply set the endpoint in the hook, and drop it from the cronjob.

Revision history for this message
Данило Шеган (danilo) wrote :

Also note that our charm contains a fix for endpoints that might affect this, and the upstream charm is behaving slightly differently here (though it does do the same delete-create dance, the values it sets the endpoints to are different).

Revision history for this message
Данило Шеган (danilo) wrote :

After discussing with David, we've agreed to make the charm behave slightly better: only go through delete/create dance if the existing "image-stream" endpoint is set to a different URL. Since we expect swift address to remain the same during installation, at least we'll reduce the chance of such raciness.

Still, keystone charm should probably behave more atomically regarding this: once it fetches endpoint data, it should use it both for the check if it's defined and to generate the URL using the template.

Changed in landscape:
assignee: nobody → Данило Шеган (danilo)
Changed in glance-simplestreams-sync-charm:
importance: Undecided → High
status: New → In Progress
tags: added: kanban
tags: removed: kanban
Changed in glance-simplestreams-sync-charm:
status: In Progress → Fix Committed
Changed in landscape:
status: New → In Progress
Changed in landscape:
status: In Progress → Fix Committed
Changed in landscape:
importance: Undecided → Medium
Changed in landscape:
milestone: none → 16.07
Changed in landscape:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers