Deleting p_s endpoint failed, it's not recreated
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Glance-Simplestreams-Sync Charm |
Triaged
|
Medium
|
Unassigned | ||
glance-simplestreams-sync (Juju Charms Collection) |
Invalid
|
Medium
|
James Page |
Bug Description
gs3 tries to update the product-streams by deleting and recreating it. Listing worked, deleting failed and it never got to the create part. In the end, p_s was broken in keystone catalog and juju couldn't bootstrap this cloud.
From update_
The endpoint is never recreated:
$ grep "Deleting existing product-streams endpoint" glance-
2
$ grep "creating product-streams endpoint" glance-
0
The logs show that listing endpoints worked, but deleting one of the listed endpoints just stalled for 30s and then failed (full logs attached, here is just an excerpt):
Listing endpoints is OK:
DEBUG * 11-17 08:46:20 [PID:13830] * requests.
DEBUG * 11-17 08:46:20 [PID:13830] * keystoneclient.
RESP BODY: {"OS-KSADM:
INFO * 11-17 08:46:20 [PID:13830] * root * ps_service_
...
Deleting fails:
INFO * 11-17 08:46:20 [PID:13830] * root * Updating product streams service.
DEBUG * 11-17 08:46:20 [PID:13830] * keystoneclient.
INFO * 11-17 08:46:20 [PID:13830] * requests.
DEBUG * 11-17 08:46:22 [PID:13830] * requests.
DEBUG * 11-17 08:46:22 [PID:13830] * keystoneclient.
RESP BODY: {"endpoints": [{"internalurl": "http://
INFO * 11-17 08:46:22 [PID:13830] * root * Executing command: ['juju-run', 'glance-
INFO * 11-17 08:46:22 [PID:13830] * root * Deleting existing product-streams endpoint:
DEBUG * 11-17 08:46:22 [PID:13830] * keystoneclient.
INFO * 11-17 08:46:22 [PID:13830] * requests.
ERROR * 11-17 08:46:52 [PID:13830] * root * Exception during syncing:
Traceback (most recent call last):
File "/usr/share/
update_
File "/usr/share/
ksc.
File "/usr/lib/
return self._delete(
File "/usr/lib/
return self.client.
File "/usr/lib/
return self.request(url, 'DELETE', **kwargs)
File "/usr/lib/
resp = super(LegacyJso
File "/usr/lib/
return self.session.
File "/usr/lib/
return func(*args, **kwargs)
File "/usr/lib/
resp = send(**kwargs)
File "/usr/lib/
raise exceptions.
ConnectFailure: Unable to establish connection to http://
The server, however, didn't log a failure for the DELETE operation:
(...)
127.0.0.1 - - [17/Nov/
127.0.0.1 - - [17/Nov/
10.96.10.116 - - [17/Nov/
127.0.0.1 - - [17/Nov/
(...)
Eventually we reach:WARNING * 11-17 08:48:19 [PID:14086] * root * found 0 product-streams endpoints in region region1, expecting one - not updating endpoint
And we have a service with no endpoint. Unfortunately I don't have a keystone catalog output for this situation.
tags: | removed: kanban-cross-team |
Changed in glance-simplestreams-sync (Juju Charms Collection): | |
assignee: | nobody → James Page (james-page) |
importance: | Undecided → Medium |
status: | New → In Progress |
Changed in charm-glance-simplestreams-sync: | |
assignee: | nobody → James Page (james-page) |
importance: | Undecided → Medium |
status: | New → In Progress |
Changed in glance-simplestreams-sync (Juju Charms Collection): | |
status: | In Progress → Invalid |
Changed in charm-glance-simplestreams-sync: | |
status: | In Progress → Triaged |
assignee: | James Page (james-page) → nobody |
Tarball with logs from keystone-*, glance-*, swift-*.