glance restarted during image upload, image stuck in "saving" state
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Glance - Simplestreams Sync Charm |
Invalid
|
Undecided
|
Данило Шеган | ||
Landscape Server |
Fix Released
|
Medium
|
Данило Шеган | ||
16.06 |
Fix Released
|
Medium
|
Данило Шеган | ||
simplestreams |
Fix Released
|
Medium
|
Данило Шеган | ||
simplestreams (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Xenial |
Won't Fix
|
Undecided
|
Unassigned | ||
Bionic |
Fix Released
|
Undecided
|
Unassigned | ||
Disco |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
[Impact]
CI run: https:/
gs3 log attached. Below some excerpts:
It looks like the problem started when it uploaded the first image (14.04) to glance. That failed:
WARNING * 05-18 13:36:48 [PID:10259] * root * No rabbitmq connection available for msg{'status': 'Syncing', 'message': 'ubuntu-
DEBUG * 05-18 13:36:50 [PID:10259] * glanceclient.
INFO * 05-18 13:37:01 [PID:10364] * root * /var/run/
ERROR * 05-18 13:37:07 [PID:10259] * root * Glance Client exception during do_sync:
Error communicating with http://
Will continue polling.
Traceback (most recent call last):
File "/usr/share/
do_
File "/usr/share/
tmirror.
File "/usr/lib/
return self.sync_
File "/usr/lib/
self.
File "/usr/lib/
return self.sync_
File "/usr/lib/
self.
File "/usr/lib/
ret = self.gclient.
File "/usr/lib/
'POST', '/v1/images', headers=hdrs, body=image_data)
File "/usr/lib/
return self._http_
File "/usr/lib/
raise exc.Communicati
CommunicationError: Error communicating with http://
INFO * 05-18 13:37:07 [PID:10259] * root * sync done.
INFO * 05-18 13:38:01 [PID:10372] * root * glance-
[Test Case]
Found evidence in the juju logs for glance/0 that it restarted right at 13:37:07:
2016-05-18 13:37:07 INFO ceph-relation-
2016-05-18 13:37:07 INFO juju-log ceph:50: Loaded template from templates/ceph.conf
2016-05-18 13:37:07 INFO juju-log ceph:50: Rendering from template: ceph.conf
2016-05-18 13:37:07 INFO juju-log ceph:50: Wrote template /var/lib/
2016-05-18 13:37:07 INFO ceph-relation-
2016-05-18 13:37:07 INFO ceph-relation-
glance/1 had its last restart later:
2016-05-18 13:32:01 INFO ceph-relation-
glance/2 at that time too:
2016-05-18 13:32:00 INFO ceph-relation-
In gs3, a few log entries later, we can see that 14.04 is in state "saving" in glance:
(...)
{"images": [{"status": "saving", "deleted_at": null, "name": "auto-sync/
(...)
It remains in this state throughout the logs.
gs3 then proceeds to download 16.04, upload it to glance, and then finally publishes the streams. It's unknown if 14.04 is part of this publication or not.
[Regression Potential]
This part was added specifically for the Xenial backport, including:
- 433-glance-
And chances of regression are small based on the MR feedback from SEG and this particular bug already stating the issue was fixed.
[Other Info]
Landscape however fails, because it doesn't find enough "active" images:
May 18 14:33:29 job-handler-1 INFO RetryingCall for '_wait_
May 18 14:33:29 job-handler-1 INFO Traceback: <class 'canonical.
Related branches
- simplestreams-dev: Pending requested
-
Diff: 15 lines (+5/-0)1 file modifiedsimplestreams/mirrors/glance.py (+5/-0)
- Rafael David Tinoco (community): Disapprove
- Bryce Harrington (community): Approve
- Felipe Reyes (community): Approve
- Edward Hope-Morley: Pending requested
-
Diff: 1851 lines (+1732/-11)10 files modifieddebian/changelog (+15/-0)
debian/patches/428-do-not-require-that-hypervisor_config-be-present.patch (+23/-0)
debian/patches/433-glance-ignore-inactive-images.patch (+42/-0)
debian/patches/435-glance-refactor-for-testing.patch (+853/-0)
debian/patches/436-glance-fix-race-conditions.patch (+479/-0)
debian/patches/450-453-454-keystone-v3-support.patch (+13/-10)
debian/patches/455-nova-lxd-support-squashfs-images.patch (+230/-0)
debian/patches/460-glance-handle-v2-auth-with-sessions.patch (+33/-0)
debian/patches/series (+8/-1)
debian/patches/skip-openstack-tests-if-no-libs.patch (+36/-0)
- Rafael David Tinoco (community): Disapprove
- Billy Olsen (community): Approve
- Scott Moser (community): Needs Resubmitting
- Felipe Reyes (community): Approve
- Eric Desrochers: Pending requested
-
Diff: 1845 lines (+1729/-11)10 files modifieddebian/changelog (+12/-0)
debian/patches/428-do-not-require-that-hypervisor_config-be-present.patch (+23/-0)
debian/patches/433-glance-ignore-inactive-images.patch (+42/-0)
debian/patches/435-glance-refactor-for-testing.patch (+853/-0)
debian/patches/436-glance-fix-race-conditions.patch (+479/-0)
debian/patches/450-453-454-keystone-v3-support.patch (+13/-10)
debian/patches/455-nova-lxd-support-squashfs-images.patch (+230/-0)
debian/patches/460-glance-handle-v2-auth-with-sessions.patch (+33/-0)
debian/patches/series (+8/-1)
debian/patches/skip-openstack-tests-if-no-libs.patch (+36/-0)
tags: | removed: kanban |
description: | updated |
Changed in glance-simplestreams-sync-charm: | |
status: | New → In Progress |
Changed in landscape: | |
status: | New → In Progress |
assignee: | nobody → Данило Шеган (danilo) |
Changed in glance-simplestreams-sync-charm: | |
status: | In Progress → Invalid |
Changed in simplestreams: | |
assignee: | nobody → Данило Шеган (danilo) |
Changed in simplestreams: | |
status: | New → Confirmed |
importance: | Undecided → Medium |
Changed in simplestreams: | |
status: | Confirmed → Fix Committed |
description: | updated |
Changed in simplestreams (Ubuntu): | |
status: | New → Fix Released |
Changed in simplestreams (Ubuntu Disco): | |
status: | New → Fix Released |
Changed in simplestreams (Ubuntu Bionic): | |
status: | New → Fix Released |
Changed in simplestreams (Ubuntu Xenial): | |
status: | New → In Progress |
assignee: | nobody → Rafael David Tinoco (rafaeldtinoco) |
Changed in simplestreams (Ubuntu Xenial): | |
status: | In Progress → Won't Fix |
Changed in simplestreams (Ubuntu Bionic): | |
status: | Fix Released → In Progress |
Changed in simplestreams (Ubuntu Xenial): | |
assignee: | Rafael David Tinoco (rafaeldtinoco) → nobody |
Changed in simplestreams (Ubuntu Bionic): | |
assignee: | nobody → Rafael David Tinoco (rafaeldtinoco) |
Changed in simplestreams (Ubuntu Bionic): | |
status: | In Progress → Fix Released |
assignee: | Rafael David Tinoco (rafaeldtinoco) → nobody |
Changed in landscape: | |
status: | Fix Committed → Fix Released |
Log showing the glance restart