Glance image-create stuck in SAVING state and when deleted image remains on the filesystem

Bug #1371121 reported by semy on 2014-09-18
40
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Glance
Medium
Nha Pham

Bug Description

I'm using Icehouse (Ubuntu 14.04) and a directory as a backend is configured in Glance to store images.

When I perform image-create and it takes a long period of time, glance ends up with "NotAuthenticated: Authentication required" exception and "Failed to upload image" in logs. Later glance reports "Unable to kill image" because of the very same error "NotAuthenticated" and because previous action failed and generally it makes the image to stuck in SAVING state.

More magic is after deleting that broken image. Glance reports that the image was deleted successfully and there is no error in logs, but it turns out only data from the DB was removed, but the image itself (file in the directory) remains untouched. It's not longer visible in glance image-list, but is still in glance's images directory.

NotAuthenticated error is probably caused by Keystone tokens. When I increased keystone token expiration time it looks like the amount of time to cause very same situation was also increased. However, when I was uploading multiple images during the same session in Horizon I ended up with all successfull image-create except the last one which was during the time when the token should expire, so still it's an issue.

Nevertheless, Glance should do the whole cleanup properly even if the image is in e.g. SAVING state and it should look for a file matching that broken image and purge it from the disk, if such file exists.

Changed in glance:
assignee: nobody → agileclipse (agileclipse)
Bellantuono Daniel (kelfen) wrote :

I have the same problem.
At the moment my token expiration time is 4 hours, so if the image-create operation hold up more than 4 hours (i'm using Swift backend) Glance fail and log this error: "NotAuthenticated"
Would it be possible to request a new token?

Dolph Mathews (dolph) on 2014-10-03
tags: added: icehouse-backport-potential
Changed in glance:
importance: Undecided → Medium
status: New → Triaged
cristi1979 (cristi-falcas) wrote :

Hi,

I'm having the same problem in Juno.

How I can increase the expiration time for keystone tokens?

cristi1979 (cristi-falcas) wrote :

sorry, just found it:
[token]
expiration=3600

Nha Pham (phqnha) wrote :

Hello, could you possibly post the error log of glance? That would help me a lot.

Changed in glance:
assignee: agileclipse (agileclipse) → Nha Pham (phqnha)
Alex Schultz (alex-schultz) wrote :

I was looking into a similar issue that I believe is caused by this. Basically we're doing a v1 glance call and, as far as I can tell, midway through the call it's getting a NotAuthorized exception [0] which leads to a 500 being returned and the image stuck in a queued state [1].

[0] http://paste.openstack.org/show/205021/
[1] http://paste.openstack.org/show/205023/

Nha Pham (phqnha) wrote :

Hi Alex, thank you very much for your help.
In your test, is the image uploaded to the server? I think the NotAuthorized Exeption when uploading image is not a bug (or a very low priority bug). However, an image is uploaded to server but cannot be deleted from the store is a serious bug, and I only concern about it.

Alan Pevec (apevec) on 2015-11-24
tags: removed: icehouse-backport-potential
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers