Glance image-create stuck in SAVING state and when deleted image remains on the filesystem

[semy]

I'm using Icehouse (Ubuntu 14.04) and a directory as a backend is configured in Glance to store images.

When I perform image-create and it takes a long period of time, glance ends up with "NotAuthenticated: Authentication required" exception and "Failed to upload image" in logs. Later glance reports "Unable to kill image" because of the very same error "NotAuthenticated" and because previous action failed and generally it makes the image to stuck in SAVING state.

More magic is after deleting that broken image. Glance reports that the image was deleted successfully and there is no error in logs, but it turns out only data from the DB was removed, but the image itself (file in the directory) remains untouched. It's not longer visible in glance image-list, but is still in glance's images directory.

NotAuthenticated error is probably caused by Keystone tokens. When I increased keystone token expiration time it looks like the amount of time to cause very same situation was also increased. However, when I was uploading multiple images during the same session in Horizon I ended up with all successfull image-create except the last one which was during the time when the token should expire, so still it's an issue.

Nevertheless, Glance should do the whole cleanup properly even if the image is in e.g. SAVING state and it should look for a file matching that broken image and purge it from the disk, if such file exists.

[Bellantuono Daniel (kelfen)]

I have the same problem.
At the moment my token expiration time is 4 hours, so if the image-create operation hold up more than 4 hours (i'm using Swift backend) Glance fail and log this error: "NotAuthenticated"
Would it be possible to request a new token?

[cristi1979 (cristi-falcas)]

Hi,

I'm having the same problem in Juno.

How I can increase the expiration time for keystone tokens?

[cristi1979 (cristi-falcas)]

sorry, just found it:
[token]
expiration=3600

[Nha Pham (phqnha)]

Hello, could you possibly post the error log of glance? That would help me a lot.

[Alex Schultz (alex-schultz)]

I was looking into a similar issue that I believe is caused by this. Basically we're doing a v1 glance call and, as far as I can tell, midway through the call it's getting a NotAuthorized exception [0] which leads to a 500 being returned and the image stuck in a queued state [1].

[0] http://paste.openstack.org/show/205021/
[1] http://paste.openstack.org/show/205023/

[Nha Pham (phqnha)]

Hi Alex, thank you very much for your help.
In your test, is the image uploaded to the server? I think the NotAuthorized Exeption when uploading image is not a bug (or a very low priority bug). However, an image is uploaded to server but cannot be deleted from the store is a serious bug, and I only concern about it.