Image could not be found after upload. The image may have been deleted during the upload.

[melanie witt]

Saw this fail a tempest check job:

http://logs.openstack.org/81/142281/3/gate/gate-tempest-dsvm-neutron-full/960927f/

The error in the console says "Details: Volume c96a0a8c-b5b4-42b5-a1cd-f537d5eee094 failed to reach available status (current: creating) within the required time (196 s)."

And I find in the screen-c-vol.txt, the volume does get created successfully, but it's a few minutes after the test failure:

http://logs.openstack.org/81/142281/3/gate/gate-tempest-dsvm-neutron-full/960927f/logs/screen-c-vol.txt.gz#_2015-01-08_17_34_23_438

I tried a logstash query on the errors in screen-c-vol.txt and find 18 hits in the last 7 days, check and gate, all failures:

message:"could not be found after upload. The image may have been deleted during the upload" AND tags:"screen-c-vol.txt"

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiY291bGQgbm90IGJlIGZvdW5kIGFmdGVyIHVwbG9hZC4gVGhlIGltYWdlIG1heSBoYXZlIGJlZW4gZGVsZXRlZCBkdXJpbmcgdGhlIHVwbG9hZFwiIEFORCB0YWdzOlwic2NyZWVuLWMtdm9sLnR4dFwiIiwiZmllbGRzIjpbImJ1aWxkX3Nob3J0X3V1aWQiLCJidWlsZF9zdGF0dXMiLCJidWlsZF9xdWV1ZSJdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MjA3NTU3MTUxMTZ9

[melanie witt (melwitt)]

e-r query patch: https://review.openstack.org/#/c/145969/

[Matt Riedemann (mriedem)]

This could be a glance (or swift) problem given the trace, let's look for errors in the glance logs.

[Matt Riedemann (mriedem)]

Looks like this started on 1/5 really, then spiked on 1/9.

The only thing suspicious in python-glanceclient in that timeframe is this:

https://github.com/openstack/python-glanceclient/commit/df02ee8e2a10d2f37a9c013dec157c88b8dce49d

I see the root errors in the glance-api logs:

http://logs.openstack.org/93/130593/10/check/check-tempest-dsvm-full/1070ad7/logs/screen-g-api.txt.gz?level=TRACE#_2015-01-09_11_51_27_954

I'm not seeing any errors in the swift logs.

[Mike Perez (thingee)]

From Cinder's perspective, Cinder is uploading the volume to an image, but that fails because the image is not found. Cinder tries to clean up by deleting the image, but it's not found. I was not able to find a request sent during the upload that would indicate a request went to glance to delete the image.

[John Griffith (john-griffith)]

Actually if you look further up:
http://logs.openstack.org/66/145066/7/gate/gate-tempest-dsvm-full/233cf6d/logs/screen-c-vol.txt.gz#_2015-01-09_22_02_00_432

The root of the problem is that we failed the call out to Glance, looks like we exceeded the pool limit for Request. I believe the Glance failure is due to the Request Pool issue...

[Matt Riedemann (mriedem)]

The python-glanceclient bug for the http request pools connection issues is bug 1341777.

[Ian Cordasco (icordasc)]

The connection pool limit has nothing to do with this. When that limit is reached, a new connection is created regardless so there's no reason for these two things to be related.

[Matt Riedemann (mriedem)]

We haven't seen this in a long time so marking it invalid.