Failed to create Glance image loaded from remote URL

[Ilya Shakhat]

Ran the following command to create Glance image from Ubuntu repo:
glance --debug image-create --name ubuntu-14.04-server-cloudimg-amd64-2 --disk-format qcow2 --container-format bare --is-public True --copy-from https://cloud-images.ubuntu.com/releases/14.04.1/release/ubuntu-14.04-server-cloudimg-amd64-disk1.img

The command itself succeeded, returning HTTP 201 and image in queued status. After some time (~3-4 minutes) the image disappeared from the list of images. Glance API log contain error from Swift: BackendException: Failed to add object to Swift.

When image is dowloaded manually and added into Glance then the operation succeed.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "5.1.1"
  api: "1.0"
  build_number: "17"
  build_id: "2014-11-16_21-00-23"
  astute_sha: "702af3db6f5bca92525bc8322d7d5d7675ec857e"
  fuellib_sha: "0d3909b9a291880af28dbe48b9c7d25215aa98ea"
  ostf_sha: "64cb59c681658a7a55cc2c09d079072a41beb346"
  nailgun_sha: "2fc6fc4261092a591779a8fb7e3fb1623c6abb85"
  fuelmain_sha: "b118fa4475833ce031ef189ce280772c676fa1c9"

DEPLOYMENT:
  HA, 5 nodes, no Ceph, Neutron GRE

[Ilya Shakhat (shakhat)]

Glance API log:

2014-11-20T13:05:46.921041+00:00 err: 2014-11-20 13:05:46.913 18408 ERROR glance.store.swift [-] Error during chunked upload to backend, deleting stale chunks
2014-11-20 13:05:46.913 18408 TRACE glance.store.swift Traceback (most recent call last):
2014-11-20 13:05:46.913 18408 TRACE glance.store.swift File "/usr/lib/python2.7/dist-packages/glance/store/swift.py", line 435, in add
2014-11-20 13:05:46.913 18408 TRACE glance.store.swift content_length=content_length)
2014-11-20 13:05:46.913 18408 TRACE glance.store.swift File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1233, in put_object
2014-11-20 13:05:46.913 18408 TRACE glance.store.swift response_dict=response_dict)
2014-11-20 13:05:46.913 18408 TRACE glance.store.swift File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1142, in _retry
2014-11-20 13:05:46.913 18408 TRACE glance.store.swift reset_func(func, *args, **kwargs)
2014-11-20 13:05:46.913 18408 TRACE glance.store.swift File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1214, in _default_reset
2014-11-20 13:05:46.913 18408 TRACE glance.store.swift % (container, obj))
2014-11-20 13:05:46.913 18408 TRACE glance.store.swift ClientException: put_object('glance', 'f6f60be3-3b7b-4c90-8e61-164ca179dcde-00001', ...) failure and no ability to reset contents for reupload.
2014-11-20 13:05:46.913 18408 TRACE glance.store.swift
2014-11-20T13:05:46.925646+00:00 err: 2014-11-20 13:05:46.924 18408 ERROR glance.store.swift [-] Failed to add object to Swift.
Got error from Swift: put_object('glance', 'f6f60be3-3b7b-4c90-8e61-164ca179dcde-00001', ...) failure and no ability to reset contents for reupload.
2014-11-20T13:05:46.929666+00:00 err: 2014-11-20 13:05:46.927 18408 ERROR glance.api.v1.upload_utils [-] Failed to upload image f6f60be3-3b7b-4c90-8e61-164ca179dcde
2014-11-20 13:05:46.927 18408 TRACE glance.api.v1.upload_utils Traceback (most recent call last):
2014-11-20 13:05:46.927 18408 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py", line 105, in upload_data_to_store
2014-11-20 13:05:46.927 18408 TRACE glance.api.v1.upload_utils store)
2014-11-20 13:05:46.927 18408 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/glance/store/__init__.py", line 382, in store_add_to_backend
2014-11-20 13:05:46.927 18408 TRACE glance.api.v1.upload_utils (location, size, checksum, metadata) = store.add(image_id, data, size)
2014-11-20 13:05:46.927 18408 TRACE glance.api.v1.upload_utils File "/usr/lib/python2.7/dist-packages/glance/store/swift.py", line 503, in add
2014-11-20 13:05:46.927 18408 TRACE glance.api.v1.upload_utils raise glance.store.BackendException(msg)
2014-11-20 13:05:46.927 18408 TRACE glance.api.v1.upload_utils BackendException: Failed to add object to Swift.
2014-11-20 13:05:46.927 18408 TRACE glance.api.v1.upload_utils Got error from Swift: put_object('glance', 'f6f60be3-3b7b-4c90-8e61-164ca179dcde-00001', ...) failure and no ability to reset contents for reupload.
2014-11-20 13:05:46.927 18408 TRACE glance.api.v1.upload_utils

[Dmitry Mescheryakov (dmitrymex)]

Glance team, I've added 6.x as affected though it needs to be checked if that is true

[Inessa Vasilevskaya (ivasilevskaya)]

The issue is reproducible with current master and devstack installation. An attempt to create an image results in the forever 'queued' image status and NotAuthhenticated errors from keystone.

glance-api logs:

2014-11-24 20:15:09.886 26073 TRACE glance.api.v1.upload_utils raise exception.NotAuthenticated(res.read())
2014-11-24 20:15:09.886 26073 TRACE glance.api.v1.upload_utils NotAuthenticated: Authentication required
2014-11-24 20:15:09.886 26073 TRACE glance.api.v1.upload_utils
Traceback (most recent call last):
  File "/home/ina/projects/glance/.venv/local/lib/python2.7/site-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
    func(*args, **kwargs)
  File "/home/ina/projects/glance/glance/api/v1/images.py", line 691, in _upload_and_activate
    location_data = self._upload(req, image_meta)
  File "/home/ina/projects/glance/glance/api/v1/images.py", line 635, in _upload
    req, image_meta, image_data, store, self.notifier)
  File "/home/ina/projects/glance/glance/api/v1/upload_utils.py", line 269, in upload_data_to_store
    content_type='text/plain')
HTTPInternalServerError: Failed to upload image 3d42b60c-ad50-408e-83e9-03727bf1431a

glance-registry logs:

2014-11-24 20:14:11.095 26083 INFO eventlet.wsgi.server [-] 127.0.0.1 - - [24/Nov/2014 20:14:11] "PUT /images/3d42b60c-ad50-408e-83e9-03727bf1431a HTTP/1.1" 200 710 0.219377
2014-11-24 20:15:09.878 26083 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2014-11-24 20:15:09.878 26083 INFO eventlet.wsgi.server [-] 127.0.0.1 - - [24/Nov/2014 20:15:09] "PUT /images/3d42b60c-ad50-408e-83e9-03727bf1431a HTTP/1.1" 401 200 0.000852
2014-11-24 20:15:09.879 26083 WARNING keystonemiddleware.auth_token [-] Unable to find authentication token in headers
2014-11-24 20:15:09.880 26083 INFO eventlet.wsgi.server [-] 127.0.0.1 - - [24/Nov/2014 20:15:09] "PUT /images/3d42b60c-ad50-408e-83e9-03727bf1431a HTTP/1.1" 401 200 0.000332
2014-11-24 20:15:09.882 26083 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2014-11-24 20:15:09.883 26083 INFO eventlet.wsgi.server [-] 127.0.0.1 - - [24/Nov/2014 20:15:09] "PUT /images/3d42b60c-ad50-408e-83e9-03727bf1431a HTTP/1.1" 401 200 0.000792
2014-11-24 20:15:09.885 26083 WARNING keystonemiddleware.auth_token [-] Unable to find authentication token in headers
2014-11-24 20:15:09.885 26083 INFO eventlet.wsgi.server [-] 127.0.0.1 - - [24/Nov/2014 20:15:09] "PUT /images/3d42b60c-ad50-408e-83e9-03727bf1431a HTTP/1.1" 401 200 0.000461

The image is successfully created with default paste flavor (no authentification). Might be that it is a keystone-related issue?

[Dmitry Mescheryakov (dmitrymex)]

The issue seems to be important, raising to high

[OSCI Robot (oscirobot)]

RPM package python-swiftclient has been built for project openstack/python-swiftclient
Package version == 2.3.1, package release == fuel5.1.2.mira0.git.57d37c6.8817eff

Changeset: https://review.fuel-infra.org/1412
project: openstack/python-swiftclient
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Dmitry Mescheryakov
committer: Dmitry Mescheryakov
subject: Merge tag '2.3.1'
status: patchset-created

Files placed on repository:
python-swiftclient-2.3.1-fuel5.1.2.mira0.git.57d37c6.8817eff.noarch.rpm
python-swiftclient-doc-2.3.1-fuel5.1.2.mira0.git.57d37c6.8817eff.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable-1412/centos

[OSCI Robot (oscirobot)]

DEB package python-swiftclient has been built for project openstack/python-swiftclient
Package version == 2.3.1, package release == fuel5.1.2~mira0+git.57d37c6.8817eff

Changeset: https://review.fuel-infra.org/1412
project: openstack/python-swiftclient
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Dmitry Mescheryakov
committer: Dmitry Mescheryakov
subject: Merge tag '2.3.1'
status: patchset-created

Files placed on repository:
python-swiftclient_2.3.1-fuel5.1.2~mira0+git.57d37c6.8817eff_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable-1412/ubuntu

[OSCI Robot (oscirobot)]

RPM package python-swiftclient has been built for project openstack/python-swiftclient
Package version == 2.3.1, package release == fuel5.1.2.mira0

Changeset: https://review.fuel-infra.org/1412
project: openstack/python-swiftclient
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Dmitry Mescheryakov
committer: Dmitry Mescheryakov
subject: Merge tag '2.3.1'
status: change-merged

Files placed on repository:
python-swiftclient-2.3.1-fuel5.1.2.mira0.noarch.rpm
python-swiftclient-doc-2.3.1-fuel5.1.2.mira0.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1.2-stable/centos

[OSCI Robot (oscirobot)]

DEB package python-swiftclient has been built for project openstack/python-swiftclient
Package version == 2.3.1, package release == fuel5.1.2~mira0

Changeset: https://review.fuel-infra.org/1412
project: openstack/python-swiftclient
branch: openstack-ci/fuel-5.1.2/2014.1.1
author: Dmitry Mescheryakov
committer: Dmitry Mescheryakov
subject: Merge tag '2.3.1'
status: change-merged

Files placed on repository:
python-swiftclient_2.3.1-fuel5.1.2~mira0_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1.2-stable/ubuntu

[Alexander Tivelkov (ativelkov)]

This one is addressed by https://review.fuel-infra.org/#/c/1412/ - the issue was caused by the outdated swiftclient being used. So, no need to write release note for it.

[1overcosc (scott-h-shorkey)]

What was the fix for this? Been running into this exact error consistently for the past few days on Fuel 5.1, is there a way to fix without having to update to 6.0 and redeploy everything?

[Timur Nurlygayanov (tnurlygayanov)]

The root of this issue in the old version of swift python client, the commit with fix updated swift client up to 2.3.1 release.
If this issue reproduced on your environment with MOS 5.x you can just install new package with swift client on all your OpenStack controller nodes and after that restart all Glance API services:

DEB:
http://mirror.fuel-infra.org/fwm/files/5.1.2-staging/ubuntu/pool/main/python-swiftclient_2.3.1-fuel5.1.2~mira0_all.deb

RPM:
http://mirror.fuel-infra.org/fwm/files/5.1.2-staging/centos/os/x86_64/Packages/python-swiftclient-2.3.1-fuel5.1.2.mira0.noarch.rpm
http://mirror.fuel-infra.org/fwm/files/5.1.2-staging/centos/os/x86_64/Packages/python-swiftclient-doc-2.3.1-fuel5.1.2.mira0.noarch.rpm