Glance: can't upload image by link from slow source

[Victor Ryzhenkin]

ISO: {"build_id": "2014-12-09_22-41-06", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "49", "auth_required": true, "api": "1.0", "nailgun_sha": "22bd43b89a17843f9199f92d61fc86cb0f8772f1", "production": "docker", "fuelmain_sha": "3aab16667f47dd8384904e27f70f7a87ba15f4ee", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-09_22-41-06", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "49", "api": "1.0", "nailgun_sha": "22bd43b89a17843f9199f92d61fc86cb0f8772f1", "production": "docker", "fuelmain_sha": "3aab16667f47dd8384904e27f70f7a87ba15f4ee", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "2c99931072d951301d395ebd5bf45c8d401301bb"}}}, "fuellib_sha": "2c99931072d951301d395ebd5bf45c8d401301bb"}

STEPS TO REPRODUCE:
1. Deploy OS with Neutron GRE, install Murano, use ha_compact mode with Swift on CentOS (1 Controller 1 Compute )
2. Navigate to Horizon dashboard-Project-Images
3. Click Create Image.
4. Specify image url (Example: http://murano-files.mirantis.com/ubuntu_14_04-murano-agent_stable_juno.qcow2)
5. Create Image

Actual Result:
Image has status 'Killed'
Glance logs have no tracebacks
In /var/log/messages we can see warning about Swift:
swift-proxy-server: Client disconnected without sending enough data

[Sergey Lukjanov (slukjanov)]

Actually, it could be related to very slow download speed from the Rackspace CDN (murano-files backend) to some locations (for example Russia). I've uploaded ~20 images today to glance using links from my own server with a very fast connection. It was done on #41 ISO.

[Timur Nurlygayanov (tnurlygayanov)]

Sergey, probably yes, it can be the reason but even in this case it is a bug which we should investigate and fix.

[Alexander Tivelkov (ativelkov)]

Seems like the download speed may indeed be the issue.
I was even able to reproduce the issue on regular devstack (with Swift): I used the same murano-files store backend - and in 2 of 5 runs I was getting errors in Swift.
However, devstack and MOs had slightly different error messages in logs, so these may be different issues actually.

What I may suggest as a workaround is to try decreasing the size of the swift chunk (swift_store_large_object_chunk_size parameter in glance-api.conf) while making sure that the size of the file is larger then the one specified by large_object indicator (swift_store_large_object_size in the same config, 5 Gb by default). Doing that will split the files into multiple smaller parts in swift and so will decrease the time needed to put a single part to swift.

This may or may not help - it depends on where the problem actually is.
Will continue investigating this tomorrow.

[Mike Fedosin (mfedosin)]

Hello!
The current status of the bug is there is a minute timeout somewhere in MOS, which breaks the connection if 16 mb chunk has not been downloaded in that time.
16 mb is hardcoded parameter in glance store https://github.com/openstack/glance_store/blob/master/glance_store/driver.py#L34
Alexander has a conversation with Zhi Yan and it was decided to lower it to 4 mb - this solution works fine. Patch in master will be done next week.
But also it was good to find the timeout cause. Vladimir Kuklin said that he knows where it may be and he will check his guesses next week when he returns.

[Alexander Tivelkov (ativelkov)]

Known Issue:

If the image is uploaded into the Glance from the remote http source (i.e. the image is created with "image-create" Glance CLI command specified with "--copy-from" parameter containing some http location, or using the equivalent set of API calls, for example when creating image by copying it from the URL in Horizon) AND the speed of downloading from that location is low (i.e. there exists a possibility that at some moment a 16 MB chunk of data will not be fetched within a minute), then the Image will not be uploaded (in several minuted of being in "Saving" state it will be moved to "killed" state, thus disappearing from list of active images).

This may be happen often for image storages on the public hosting providers, such as murano-files.mirantis.com or sahara-files.mirantis.com - these storages tend to throttle connection speeds when the load is high, which sometimes may reduce the speed below the aforementioned threshold.

Workaround:
In the case of slow connection to the external image storage, it is better to manually download the image from the external storage to the local filesystem of the controller (e.g. with wget command) and then put it to the Glance directly (execute an "image-create" Glance CLI command with "--file" parameter pointing to the downloaded file, or execute an equivalent set of API calls)

[OSCI Robot (oscirobot)]

package glance_store has been built for project openstack/glance_store
Package version == 0.1.10, package release == fuel6.1.mira3.git.004e515.95e9349

Changeset: https://review.fuel-infra.org/4393
project: openstack/glance_store
branch: openstack-ci/fuel-6.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Fix chunk size
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /centos-fuel-6.1-stable-4393/

[OSCI Robot (oscirobot)]

package glance_store has been built for project openstack/glance_store
Package version == 0.1.10, package release == fuel6.1~mira3+git.004e515.95e9349

Changeset: https://review.fuel-infra.org/4393
project: openstack/glance_store
branch: openstack-ci/fuel-6.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Fix chunk size
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /ubuntu-fuel-6.1-stable-4393/

[OSCI Robot (oscirobot)]

DEB package glance_store has been built for project openstack/glance_store
Package version == 0.1.10, package release == fuel6.1~mira1+git.004e515.95e9349

Changeset: https://review.fuel-infra.org/4393
project: openstack/glance_store
branch: openstack-ci/fuel-6.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Fix chunk size
status: patchset-created

Files placed on repository:
python-glance-store-doc_0.1.10-fuel6.1~mira1+git.004e515.95e9349_all.deb
python-glance-store_0.1.10-fuel6.1~mira1+git.004e515.95e9349_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://obs-1.mirantis.com:82/trusty-fuel-6.1-stable-4393/ubuntu

[OSCI Robot (oscirobot)]

RPM package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.1.mira2.git.c76b841.15269c0

Changeset: https://review.fuel-infra.org/4394
project: openstack/glance_store
branch: openstack-ci/fuel-6.0.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Fix chunk size
status: patchset-created

Files placed on repository:
python-glance-store-0.1.7-fuel6.0.1.mira2.git.c76b841.15269c0.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-4394/centos

[OSCI Robot (oscirobot)]

DEB package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.1~mira2+git.c76b841.15269c0

Changeset: https://review.fuel-infra.org/4394
project: openstack/glance_store
branch: openstack-ci/fuel-6.0.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Fix chunk size
status: patchset-created

Files placed on repository:
python-glance-store-doc_0.1.7-fuel6.0.1~mira2+git.c76b841.15269c0_all.deb
python-glance-store_0.1.7-fuel6.0.1~mira2+git.c76b841.15269c0_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-4394/ubuntu

[OSCI Robot (oscirobot)]

RPM package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.1.mira2.git.63662dd.15269c0

Changeset: https://review.fuel-infra.org/4394
project: openstack/glance_store
branch: openstack-ci/fuel-6.0.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Fix chunk size
status: patchset-created

Files placed on repository:
python-glance-store-0.1.7-fuel6.0.1.mira2.git.63662dd.15269c0.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable-4394/centos

[OSCI Robot (oscirobot)]

DEB package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.1~mira2+git.63662dd.15269c0

Changeset: https://review.fuel-infra.org/4394
project: openstack/glance_store
branch: openstack-ci/fuel-6.0.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Fix chunk size
status: patchset-created

Files placed on repository:
python-glance-store-doc_0.1.7-fuel6.0.1~mira2+git.63662dd.15269c0_all.deb
python-glance-store_0.1.7-fuel6.0.1~mira2+git.63662dd.15269c0_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable-4394/ubuntu

[OSCI Robot (oscirobot)]

package glance_store has been built for project openstack/glance_store
Package version == 0.1.10, package release == fuel6.1.mira3.git.3a16210.95e9349

Changeset: https://review.fuel-infra.org/4393
project: openstack/glance_store
branch: openstack-ci/fuel-6.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Fix chunk size
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /centos-fuel-6.1-stable-4393/

[OSCI Robot (oscirobot)]

DEB package glance_store has been built for project openstack/glance_store
Package version == 0.1.10, package release == fuel6.1~mira1+git.3a16210.95e9349

Changeset: https://review.fuel-infra.org/4393
project: openstack/glance_store
branch: openstack-ci/fuel-6.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Fix chunk size
status: patchset-created

Files placed on repository:
python-glance-store-doc_0.1.10-fuel6.1~mira1+git.3a16210.95e9349_all.deb
python-glance-store_0.1.10-fuel6.1~mira1+git.3a16210.95e9349_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://obs-1.mirantis.com:82/trusty-fuel-6.1-stable-4393/ubuntu

[OSCI Robot (oscirobot)]

package glance_store has been built for project openstack/glance_store
Package version == 0.1.10, package release == fuel6.1.mira3

Changeset: https://review.fuel-infra.org/4393
project: openstack/glance_store
branch: openstack-ci/fuel-6.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Fix chunk size
status: change-merged

Files placed on repository:

Changeset merged. Package placed on primary repository
 repository URL: /centos-fuel-6.1-stable/

[OSCI Robot (oscirobot)]

package glance_store has been built for project openstack/glance_store
Package version == 0.1.10, package release == fuel6.1~mira3

Changeset: https://review.fuel-infra.org/4393
project: openstack/glance_store
branch: openstack-ci/fuel-6.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Fix chunk size
status: change-merged

Files placed on repository:

Changeset merged. Package placed on primary repository
 repository URL: /ubuntu-fuel-6.1-stable/

[OSCI Robot (oscirobot)]

DEB package glance_store has been built for project openstack/glance_store
Package version == 0.1.10, package release == fuel6.1~mira1

Changeset: https://review.fuel-infra.org/4393
project: openstack/glance_store
branch: openstack-ci/fuel-6.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Fix chunk size
status: change-merged

Files placed on repository:
python-glance-store-doc_0.1.10-fuel6.1~mira1_all.deb
python-glance-store_0.1.10-fuel6.1~mira1_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://obs-1.mirantis.com:82/trusty-fuel-6.1-stable/ubuntu

[OSCI Robot (oscirobot)]

package glance_store has been built for project openstack/glance_store
Package version == 0.1.10, package release == fuel6.1~mira4+git.3a16210.95e9349

Changeset: https://review.fuel-infra.org/4393
project: openstack/glance_store
branch: openstack-ci/fuel-6.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Fix chunk size
status: patchset-created

Files placed on repository:

NOTE: Changeset is not merged, created temporary package repository.
 repository URL: /ubuntu-fuel-6.1-stable-4393/

[OSCI Robot (oscirobot)]

RPM package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.1.mira2

Changeset: https://review.fuel-infra.org/4394
project: openstack/glance_store
branch: openstack-ci/fuel-6.0.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Fix chunk size
status: change-merged

Files placed on repository:
python-glance-store-0.1.7-fuel6.0.1.mira2.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable/centos

[OSCI Robot (oscirobot)]

DEB package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.1~mira2

Changeset: https://review.fuel-infra.org/4394
project: openstack/glance_store
branch: openstack-ci/fuel-6.0.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Fix chunk size
status: change-merged

Files placed on repository:
python-glance-store-doc_0.1.7-fuel6.0.1~mira2_all.deb
python-glance-store_0.1.7-fuel6.0.1~mira2_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable/ubuntu

[OSCI Robot (oscirobot)]

RPM package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.mira2.git.bec28a0.15269c0

Changeset: https://review.fuel-infra.org/4760
project: openstack/glance_store
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Alexandr Nevenchannyy
committer: Alexandr Nevenchannyy
subject: Fix chunk size
status: patchset-created

Files placed on repository:
python-glance-store-0.1.7-fuel6.0.mira2.git.bec28a0.15269c0.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable-4760/centos

[OSCI Robot (oscirobot)]

DEB package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0~mira2+git.bec28a0.15269c0

Changeset: https://review.fuel-infra.org/4760
project: openstack/glance_store
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Alexandr Nevenchannyy
committer: Alexandr Nevenchannyy
subject: Fix chunk size
status: patchset-created

Files placed on repository:
python-glance-store-doc_0.1.7-fuel6.0~mira2+git.bec28a0.15269c0_all.deb
python-glance-store_0.1.7-fuel6.0~mira2+git.bec28a0.15269c0_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable-4760/ubuntu

[OSCI Robot (oscirobot)]

RPM package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.mira2

Changeset: https://review.fuel-infra.org/4760
project: openstack/glance_store
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Alexandr Nevenchannyy
committer: Alexandr Nevenchannyy
subject: Fix chunk size
status: change-merged

Files placed on repository:
python-glance-store-0.1.7-fuel6.0.mira2.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable/centos

[OSCI Robot (oscirobot)]

DEB package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0~mira2

Changeset: https://review.fuel-infra.org/4760
project: openstack/glance_store
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Alexandr Nevenchannyy
committer: Alexandr Nevenchannyy
subject: Fix chunk size
status: change-merged

Files placed on repository:
python-glance-store-doc_0.1.7-fuel6.0~mira2_all.deb
python-glance-store_0.1.7-fuel6.0~mira2_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable/ubuntu

[OSCI Robot (oscirobot)]

RPM package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.1.mira3

Changeset: https://review.fuel-infra.org/5182
project: openstack/glance_store
branch: openstack-ci/fuel-6.0.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Check VMware session before uploading image
status: change-merged

Files placed on repository:
python-glance-store-0.1.7-fuel6.0.1.mira3.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0.1-stable/centos

[OSCI Robot (oscirobot)]

DEB package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.1~mira3

Changeset: https://review.fuel-infra.org/5182
project: openstack/glance_store
branch: openstack-ci/fuel-6.0.1/2014.2
author: Mikhail Fedosin
committer: Mikhail Fedosin
subject: Check VMware session before uploading image
status: change-merged

Files placed on repository:
python-glance-store-doc_0.1.7-fuel6.0.1~mira3_all.deb
python-glance-store_0.1.7-fuel6.0.1~mira3_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0.1-stable/ubuntu

[OSCI Robot (oscirobot)]

RPM package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0.mira3

Changeset: https://review.fuel-infra.org/5184
project: openstack/glance_store
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Mikhail Fedosin
committer: Alex Khivin
subject: Check VMware session before uploading image
status: change-merged

Files placed on repository:
python-glance-store-0.1.7-fuel6.0.mira3.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-updates-stable/centos

[OSCI Robot (oscirobot)]

DEB package glance_store has been built for project openstack/glance_store
Package version == 0.1.7, package release == fuel6.0~mira3

Changeset: https://review.fuel-infra.org/5184
project: openstack/glance_store
branch: openstack-ci/fuel-6.0-updates/2014.2
author: Mikhail Fedosin
committer: Alex Khivin
subject: Check VMware session before uploading image
status: change-merged

Files placed on repository:
python-glance-store-doc_0.1.7-fuel6.0~mira3_all.deb
python-glance-store_0.1.7-fuel6.0~mira3_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-updates-stable/ubuntu

[Kyrylo Romanenko (kromanenko)]

Verified on
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "471"
  build_id: "2015-05-26_20-59-56"

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Mike Fedosin <email address hidden>
Review: https://review.fuel-infra.org/8115

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Mike Fedosin <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/8115