Mirantis OpenStack

New image is not uploaded if time to upload of image greater than time to life of keystone token.

Series 7.0.x
Bug #1456573

Bug #1456573 reported by Vadim Rovachev on 2015-05-19

This bug affects 3 people

	Status	Importance	Assigned to	Milestone
Mirantis OpenStack	Fix Released	High	Mike Fedosin	Mirantis OpenStack 8.0
7.0.x	In Progress	High	Mike Fedosin	Mirantis OpenStack 7.0-updates
8.0.x	Fix Released	High	Mike Fedosin	Mirantis OpenStack 8.0

Bug Description

Steps to reproduce:
1. Go to controller node.
2. Create large file:
# fallocate -l 100G image.img
3. Try to upload this file to glance:
glance image-create --name 'test' --disk-format raw --container-format bare --progress --file image.img

[=============================>] 100%
<html>
<head>
  <title>500 Internal Server Error</title>
</head>
<body>
  <h1>500 Internal Server Error</h1>
  Failed to upload image 5a343db1-f65f-4558-a6ae-069400663d85<br /><br />

</body>
</html> (HTTP 500)
You have new mail in /var/spool/mail/root

TRACE in /var/log/glance/api.log:
http://paste.openstack.org/show/228091/

Tags:

Vadim Rovachev (vrovachev) on 2015-05-19

tags:

added: glance

Nastya Urlapova (aurlapova) on 2015-05-19

Changed in fuel:
milestone:	none → 6.1
assignee:	nobody → MOS Glance (mos-glance)
status:	New → Confirmed

Revision history for this message

Vadim Rovachev (vrovachev) wrote on 2015-05-19:

diagnostic snapshot:
https://drive.google.com/a/mirantis.com/file/d/0BxAwYghQ8JfbM3lUM3dYYkZtRXM/view?usp=sharing

Revision history for this message

Inessa Vasilevskaya (ivasilevskaya) wrote on 2015-05-19:

Sadly, but its a known issue. Some cosmetic surgery is possible at the moment by applying this patch: https://review.fuel-infra.org/#/c/6303/ (it turns 500 into 400 BadRequest), but, alas, it won't help uploading the image.

A different mechanism for large file upload has to be implemented, possibly based on keystone trusts (https://bugs.launchpad.net/fuel/6.0.x/+bug/1441156 - see Mike Fedosin's last comment on the subject).

ruhe (ruhe) on 2015-05-19

Changed in fuel:
importance:	Undecided → Medium

Inessa Vasilevskaya (ivasilevskaya) on 2015-05-19

tags:

added: release-notes

ruhe (ruhe) on 2015-05-19

Changed in fuel:
importance:	Medium → High
status:	Confirmed → Won't Fix
affects:	fuel → mos
Changed in mos:
milestone:	6.1 → none
milestone:	none → 6.1

Olena Logvinova (ologvinova) on 2015-05-27

tags:

added: release-notes-done
removed: release-notes

Revision history for this message

Mike Fedosin (mfedosin) wrote on 2015-07-17:

Upstream fix on review: https://review.openstack.org/#/c/199049/

Revision history for this message

Mike Fedosin (mfedosin) wrote on 2015-08-19:

We have a fix for this bug for ceph - https://review.openstack.org/#/c/213757/. Commit for Swift is under review in community (link above) - when it's merged I will backport it to MOS 7.0.

Currently I set this bug a duplicate of https://bugs.launchpad.net/fuel/+bug/1485639. Swift related bug for MOS 8.0 will be added soon.

Revision history for this message

Fuel Devops McRobotson (fuel-devops-robot) wrote on 2016-01-05: Fix merged to openstack/glance (openstack-ci/fuel-8.0/liberty)

Reviewed: https://review.fuel-infra.org/14329
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: fedcb444d11eacbfa51521f2f5479c62eb93e093
Author: Mike Fedosin <email address hidden>
Date: Mon Dec 28 14:05:06 2015

Implement trust support for Glance

This commit adds new functionality in Glance,
called 'trusts', that are used to prevent token
expiration during longtime upload.

The algorighm is the following:
1. If galnce uses registry server to communicate with db, create a trust
2. Upload an image
3. Try to update the image status to 'active' in registry
4. If trust has been created succcessfully and 401 occured during
updating the image status then renew the token and try to update
the image record in registry again

If keystone server doesn't support trust functionality
then glance automatically returns to the basic workflow
with user token.

Co-Authored-By: Mike Fedosin <email address hidden>
Co-Authored-By: Kairat Kushaev <email address hidden>

Closes-bug: #1456573

Change-Id: Ia3b82782b14f5dfc93457620633c1039c38fc366

Sergey Novikov (snovikov) on 2016-01-19

tags:

added: on-verification

Revision history for this message

Mike Fedosin (mfedosin) wrote on 2016-01-19:

QA testing on devstack:
0. Preparation: Set glance backend to RBD (Ceph), set 'expiration' option in /etc/keystone/keystone.conf to '40', set 'token_cache_time' in /etc/glance/glance-registry.conf to '-1', set 'data_api' in /etc/glance/glance-api.conf to 'glance.db.registry.api'.
1. Try to upload big image (when upload takes at least 1 minute). (glance image-create --name 'test' --disk-format raw --container-format bare --progress --file image.img) Make sure that upload was successfully finished.
2. Perform step 1 again with glance v1 api (glance --os-image-api-version 1 image-create --name 'test' --disk-format raw --container-format bare --progress --file image.img) Make sure that upload was successfully finished.

Revision history for this message

Mike Fedosin (mfedosin) wrote on 2016-01-19:

This fix doesn't work with Swift backend (RadosGW works good), because there is another issue in Swift driver. Code that fixes the issue is on review in upstream. When it's merged it will be backported in MOS. https://review.openstack.org/#/c/251851/11

Revision history for this message

Sergey Novikov (snovikov) wrote on 2016-01-20:

Verified on VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "457"
  build_id: "457"
  fuel-nailgun_sha: "db88977ea5ec8a034379a953f72627c3819dfd24"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "0c1b17537a35d1b56f382fcc48197d0ee0c02d90"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "42f1059465ccd7e6b501d9bdca0076fb89a64339"
  fuel-ostf_sha: "f66b1208bb9214513a042d52226583473f6117ea"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "6c1888a48e2c3244d7fdf6409c22874aac8a7495"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "4a028ee035e70d1e0ccf51c5a5588c9b8a6f78c9"

STR:

1. Deploy cluster with Ceph backeng for glance.
2. Set 'expiration' option in /etc/keystone/keystone.conf to '30'
3. Set 'token_cache_time' in /etc/glance/glance-registry.conf to '-1'
4. Set 'data_api' in /etc/glance/glance-api.conf to 'glance.db.registry.api'.
5. Restart glance services and apache
6. Go to controller node and create large file:
# fallocate -l 10G image.img
7. Try to upload this file to glance:
#glance image-create --name 'test' --disk-format raw --container-format bare --progress --file image.img
2. Perform step 7 again with glance v1 api
# glance --os-image-api-version 1 image-create --name 'test' --disk-format raw --container-format bare --progress --file image.img

Revision history for this message

Mike Fedosin (mfedosin) wrote on 2016-01-21:

Sergey, thank you very much!

Sergey Novikov (snovikov) on 2016-02-02

tags:

removed: glance on-verification

Revision history for this message

Laukik (laukikd) wrote on 2016-02-10:

#10

All,
Is there an approximate time frame that this fix will be released in 7.0 updates?
Regards
Laukik

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.