Cannot refresh user token when uploading image to swift backend

[Kairat Kushaev]

Verification on iso #372 (9.0-mos.all).
Shotgun report: http://paste.openstack.org/show/497853
Use glance with Swift backend only.
This bug was detected during https://bugs.launchpad.net/mos/+bug/1576035 check.

Steps:

[on all controllers]
1. Change token expiration from 3600 to 60.
2. Restart apache2,glance-api,glance-registry.

[on slave by cli]
1. Create new image file by command: fallocate -l 4G temp.img
2. Upload into glance: glance image-create --file temp.img --progress --container-format bare --disk-format qcow2

Excepted result:
 Successfully uploaded.

Actual result:
 Upload failed. API logs: https://paste.mirantis.net/show/2287/

The root cause of the following: during uploading of big images Glance splits images to chunks.
Each chunk upload requires token. So when token expires uploading fails. Glance store has a mechanism to prevent such error but we need to use keystone v3 to create trust for swift driver in glance_store.

So in order to use re-auth in glance_store we need to do the following:
1. add another reference to
/etc/glance/glance-swift.conf file. Example:
[ref2]
user = services:glance
key = <some_key>
user_domain_id = default
project_domain_id = default
auth_version = 3
auth_address = http://<keystone_url>:5000/v3

Please note that we are using v3 for authentication.
Also we can replace old reference (ref1) with the reference above if there are no active images in database.

2. Change default_swift_reference to ref2 in glance-api.conf.

In that case glance can upload any images to swift without auth errors.

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Denis Egorenko (degorenko)]

Fix: https://review.openstack.org/#/c/319983/

[Denis Egorenko (degorenko)]

Mitaka: https://review.openstack.org/320845

[Ilya Menkov (imenkov)]

Fix works fine. It was verified on 9.0 mos iso #424

[Denis Meltsaykin (dmeltsaykin)]

This was not fixed properly, see Bug #1731054.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0/mitaka
Change author: Denis V. Meltsaykin <email address hidden>
Review: https://review.fuel-infra.org/37150

[Dmitry Sutyagin (dsutyagin)]

I can confirm this patch resolves the issue. I've filed a separate bug https://bugs.launchpad.net/mos/+bug/1732033 against the other related issue which happens if token expires during _delete_stale_chunks

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/37150
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 2982fadc0a9fa7487656a87548bae7540becbd20
Author: Denis V. Meltsaykin <email address hidden>
Date: Sat Nov 11 20:43:15 2017

Use cached auth_ref instead of gettin a new one each time

The commit fixes an issue in renewing connections to Swift in case
if a given token is about to expire soon. get_auth_ref() returned
a new auth_ref each time it was called so the will_expire_soon()
call was useless since it always returned False. Now the cached
auth_ref property is used instead.

Change-Id: I68b69e95888d818daea26f406c806c25f66d8d7e
Closes-Bug: #1584070

[Dmitry (dtsapikov)]

Verified on 9.2+mu4