Glance Store fails to authenticate to Swift with Keystone v3 API

[Jesse Pretorius]

In the Mitaka current RC for Glance, using HEAD of "stable/mitaka" as of 23.03.2016 (SHA ab0562550c8c568dcdc7da68afdcac5f58d20e69), glance_store fails to authenticate via the
Keystone v3 API to Swift.

Configuration and the error are available here: https://gist.github.com/odyssey4me/79a1e8d7dea35ddf818c

It appears that this may be a regression (this worked just fine in Liberty) introduced by https://github.com/openstack/glance_store/commit/1b782cee8552ec02f7303ee6f9ba9d1f2c180d07

[Kairat Kushaev (kkushaev)]

Could you please provide glance-api configuration?
It would be perfect to know if single-tenant or multi-tenant store is used.
Also it would be perfect to know if you have is_multiple_swift_store_accounts_enabled=True i.e. are you using specific config file (usually names glance-swift.conf) in your env.

[Stuart McLaren (stuart-mclaren)]

I don't see the glance_store config options in the sample logs.

Eg these kind of parameters

swift_store_config_file
swift_store_auth_version
swift_store_auth_address = <None>

They'd help with reproducing.

[Jesse Pretorius (jesse-pretorius)]

I've updated the gist with the requested information and the output of a pip freeze from the glance venv. Let me know if anything else will help.

[Matt Thompson (mattt416)]

Pretty sure it's https://github.com/openstack/glance_store/commit/1b782cee8552ec02f7303ee6f9ba9d1f2c180d07 -- we tried upgrading to different tags to isolate the error and noticed that 0.11.0 worked whilst 0.12.0 didn't. From there we walked through commits that touched the swift backend between these two tags and 142cf34c2f63900ac17f7e4c67a0f735deba84e4 worked fine but 1b782cee8552ec02f7303ee6f9ba9d1f2c180d07 caused an error to be raised when attempting to upload an image.

[Kairat Kushaev (kkushaev)]

We used sessions and auth_plugins when introducing this change, unfortunately, it turned out that it doesn't do any defaulting.
The fix is here:
https://review.openstack.org/#/c/297665/

[Kairat Kushaev (kkushaev)]

Here is piece of code that worked before:
https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/httpclient.py#L344
It seems that introducing sessions, this defaulting doesn't work.

[Kevin Carter (kevin-carter)]

++ this is a fairly large issue for our deployment team because we back glance images to swift almost 100% of the time. I'm currently testing the patch https://review.openstack.org/#/c/297665 and will follow up on the review however in my opinion this should be a release critical issue for glance as it's likely to break a significant portion of deployments for both greenfield and upgrades.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/296799
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=f635e090bd62c202e89d5fd87bd8198cfa68c534
Submitter: Jenkins
Branch: master

commit f635e090bd62c202e89d5fd87bd8198cfa68c534
Author: Jesse Pretorius <email address hidden>
Date: Wed Mar 23 21:04:07 2016 +0000

    Updates all repo SHAs to prepare for Mitaka release

    This patch includes updates of all SHA's and pip, wheel and setuptools
    pins.

    Due to Bug #1561947, the backend glance store has been changed from swift
    to file in the AIO gate. This has been done becuase the glance store project
    has an issue with session that causes it to be incompatible with swift. Being
    that the master integration gate is depending on this change to go it is
    needed to temporarily disable the swift image store in glance until this issue
    is fixed.

    The branch updater sciprt was modified to search for and update roles if they're
    found within the default or user provided role path. This change is resolving
    bash errors that happen when the script is run in mitaka+.

    Related-Bug: 1561947
    Change-Id: I2e09e3e0abb61f5e97e2af7b283f0aed6dadd853
    Signed-off-by: Kevin Carter <email address hidden>

[Kairat Kushaev (kkushaev)]

Guys, you can use swift store config file currently to get rid of this issue.
The problem you faced is just because keystone v3 is used but swift keystone creds were specified in glance-api.conf directly.
It is not recommended way to use swift driver. It is better to specify swift_store_config_file and default_swift_reference options where file contains swift user creds like:
[ref1]
auth_version = 3
project_domain_id = default
user_domain_id = default
auth_address = http://192.168.122.191:5000/v3
key = XXXXXXXXX
user = service:glance-swift
Please replace that ^ with your creds and everything should work fine.
This file allows to have several users working with swift so old images will be working if the new user will be created for swift-glance communication. Otherwise you will have big troubles when changing default user in swift driver.

[Kairat Kushaev (kkushaev)]

http://docs.openstack.org/developer/glance/configuring.html#configuring-the-swift-storage-backend - here is some doc. Hope that helps.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/299269

[Jesse Pretorius (jesse-pretorius)]

I'm trying out a patch to change the way things are configured but it's not working just yet - see https://gist.github.com/odyssey4me/7b916f5f67009344f15a3c7a55690275

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/299269
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_glance/commit/?id=b3f12253754035671335aa4ea31ce00534a6de0f
Submitter: Jenkins
Branch: master

commit b3f12253754035671335aa4ea31ce00534a6de0f
Author: Jesse Pretorius <email address hidden>
Date: Wed Mar 30 11:12:07 2016 +0100

    Adjust Glance/Swift backend configuration

    As per [1] The way of configuring a Swift back-end for Glance
    has changed. This change has been done in such a way that when
    using Keystone v3 for authentication, the current configuraiton
    method breaks.

    This patch adjusts the way the configuration is done in order
    to achieve the same result as before - a single working Swift
    backend for Glance if the deployment opts to use Swift as a
    backend.

    [1] http://docs.openstack.org/developer/glance/configuring.html#configuring-swift-configuration-file

    Change-Id: I02b69cf5985e8788513db58f7e8015f8135b9d58
    Closes-Bug: #1561947

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/299637

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/299637
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=c2d49cbff06f348acde42ad583a1401767e52806
Submitter: Jenkins
Branch: master

commit c2d49cbff06f348acde42ad583a1401767e52806
Author: Jesse Pretorius <email address hidden>
Date: Wed Mar 30 22:45:01 2016 +0100

    Set default Glance store for the AIO back to Swift

    In I02b69cf5985e8788513db58f7e8015f8135b9d58 the Glance Store
    for Swift was appropriately configured to work with the Keystone
    v3 API and in the new expected way for the Glance Swift Store to
    be configured for Mitaka.

    This patch returns the default AIO to use Swift as the backing
    store for Glance.

    Change-Id: Iec233f990bd032fb29f633d74137c29bc89f63f5
    Closes-Bug: #1561947

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible 13.0.0 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/openstack-ansible 13.0.0 release.

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/glance_store 0.14.0 release.