with keystone auth, image objects left behind in swift post-deletion

[Eoghan Glynn]

When the swift backend store is in use, with the keystone auth strategy enabled, and delayed_delete configured false, image deletion in glance leads to the corresponding swift object being leaked.

This results from the attempted object deletion in glance/store/swift.py failing silently with:

  Auth GET failed: http://keystone_host:5000/tokens 404 Not Found

The root cause is that the auth url associated with the swift connection used for deletion is missing a trailing forward slash.

Ubuntu SRU Justification
-------------------------
[Impact]
When Glance is configured to use Swift as a backend store and when Keystone authentication is enabled, requests are sent to the Keystone Auth URL without a trailing slash. This results in 404s from the Keystone API server, causing authentication requests in Glance's swift client middleware to fail. The original report cites situations where image deletion silently fails, though others report this completely breaks Swift+Glance integration.

[Development Fix]
This issue has been fixed since the release of Essex and has been released in the first Openstack Folsom milestone (f1) https://review.openstack.org/6480

[Stable Fix]
The fix has been backported upstream to the stable/essex branch. Cherry picking the commit and applying it to the Essex/12.04 packaging should be enough to fix the issue. http://bazaar.launchpad.net/~gandelman-a/ubuntu/precise/glance/sru_979745/revision/52

[Test Case]
Configure glance to use Swift as a backing store with Keystone authentication. Uploading, deleting and getting images should result in 404s from Keystone for requests to the configured auth URL.

[Regression Potential]
Minimal. The patch simply formats the keystone URL appropriately within Glance's swift client code. This code path is not hit unless the glance server is configured to use a Keystone-authenticated Swift backend.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/6480

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/6480
Committed: http://github.com/openstack/glance/commit/fa82103f1cdb9bb26473df3a4ee9ddc077c0541e
Submitter: Jenkins
Branch: master

commit fa82103f1cdb9bb26473df3a4ee9ddc077c0541e
Author: Eoghan Glynn <email address hidden>
Date: Thu Apr 12 11:27:18 2012 +0100

    Ensure swift auth URL includes trailing slash

    Fixes bug 979745

    Image objects in swift were previously leaked post-deletion
    due to a silent auth failure caused by the absense of the
    trailing forward slash on the swift connection auth URL.

    Change-Id: I9c73a2f75a6466e73801ababdd81db77701ccb20

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/6490

[Sam Morrison (sorrison)]

Using the official precise packages I also needed this patch to get nova to download images from swift

[Eoghan Glynn (eglynn)]

Hi Sam,

This fix is already back-ported to the stable/essex branch so will be available in the first post-essex bug-fix release.

The stable-maint team are getting the ball rolling on that release soon, so the fix should be available in officially released packages before long.

Cheers,
Eoghan

[Martin Gerhard Loschwitz (martin-loschwitz)]

It's still not in, and the bug keeps the Glance+Swift combo on Ubuntu 12.04 LTS from running. Can somebody please take care of this issue?

[Martin Gerhard Loschwitz (martin-loschwitz)]

Hi there,

the problem still exists; "before long" was 20 days ago and using Glance+Swift is still impossible on Ubuntu 12.04 LTS without this patch. Can somebody of the Ubuntu team please get hold of the issue and at least release appropriately patched Glance packages?

Best regards
Martin

[Martin Gerhard Loschwitz (martin-loschwitz)]

Added the Ubuntu glance package to the list of affected pieces of software to make Ubuntu packagers aware of this.

[Eoghan Glynn (eglynn)]

Hi Martin,

This fix has already been backported into Fedora/EPEL essex packages.

Can you request that the Ubuntu package maintainers do the same?

The first stable/essex release has not occurred as yet - the stable-maint team plan to do a release, but have not set a hard date as yet for this.

Cheers,
Eoghan

[Martin Gerhard Loschwitz (martin-loschwitz)]

On it, doing my best :)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Adam Gandelman (gandelman-a)]

Fixed upstream and released into Quantal in glance 2012.2~f2~20120524.1541-0ubuntu1

[Adam Gandelman (gandelman-a)]

glance 2012.1-0ubuntu2.1 awaiting approval for upload into -proposed

[Brian Murray (brian-murray)]

Hello Eoghan, or anyone else affected,

Accepted glance into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Martin Gerhard Loschwitz (martin-loschwitz)]

I have these packages installed:

root@warthog:~# dpkg --list | grep glance
ii glance 2012.1-0ubuntu2.1 OpenStack Image Registry and Delivery Service - Daemons
ii glance-api 2012.1-0ubuntu2.1 OpenStack Image Registry and Delivery Service - API
ii glance-client 2012.1-0ubuntu2.1 OpenStack Image Registry and Delivery Service - Registry
ii glance-common 2012.1-0ubuntu2.1 OpenStack Image Registry and Delivery Service - Common
ii glance-registry 2012.1-0ubuntu2.1 OpenStack Image Registry and Delivery Service - Registry
ii python-glance 2012.1-0ubuntu2.1 OpenStack Image Registry and Delivery Service - Python library

I can confirm that the problem does not appear anymore with these packages. Thanks for the great work, guys!

[Launchpad Janitor (janitor)]

This bug was fixed in the package glance - 2012.1-0ubuntu2.1

---------------
glance (2012.1-0ubuntu2.1) precise-proposed; urgency=low

  * debian/patches/swift_auth_url_trailing_slash.patch: Ensure swift auth URL
    includes trailing slash (LP: #979745). Backported from stable/essex, can
    be dropped with first upstream stable update.
 -- Adam Gandelman <email address hidden> Wed, 23 May 2012 16:19:50 -0700

[Brian Murray (brian-murray)]

Hello Eoghan, or anyone else affected,

Accepted glance into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Steve Langasek (vorlon)]

This bug was referenced in the changelog of glance 2012.1+stable~20120608-5462295-0ubuntu2.2 in error, having already been fixed in SRU. Marking verification-done.