with keystone auth, image objects left behind in swift post-deletion

Bug #979745 reported by Eoghan Glynn on 2012-04-12
42
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Glance
Medium
Eoghan Glynn
Essex
Undecided
Unassigned
glance (Ubuntu)
High
Unassigned
Precise
High
Unassigned
Quantal
High
Unassigned

Bug Description

When the swift backend store is in use, with the keystone auth strategy enabled, and delayed_delete configured false, image deletion in glance leads to the corresponding swift object being leaked.

This results from the attempted object deletion in glance/store/swift.py failing silently with:

  Auth GET failed: http://keystone_host:5000/tokens 404 Not Found

The root cause is that the auth url associated with the swift connection used for deletion is missing a trailing forward slash.

Ubuntu SRU Justification
-------------------------
[Impact]
When Glance is configured to use Swift as a backend store and when Keystone authentication is enabled, requests are sent to the Keystone Auth URL without a trailing slash. This results in 404s from the Keystone API server, causing authentication requests in Glance's swift client middleware to fail. The original report cites situations where image deletion silently fails, though others report this completely breaks Swift+Glance integration.

[Development Fix]
This issue has been fixed since the release of Essex and has been released in the first Openstack Folsom milestone (f1) https://review.openstack.org/6480

[Stable Fix]
The fix has been backported upstream to the stable/essex branch. Cherry picking the commit and applying it to the Essex/12.04 packaging should be enough to fix the issue. http://bazaar.launchpad.net/~gandelman-a/ubuntu/precise/glance/sru_979745/revision/52

[Test Case]
Configure glance to use Swift as a backing store with Keystone authentication. Uploading, deleting and getting images should result in 404s from Keystone for requests to the configured auth URL.

[Regression Potential]
Minimal. The patch simply formats the keystone URL appropriately within Glance's swift client code. This code path is not hit unless the glance server is configured to use a Keystone-authenticated Swift backend.

Eoghan Glynn (eglynn) on 2012-04-12
Changed in glance:
status: New → Confirmed
importance: Undecided → Medium
assignee: nobody → Eoghan Glynn (eglynn)
milestone: none → folsom-1
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/6480
Committed: http://github.com/openstack/glance/commit/fa82103f1cdb9bb26473df3a4ee9ddc077c0541e
Submitter: Jenkins
Branch: master

commit fa82103f1cdb9bb26473df3a4ee9ddc077c0541e
Author: Eoghan Glynn <email address hidden>
Date: Thu Apr 12 11:27:18 2012 +0100

    Ensure swift auth URL includes trailing slash

    Fixes bug 979745

    Image objects in swift were previously leaked post-deletion
    due to a silent auth failure caused by the absense of the
    trailing forward slash on the swift connection auth URL.

    Change-Id: I9c73a2f75a6466e73801ababdd81db77701ccb20

Changed in glance:
status: In Progress → Fix Committed
Sam Morrison (sorrison) on 2012-04-30
tags: added: essex-backport
Sam Morrison (sorrison) wrote :

Using the official precise packages I also needed this patch to get nova to download images from swift

Eoghan Glynn (eglynn) wrote :

Hi Sam,

This fix is already back-ported to the stable/essex branch so will be available in the first post-essex bug-fix release.

The stable-maint team are getting the ball rolling on that release soon, so the fix should be available in officially released packages before long.

Cheers,
Eoghan

It's still not in, and the bug keeps the Glance+Swift combo on Ubuntu 12.04 LTS from running. Can somebody please take care of this issue?

Hi there,

the problem still exists; "before long" was 20 days ago and using Glance+Swift is still impossible on Ubuntu 12.04 LTS without this patch. Can somebody of the Ubuntu team please get hold of the issue and at least release appropriately patched Glance packages?

Best regards
Martin

Added the Ubuntu glance package to the list of affected pieces of software to make Ubuntu packagers aware of this.

Thierry Carrez (ttx) on 2012-05-23
Changed in glance:
status: Fix Committed → Fix Released
Eoghan Glynn (eglynn) wrote :

Hi Martin,

This fix has already been backported into Fedora/EPEL essex packages.

Can you request that the Ubuntu package maintainers do the same?

The first stable/essex release has not occurred as yet - the stable-maint team plan to do a release, but have not set a hard date as yet for this.

Cheers,
Eoghan

On it, doing my best :)

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in glance (Ubuntu):
status: New → Confirmed
Chuck Short (zulcss) on 2012-05-23
Changed in glance (Ubuntu):
milestone: none → precise-updates
Changed in glance (Ubuntu Precise):
status: New → Confirmed
James Page (james-page) on 2012-05-24
Changed in glance (Ubuntu Quantal):
milestone: precise-updates → quantal-alpha-1
Changed in glance (Ubuntu Precise):
milestone: none → precise-updates
importance: Undecided → High
Changed in glance (Ubuntu Quantal):
importance: Undecided → High
Changed in glance (Ubuntu Precise):
status: Confirmed → Triaged
Changed in glance (Ubuntu Quantal):
status: Confirmed → Triaged
Adam Gandelman (gandelman-a) wrote :

Fixed upstream and released into Quantal in glance 2012.2~f2~20120524.1541-0ubuntu1

Changed in glance (Ubuntu Quantal):
status: Triaged → Fix Released
description: updated
Adam Gandelman (gandelman-a) wrote :

glance 2012.1-0ubuntu2.1 awaiting approval for upload into -proposed

Hello Eoghan, or anyone else affected,

Accepted glance into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in glance (Ubuntu Precise):
status: Triaged → Fix Committed
tags: added: verification-needed

I have these packages installed:

root@warthog:~# dpkg --list | grep glance
ii glance 2012.1-0ubuntu2.1 OpenStack Image Registry and Delivery Service - Daemons
ii glance-api 2012.1-0ubuntu2.1 OpenStack Image Registry and Delivery Service - API
ii glance-client 2012.1-0ubuntu2.1 OpenStack Image Registry and Delivery Service - Registry
ii glance-common 2012.1-0ubuntu2.1 OpenStack Image Registry and Delivery Service - Common
ii glance-registry 2012.1-0ubuntu2.1 OpenStack Image Registry and Delivery Service - Registry
ii python-glance 2012.1-0ubuntu2.1 OpenStack Image Registry and Delivery Service - Python library

I can confirm that the problem does not appear anymore with these packages. Thanks for the great work, guys!

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package glance - 2012.1-0ubuntu2.1

---------------
glance (2012.1-0ubuntu2.1) precise-proposed; urgency=low

  * debian/patches/swift_auth_url_trailing_slash.patch: Ensure swift auth URL
    includes trailing slash (LP: #979745). Backported from stable/essex, can
    be dropped with first upstream stable update.
 -- Adam Gandelman <email address hidden> Wed, 23 May 2012 16:19:50 -0700

Changed in glance (Ubuntu Precise):
status: Fix Committed → Fix Released
Brian Murray (brian-murray) wrote :

Hello Eoghan, or anyone else affected,

Accepted glance into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

tags: removed: verification-done
tags: added: verification-needed
Steve Langasek (vorlon) wrote :

This bug was referenced in the changelog of glance 2012.1+stable~20120608-5462295-0ubuntu2.2 in error, having already been fixed in SRU. Marking verification-done.

tags: added: verification-done
removed: verification-needed
Thierry Carrez (ttx) on 2012-09-27
Changed in glance:
milestone: folsom-1 → 2012.2
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers