OpenStack Heat

Stack delete fails when swift container has objects

Bug #1406263 reported by Maish
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Undecided
Pavlo Shchelokovskyy
OpenStack Heat 2015.1.0 "kilo"

Bug Description

When creating a swift container in Heat - and thereafter populating that container with files - it will not be possible to delete the stack - without removing the files within the container beforehand.

If stack-delete is run - it returns DELETE_FAILED
heat-stack show <stack_name> returns

" Resource DELETE failed: ClientException: Container
DELETE failed: http://10.56.187.25:8080/v1/AUTH_281a250
8384e4879b6d02798dd132587/swift-SwiftContainerWebsite-
ex27jdxpzjcq 409 Conflict [first 60 chars of response]
<html><h1>Conflict</h1><p>There was a c "

Error message is non-comprehensible.

I assume this is connected to this bug - https://bugs.launchpad.net/horizon/+bug/1096084

Shouldn't deleting a stack also delete the container and all it's contents?

Tags: heat
Revision history for this message
Pavlo Shchelokovskyy (pshchelo) wrote :

At the basis it is not really Heat's problem, as Heat only manages the container itself (we do not have resources for objects inside the containers). If Swift demands a container to be empty on delete so be it. We probably could list all the objects in the container and delete them one by one, but that would be a deviation from Swift behaviour, plus if it was not created by Heat, Heat has no business deleting it.

But surely, the error message can be improved.

Changed in heat:
assignee: nobody → Pavlo Shchelokovskyy (pshchelo)
Revision history for this message
Maish (maishsk) wrote :

Hi Pavlo - in essence I agree, Heat does manage the files in the container.

Swift has the option to remove a container and all objects within the container http://docs.openstack.org/cli-reference/content/swiftclient_commands.html#swiftclient_subcommand_delete
swift delete --all

I would further ask then what is the use of the deletion policy ?
http://docs.openstack.org/developer/heat/api/heat.engine.resources.swift.html#heat.engine.resources.swift.SwiftContainer

Revision history for this message
Pavlo Shchelokovskyy (pshchelo) wrote :

Maish, agree, I also see that shell command of "swift delete XXX" just recursively deletes the container and all of its content, although Python API does not allow to do that with delete_container() method without specific options. May be we could use a deletion policy to specify if deleting non-empty containers is allowed or not.

Revision history for this message
Jason Dunsmore (jasondunsmore) wrote :

I think it makes sense to delete the container only if it's empty, like in the SwiftSignal resource.
https://github.com/openstack/heat/blob/master/heat/engine/resources/swiftsignal.py#L124-L131

Ideally, we'd have a way to warn the user that the container wasn't empty, so it wasn't deleted. I'm not sure we should fail the delete.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/144317

Changed in heat:
status: New → In Progress
Revision history for this message
Pavlo Shchelokovskyy (pshchelo) wrote :

After some thought I ended up just failing the delete with a more clear message as a reason of failure. AFAIU we can not warn the heatclient user without failing a delete, and leaving the container to exist after successfully deleting the stack will leave a presumably billable resource behind, which we probably do not want (as for TempUrls they have limited time span, so while still not ideal, the user would not end up with unlimited sum billed for them).

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to heat (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/144476

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/144317
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=ae0ccc4b51d3f6755e53c67f2e7895f3713b56dc
Submitter: Jenkins
Branch: master

commit ae0ccc4b51d3f6755e53c67f2e7895f3713b56dc
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Mon Dec 29 15:37:41 2014 +0000

    Better error on deleting non-empty Swift container

    Swift client method delete_container() does not allow to delete
    non-empty containers, in which case user gets a confusing 409 error
    exessively truncated by Swift with no actual reason information.

    This patch catches 409 errors from Swift client on container delete,
    checks for objects in the container and raises a suitable error if
    container is not empty, so that the DELETE_FAILED reason is clearly
    seen.

    Change-Id: Ibca3851d11b509413b739a693b1ddd244479d37b
    Closes-Bug: #1406263

Changed in heat:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to heat (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/144779

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to heat (master)

Reviewed: https://review.openstack.org/144779
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=3eda66271314c327668272f902e0cafe9e4f1428
Submitter: Jenkins
Branch: master

commit 3eda66271314c327668272f902e0cafe9e4f1428
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Fri Jan 2 19:09:41 2015 +0200

    Gracefully fail to delete nonempty S3 Bucket

    The AWS::S3::Bucket resource is susceptible to bug #1406263
    as it is backed by the same Swift container - on deleting non empty
    container it produces incomprehensible truncated message.

    This patch uses the same logic as it is now in OS::Swift::Container
    resource, making the reason of failure clear. The error message is made
    to resemble the error produced by this resource on real AWS
    in the same usage scenario.

    Change-Id: Ib67c55f010613c8edb913bb264d3cc4a7315d847
    Related-Bug: #1406263

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/144476
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=db4b79f3a884dd197c0e3c32bc38464e46de42d5
Submitter: Jenkins
Branch: master

commit db4b79f3a884dd197c0e3c32bc38464e46de42d5
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Tue Dec 30 16:23:18 2014 +0000

    Add PurgeOnDelete property to OS::Swift::Container

    When set to True, this resource will delete all the objects inside
    the container before deleting container itself.
    Default is False for backward compatibility.

    Change-Id: I34582e97d5597cb81c1f001dedb17e2b8436afc1
    Related-Bug: #1406263

Thierry Carrez (ttx)
Changed in heat:
milestone: none → kilo-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in heat:
milestone: kilo-2 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.