Resources owned by a project/tenant are not cleaned up after that project is deleted from keystone

[Lars Erik Pedersen]

If you have running instances in a tenant, then remove all the users, and finally delete the tenant, the instances are still running. Causing serious trouble, since nobody has access to delete them. Also affects the "instances" page in horizon. It will break if this scenario occurs.

[Gabriel Hurley (gabriel-hurley)]

This is actually tracked in the blueprint tenant-deletion (https://blueprints.launchpad.net/horizon/+spec/tenant-deletion), but I'll leave the bug as a reminder.

[Joseph Heck (heckj)]

Needs some discussion about how resources should be treated from a UX point of view at the Folsom design summit. My inclination would be to destroy all resources owned by a tenant when it's being removed, but that may be too harsh, and doesn't take into account future states where trust may be delegated.

Regardless, that also implies an inversion of control where Keystone would potentially need to reach into other systems with administrative access to impact resources, which I'm not sure I like.

[Jay Pipes (jaypipes)]

For the record, I agree that the action should be to terminate and delete all resources attached to a tenant when the tenant is deleted...

[Matt Joyce (matt-nycresistor)]

I agree that, when a tenant is deleted, the running instances associated with it should be terminated.

[Tong Li (litong01)]

The easier and non destructive approach for this problem may be to check the resources (such as images, instances, volumes) under the tenant and make sure there is not anything left to be hanging around without anyone can access them when tenant get deleted. When the detects any such resource, the delete tenant should not proceed and should tell the user that it can not be deleted due to the resource issues. It also forces the operator to handle these resources first. In many cases, this approach might be a safer way to deal with deletion.

[Stephan Fabel (sfabel)]

In regards to #5, for what it's worth, I would like a choice... let me delete all resources but confirm with me first.

[Craig Hadix (info-jjcftv6wldnzq84cskygyvhqqb9qwjfcq0yfnwzcca0ux8ircw2a3om624q2ycdp941uw5474gcdbi2qtcnliiwmmp1l)]

I like a verification here where there is both a 'are you sure' along with an optional password, or other validated check, will prevent the last user from being deleted if resources are still active... however if the override is approved then the instances should be blown away with the last user

[Gabriel Hurley (gabriel-hurley)]

Closing this as a bug as it is superseded by the blueprint https://blueprints.launchpad.net/horizon/+spec/tenant-deletion

[Joe Gordon (jogo)]

This won't be fixed in Nova in Grizzly, and is something that is cross project and needs to be hashed out at the Summit.

[Joshua Harlow (harlowja)]

Wouldn't a good way be to have keystone broadcast user/tenant/role deletion/creation and let downstream systems consume said messages? Then nova can react however it wants, glance to, and any other downstream system that desires to take some kind of action when users/tenants/roles are deleted/added...

[Thierry Carrez (ttx)]

<dolphm> that's being worked in bp notifications
<dolphm> https://blueprints.launchpad.net/keystone/+spec/notifications
<ttx> so for havana-2 ?
<dolphm> which is blocked by https://blueprints.launchpad.net/keystone/+spec/unified-logging-in-keystone ... which we just added to m1 and is nearly complete
<dolphm> ttx: yes, looks to be on pace for m2

[Adam Young (ayoung)]

The RPC mechanism assumes eventlet. Many people are running with Keystone in HTTPD and without eventlet. Is it possible that we can do the notification mechanism using straight AMQP and punt on 0Mq for a first implementation? We just need to be able to publish notifications, not accept them, so Keystone does not need the full RPC.

[Dolph Mathews (dolph)]

Removed m2 target because bp notifications has some blockers; it looks like it won't land during Havana at this point.

[Dolph Mathews (dolph)]

keystone now emits notifications when projects/tenants are delete as part of https://blueprints.launchpad.net/keystone/+spec/notifications

[Dolph Mathews (dolph)]

Added glance as a result of https://bugs.launchpad.net/keystone/+bug/1221732

[Iccha Sethi (iccha-sethi)]

I feel it is the responsibility of something external which does clean tenant deletion like horizon(https://blueprints.launchpad.net/horizon/+spec/tenant-deletion) should be doing the image members clean up

[Mark Washenberger (markwash)]

I don't think it makes sense to look at this as a glance issue.

I agree some work may be needed in glance as in other projects to respond to tenant deletion events in a sensible way. But glance shouldn't be innovating that independently. Sounds like a cross-cutting-concern for all projects.

[Dolph Mathews (dolph)]

This was brought up again (specifically for nova) in bug https://bugs.launchpad.net/nova/+bug/1288230

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/92599

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/92600

[kesten broughton (dathomir)]

Not just the instances, but likely the subnet, router, gateway etc. Once
the tenant is gone it is very difficult to get the id's you need. If you
do a quantum (nova) port-list you will likely see lots of floating and
fixed ips related to it as well.

On Wed, May 7, 2014 at 9:37 AM, Openstack Gerrit
<email address hidden>wrote:

> Fix proposed to branch: master
> Review: https://review.openstack.org/92599
>
> ** Changed in: neutron
> Status: Confirmed => In Progress
>
> --
> You received this bug notification because you are subscribed to
> neutron.
> https://bugs.launchpad.net/bugs/967832
>
> Title:
> Resources owned by a project/tenant are not cleaned up after that
> project is deleted from keystone
>
> Status in OpenStack Image Registry and Delivery Service (Glance):
> Won't Fix
> Status in OpenStack Dashboard (Horizon):
> Won't Fix
> Status in OpenStack Identity (Keystone):
> Fix Released
> Status in OpenStack Neutron (virtual network service):
> In Progress
> Status in OpenStack Compute (Nova):
> Confirmed
>
> Bug description:
> If you have running instances in a tenant, then remove all the users,
> and finally delete the tenant, the instances are still running.
> Causing serious trouble, since nobody has access to delete them. Also
> affects the "instances" page in horizon. It will break if this
> scenario occurs.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/glance/+bug/967832/+subscriptions
>

[Assaf Muller (amuller)]

Neutron spec:
https://review.openstack.org/#/c/98097/

[Assaf Muller (amuller)]

Ryan, I *just* resumed working on this issue today (In Neutron). I think we should have a summit design session about an OpenStack wide solution. I hope you're coming to Paris :)

[Ryan Hallisey (rthall14)]

Hey Assaf,
I also picked this up again only about a week ago.
That would be awesome if we could do a summit design session about this solution!
I sumbitted a presentation related to openstack and selinux, so hopefully
I will get accepted and be headed to Paris!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/115964

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/115965

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/115966

[OpenStack Infra (hudson-openstack)]

Change abandoned by Salvatore Orlando (<email address hidden>) on branch: master
Review: https://review.openstack.org/92600
Reason: This patch has been inactive long enough that I think it's safe to abandon.
The author can resurrect it if needed.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Salvatore Orlando (<email address hidden>) on branch: master
Review: https://review.openstack.org/115964
Reason: This patch has been inactive long enough that I think it's safe to abandon.
The author can resurrect it if needed.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Salvatore Orlando (<email address hidden>) on branch: master
Review: https://review.openstack.org/115965
Reason: This patch has been inactive long enough that I think it's safe to abandon.
The author can resurrect it if needed.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Salvatore Orlando (<email address hidden>) on branch: master
Review: https://review.openstack.org/115966
Reason: This patch has been inactive long enough that I think it's safe to abandon.
The author can resurrect it if needed.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Assaf Muller (<email address hidden>) on branch: master
Review: https://review.openstack.org/92599

[Matt Riedemann (mriedem)]

Did anything happen at the Kilo summit in Paris about this? Are there any mailing list threads on this because if it hasn't come to a summit yet we should talk about it in the cross-project sessions in Vancouver.

A simple approach w/o listening on event notifications (which would be the slickest way) would be to have a periodic task running which is checking resource tenants to see if the tenant exists and if not, reap them (like we have for orphaned instances).

[Matt Riedemann (mriedem)]

Started a mailing list thread: http://lists.openstack.org/pipermail/openstack-dev/2015-February/055801.html

[Dolph Mathews (dolph)]

I agree with Mark's assertion in comment #17 that Glance shouldn't be going about a solution alone, but this certainly affects Glance.

[Ian Cordasco (icordasc)]

This also certainly affects Horizon

[Sean Dague (sdague)]

The Tokyo Summit solution here was that this should be done via an osc plugin. There are really dramatic issues with auto delete from keystone deletes. Many sites need an archive process. Nova itself soft deletes many resources, and even has the ability to set an undo time on some of them.

This shouldn't be an automatic process in the cloud, it should be deliberate. Just like not deleting all the files on your system owned by a user if you remove that user from /etc/passwd.