A project shouldn't be deleted when there are instances running

Bug #1288230 reported by Yogev Rabl
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Won't Fix
Low
Unassigned
OpenStack Dashboard (Horizon)
Confirmed
Low
Unassigned

Bug Description

Description of problem:
currently, a project that has an instance (or instances) can be deleted without even a warning message in the Horizon by a user with an administrative permissions. An active project (meaning a project that has instances running) should have the protection from deletion. If the administrator would like to delete it he should delete the instances first.

Version-Release number of selected component (if applicable):
openstack-nova-cert-2013.2.2-2.el6ost.noarch
python-novaclient-2.15.0-2.el6ost.noarch
openstack-nova-common-2013.2.2-2.el6ost.noarch
openstack-nova-api-2013.2.2-2.el6ost.noarch
openstack-nova-compute-2013.2.2-2.el6ost.noarch
openstack-nova-conductor-2013.2.2-2.el6ost.noarch
openstack-nova-novncproxy-2013.2.2-2.el6ost.noarch
openstack-nova-scheduler-2013.2.2-2.el6ost.noarch
python-nova-2013.2.2-2.el6ost.noarch
openstack-nova-console-2013.2.2-2.el6ost.noarch
openstack-nova-network-2013.2.2-2.el6ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Create an new project.
2. Launch one (or more) instances.
3. Try to delete the project with the admin.

Actual results:
The instances are still running, but they are accessible only through the admin -> intances tab.

Expected results:
The administrator shouldn't be able to delete the project as long as there are instances running.

Tags: compute nova
Changed in nova:
assignee: nobody → Santiago Baldassin (santiago-b-baldassin)
Changed in nova:
assignee: Santiago Baldassin (santiago-b-baldassin) → nobody
Tracy Jones (tjones-i)
tags: added: compute
Revision history for this message
David Lapsley (dlapsley) wrote :

Hi Folks:

After discussing this with one of the Keystone core developers, it seems that the preferred approach would be something like the following:

1. Horizon warns on project delete for projects with active instances, but allows it (upon confirm)
2. Keystone notification driver sends notification out to the Queue regarding the project delete
3. Nova listens for these notifications and acts upon them in a configurable manner, doing one of the following:
    - deleting all instances in the project
    - migrating instances on the deleted project to another project
    - nothing

It seems to make sense to do the bulk of the processing within Nova. Attempting to have Horizon delete a non-trivial number of VMs as part of project delete could take a considerable amount of time (imagine deleting a project with thousands of VMs!).

melanie witt (melwitt)
Changed in nova:
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Yogev Rabl (yrabl) wrote :

David, I can see some problems with migrating the instances to another project:
1. How will the migration affect the receiving project's quota? There's the possibility that the number of instances exceed it.
2. The user that created the instances might not have permissions to access the other project & other users that did had access to these instances now have access. Meaning, it is a security issue.

How can we address these problems?

Revision history for this message
Dolph Mathews (dolph) wrote :

+1 for the approach described in comment #1 - except for the possibility of migrating those instances to another project. I find that to be completely counter-intuitive.

Revision history for this message
Dolph Mathews (dolph) wrote :

Also, step 3 is already tracked here for nova: https://bugs.launchpad.net/keystone/+bug/967832

Revision history for this message
Assaf Muller (amuller) wrote :

Imagine a public cloud: Migrating instances from one tenant to another would mean taking instances that I pay for and moving them to (a random?) project that I'm not paying for. The receiving project might not even be familiar with these instances. David, I assume then you mean another project which the user belongs to. If none exists, what do you do, fallback to methods 1 or 3?

I suggest that Horizon has a warning dialog, and deleting a project deletes all resources owned by that project. I don't think it needs to be configurable. Even if it will be configurable, what will be the default?

Dolph Mathews (dolph)
no longer affects: keystone
Revision history for this message
David Lapsley (dlapsley) wrote :

Hi Folks:

The migration component in my comment above (#1) is more of a convenience feature. It seems that this might be more confusing than useful, so sounds like it is better for us to leave it out for now. If we leave this out, sounds like we have consensus on the approach?

Cheers, David.

Changed in nova:
assignee: nobody → Ravindra Joshi (ravindra-joshi)
Changed in nova:
assignee: Ravindra Joshi (ravindra-joshi) → nobody
melanie witt (melwitt)
Changed in nova:
status: Confirmed → Triaged
importance: Medium → Low
Changed in nova:
assignee: nobody → tcs_openstack_group (tcs-openstack-group)
Changed in horizon:
assignee: nobody → tcs_openstack_group (tcs-openstack-group)
Revision history for this message
Abhishek Talwar (abhishek-talwar) wrote :

An administrator has the rights to delete a project , but according to the bug it should first check if there are any instances running. As of now keystone doesnt check if there are any instances running in the project and deletes it directly. When the amdin tries to list the vm's of a particular project from keystone using uuid of that tenant , an error occurs. Although if we try to list the vm's of admin itself, we get the desired list.
Moreover the command “nova list –tenant <project-name>” (admin only) doesn't work. So kindly provide me some information on this that how should we proceed to list the vm's of a particular project through an admin user.

Changed in horizon:
status: New → Triaged
importance: Undecided → Low
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to horizon (master)

Fix proposed to branch: master
Review: https://review.openstack.org/114923

Changed in horizon:
assignee: tcs_openstack_group (tcs-openstack-group) → Kanchan Gupta (kanchan-gupta1)
status: Triaged → In Progress
Akihiro Motoki (amotoki)
tags: added: nova
Changed in nova:
status: Triaged → In Progress
Changed in nova:
assignee: tcs_openstack_group (tcs-openstack-group) → Abhishek Talwar (abhishek-talwar)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on horizon (master)

Change abandoned by David Lyle (<email address hidden>) on branch: master
Review: https://review.openstack.org/114923
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

Removing "In Progress" status and assignee as change is abandoned.

Changed in nova:
status: In Progress → Confirmed
assignee: Abhishek Talwar (abhishek-talwar) → nobody
Sailaja (sailajap)
Changed in nova:
status: Confirmed → In Progress
assignee: nobody → Sailaja (sailajap)
Revision history for this message
Sailaja (sailajap) wrote :

Hello Kanchan,

Are you working on this bug?

Revision history for this message
Kanchan Gupta (kanchan-gupta1) wrote :

Hello Sailaja,

No I am not working on this bug, you can pick it up.

Thanks

Revision history for this message
Sailaja (sailajap) wrote : Re: [Bug 1288230] Re: A project shouldn't be deleted when there are instances running

Thanks for the update.

On Wed, May 13, 2015 at 3:37 PM, Kanchan Gupta <email address hidden>
wrote:

> Hello Sailaja,
>
> No I am not working on this bug, you can pick it up.
>
> Thanks
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1288230
>
> Title:
> A project shouldn't be deleted when there are instances running
>
> Status in OpenStack Dashboard (Horizon):
> In Progress
> Status in OpenStack Compute (Nova):
> In Progress
>
> Bug description:
> Description of problem:
> currently, a project that has an instance (or instances) can be deleted
> without even a warning message in the Horizon by a user with an
> administrative permissions. An active project (meaning a project that has
> instances running) should have the protection from deletion. If the
> administrator would like to delete it he should delete the instances first.
>
> Version-Release number of selected component (if applicable):
> openstack-nova-cert-2013.2.2-2.el6ost.noarch
> python-novaclient-2.15.0-2.el6ost.noarch
> openstack-nova-common-2013.2.2-2.el6ost.noarch
> openstack-nova-api-2013.2.2-2.el6ost.noarch
> openstack-nova-compute-2013.2.2-2.el6ost.noarch
> openstack-nova-conductor-2013.2.2-2.el6ost.noarch
> openstack-nova-novncproxy-2013.2.2-2.el6ost.noarch
> openstack-nova-scheduler-2013.2.2-2.el6ost.noarch
> python-nova-2013.2.2-2.el6ost.noarch
> openstack-nova-console-2013.2.2-2.el6ost.noarch
> openstack-nova-network-2013.2.2-2.el6ost.noarch
>
> How reproducible:
> 100%
>
> Steps to Reproduce:
> 1. Create an new project.
> 2. Launch one (or more) instances.
> 3. Try to delete the project with the admin.
>
> Actual results:
> The instances are still running, but they are accessible only through
> the admin -> intances tab.
>
> Expected results:
> The administrator shouldn't be able to delete the project as long as
> there are instances running.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/horizon/+bug/1288230/+subscriptions
>

Changed in horizon:
assignee: Kanchan Gupta (kanchan-gupta1) → Sailaja (sailajap)
Revision history for this message
Sailaja (sailajap) wrote :

Hi All,

The below are the approaches to fix this bug for the nova-compute scenario.

• To invoke Nova-API from Keystone when the tenant-delete request is issued. For now, atleast the message that instances are active is enough.

• To access the Nova database from Keystone e.g. “instances” table.

• To enable notifications on both Keystone and Nova with a new queue created in Rabbitmq.

Please do let me know the inputs so that I can consolidate the approach to proceed further.

Thanks in advance,

Revision history for this message
Sailaja (sailajap) wrote :

Hi All,

The below are the approaches to fix this bug for the openstack-dashboard scenario.

    1. Create a database entry for each tenant with the number of vms provisioned by the tenant.

    2. For the request to delete the project from dahsboard,check the nova database and display a message that the project has active instances.

    3. When there is a request to delete a project,check the list of instances for the project and throw an error message.(the entire flow should occur in the openstack-dashboard )

Please let me know the inputs about these approaches.

Thanks in advance.

Revision history for this message
Matthias Runge (mrunge) wrote :

Sailaja,

horizon doesn't have a database. Any check you intend to do needs to rely on API calls. Your second proposal relies on a database. Further more, it would be doomed, if any of the tenants would use nova cli to create another instance.

Revision history for this message
Eric Peterson (ericpeterson-l) wrote :

Why is this just focussed on instances? This same situation exists for swift etc and bunches of other possible projects. Should floating IP addresses also be released as well etc?

I agree this is an issue, but I hope a general purpose solution is pursued that all projects can leverage.

Revision history for this message
Roysimkes-t (roysimkes-t) wrote :

I agree on Eric. It's not just instances but also networks, routers, volumes (even glance images). I do believe that Horizon should just show an error/confirmation message, telling that "there are still undeleted instances/volumes etc assigned to this project, please confirm that you want to delete it". Migrating and other kind of things causes problems and deleting them all is also a bit risky when it's a big project, so the best course of action should be to keep those resources as is. Maybe something like "purge" can be implemented in the horizon when user selects it then everything related to that project is removed, including instances, networks, ips, volumes, etc.

I vote on #2, but it should not just check instances but also networks, volumes etc.

Changed in nova:
assignee: Sailaja (sailajap) → nobody
status: In Progress → Confirmed
Revision history for this message
Sean Dague (sdague) wrote :

Removing Nova from this bug. As has been expressed this is true of anything in OpenStack.

It's fine for Horizon to warn/block project deletion on these cases. At some level keystone is going to let you do the delete. The theory from Tokyo was that OSC should have a cleanup project plugin

Changed in nova:
status: Confirmed → Won't Fix
Ivan Kolodyazhny (e0ne)
Changed in horizon:
assignee: Sailaja (sailajap) → Ivan Kolodyazhny (e0ne)
status: In Progress → Confirmed
assignee: Ivan Kolodyazhny (e0ne) → nobody
Revision history for this message
Akihiro Motoki (amotoki) wrote :

Horizon can implement this kind of things, but IIRC in OpenStackSDK/OSC discussion SDK will implement a framework for project deletion so the reasonable way is to consume it once it is implemented. This avoid duplicated work.

If someone is interested in working on this, check the status of openstacksdk (and OSC) first. Please avoid implementing all logic in horizon directly. If so, consider contributing openstacksdk first.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers