A project shouldn't be deleted when there are instances running

[Yogev Rabl]

Description of problem:
currently, a project that has an instance (or instances) can be deleted without even a warning message in the Horizon by a user with an administrative permissions. An active project (meaning a project that has instances running) should have the protection from deletion. If the administrator would like to delete it he should delete the instances first.

Version-Release number of selected component (if applicable):
openstack-nova-cert-2013.2.2-2.el6ost.noarch
python-novaclient-2.15.0-2.el6ost.noarch
openstack-nova-common-2013.2.2-2.el6ost.noarch
openstack-nova-api-2013.2.2-2.el6ost.noarch
openstack-nova-compute-2013.2.2-2.el6ost.noarch
openstack-nova-conductor-2013.2.2-2.el6ost.noarch
openstack-nova-novncproxy-2013.2.2-2.el6ost.noarch
openstack-nova-scheduler-2013.2.2-2.el6ost.noarch
python-nova-2013.2.2-2.el6ost.noarch
openstack-nova-console-2013.2.2-2.el6ost.noarch
openstack-nova-network-2013.2.2-2.el6ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Create an new project.
2. Launch one (or more) instances.
3. Try to delete the project with the admin.

Actual results:
The instances are still running, but they are accessible only through the admin -> intances tab.

Expected results:
The administrator shouldn't be able to delete the project as long as there are instances running.

[David Lapsley (dlapsley)]

Hi Folks:

After discussing this with one of the Keystone core developers, it seems that the preferred approach would be something like the following:

1. Horizon warns on project delete for projects with active instances, but allows it (upon confirm)
2. Keystone notification driver sends notification out to the Queue regarding the project delete
3. Nova listens for these notifications and acts upon them in a configurable manner, doing one of the following:
    - deleting all instances in the project
    - migrating instances on the deleted project to another project
    - nothing

It seems to make sense to do the bulk of the processing within Nova. Attempting to have Horizon delete a non-trivial number of VMs as part of project delete could take a considerable amount of time (imagine deleting a project with thousands of VMs!).

[Yogev Rabl (yrabl)]

David, I can see some problems with migrating the instances to another project:
1. How will the migration affect the receiving project's quota? There's the possibility that the number of instances exceed it.
2. The user that created the instances might not have permissions to access the other project & other users that did had access to these instances now have access. Meaning, it is a security issue.

How can we address these problems?

[Dolph Mathews (dolph)]

+1 for the approach described in comment #1 - except for the possibility of migrating those instances to another project. I find that to be completely counter-intuitive.

[Dolph Mathews (dolph)]

Also, step 3 is already tracked here for nova: https://bugs.launchpad.net/keystone/+bug/967832

[Assaf Muller (amuller)]

Imagine a public cloud: Migrating instances from one tenant to another would mean taking instances that I pay for and moving them to (a random?) project that I'm not paying for. The receiving project might not even be familiar with these instances. David, I assume then you mean another project which the user belongs to. If none exists, what do you do, fallback to methods 1 or 3?

I suggest that Horizon has a warning dialog, and deleting a project deletes all resources owned by that project. I don't think it needs to be configurable. Even if it will be configurable, what will be the default?

[David Lapsley (dlapsley)]

Hi Folks:

The migration component in my comment above (#1) is more of a convenience feature. It seems that this might be more confusing than useful, so sounds like it is better for us to leave it out for now. If we leave this out, sounds like we have consensus on the approach?

Cheers, David.

[Abhishek Talwar (abhishek-talwar)]

An administrator has the rights to delete a project , but according to the bug it should first check if there are any instances running. As of now keystone doesnt check if there are any instances running in the project and deletes it directly. When the amdin tries to list the vm's of a particular project from keystone using uuid of that tenant , an error occurs. Although if we try to list the vm's of admin itself, we get the desired list.
Moreover the command “nova list –tenant <project-name>” (admin only) doesn't work. So kindly provide me some information on this that how should we proceed to list the vm's of a particular project through an admin user.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/114923

[OpenStack Infra (hudson-openstack)]

Change abandoned by David Lyle (<email address hidden>) on branch: master
Review: https://review.openstack.org/114923
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Davanum Srinivas (DIMS) (dims-v)]

Removing "In Progress" status and assignee as change is abandoned.

[Sailaja (sailajap)]

Hello Kanchan,

Are you working on this bug?

[Kanchan Gupta (kanchan-gupta1)]

Hello Sailaja,

No I am not working on this bug, you can pick it up.

Thanks

[Sailaja (sailajap)]

Thanks for the update.

On Wed, May 13, 2015 at 3:37 PM, Kanchan Gupta <email address hidden>
wrote:

> Hello Sailaja,
>
> No I am not working on this bug, you can pick it up.
>
> Thanks
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1288230
>
> Title:
> A project shouldn't be deleted when there are instances running
>
> Status in OpenStack Dashboard (Horizon):
> In Progress
> Status in OpenStack Compute (Nova):
> In Progress
>
> Bug description:
> Description of problem:
> currently, a project that has an instance (or instances) can be deleted
> without even a warning message in the Horizon by a user with an
> administrative permissions. An active project (meaning a project that has
> instances running) should have the protection from deletion. If the
> administrator would like to delete it he should delete the instances first.
>
> Version-Release number of selected component (if applicable):
> openstack-nova-cert-2013.2.2-2.el6ost.noarch
> python-novaclient-2.15.0-2.el6ost.noarch
> openstack-nova-common-2013.2.2-2.el6ost.noarch
> openstack-nova-api-2013.2.2-2.el6ost.noarch
> openstack-nova-compute-2013.2.2-2.el6ost.noarch
> openstack-nova-conductor-2013.2.2-2.el6ost.noarch
> openstack-nova-novncproxy-2013.2.2-2.el6ost.noarch
> openstack-nova-scheduler-2013.2.2-2.el6ost.noarch
> python-nova-2013.2.2-2.el6ost.noarch
> openstack-nova-console-2013.2.2-2.el6ost.noarch
> openstack-nova-network-2013.2.2-2.el6ost.noarch
>
> How reproducible:
> 100%
>
> Steps to Reproduce:
> 1. Create an new project.
> 2. Launch one (or more) instances.
> 3. Try to delete the project with the admin.
>
> Actual results:
> The instances are still running, but they are accessible only through
> the admin -> intances tab.
>
> Expected results:
> The administrator shouldn't be able to delete the project as long as
> there are instances running.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/horizon/+bug/1288230/+subscriptions
>

[Sailaja (sailajap)]

Hi All,

The below are the approaches to fix this bug for the nova-compute scenario.

• To invoke Nova-API from Keystone when the tenant-delete request is issued. For now, atleast the message that instances are active is enough.

• To access the Nova database from Keystone e.g. “instances” table.

• To enable notifications on both Keystone and Nova with a new queue created in Rabbitmq.

Please do let me know the inputs so that I can consolidate the approach to proceed further.

Thanks in advance,

[Sailaja (sailajap)]

Hi All,

The below are the approaches to fix this bug for the openstack-dashboard scenario.

    1. Create a database entry for each tenant with the number of vms provisioned by the tenant.

    2. For the request to delete the project from dahsboard,check the nova database and display a message that the project has active instances.

    3. When there is a request to delete a project,check the list of instances for the project and throw an error message.(the entire flow should occur in the openstack-dashboard )

Please let me know the inputs about these approaches.

Thanks in advance.

[Matthias Runge (mrunge)]

Sailaja,

horizon doesn't have a database. Any check you intend to do needs to rely on API calls. Your second proposal relies on a database. Further more, it would be doomed, if any of the tenants would use nova cli to create another instance.

[Eric Peterson (ericpeterson-l)]

Why is this just focussed on instances? This same situation exists for swift etc and bunches of other possible projects. Should floating IP addresses also be released as well etc?

I agree this is an issue, but I hope a general purpose solution is pursued that all projects can leverage.

[Roysimkes-t (roysimkes-t)]

I agree on Eric. It's not just instances but also networks, routers, volumes (even glance images). I do believe that Horizon should just show an error/confirmation message, telling that "there are still undeleted instances/volumes etc assigned to this project, please confirm that you want to delete it". Migrating and other kind of things causes problems and deleting them all is also a bit risky when it's a big project, so the best course of action should be to keep those resources as is. Maybe something like "purge" can be implemented in the horizon when user selects it then everything related to that project is removed, including instances, networks, ips, volumes, etc.

I vote on #2, but it should not just check instances but also networks, volumes etc.

[Sean Dague (sdague)]

Removing Nova from this bug. As has been expressed this is true of anything in OpenStack.

It's fine for Horizon to warn/block project deletion on these cases. At some level keystone is going to let you do the delete. The theory from Tokyo was that OSC should have a cleanup project plugin

[Akihiro Motoki (amotoki)]

Horizon can implement this kind of things, but IIRC in OpenStackSDK/OSC discussion SDK will implement a framework for project deletion so the reasonable way is to consume it once it is implemented. This avoid duplicated work.

If someone is interested in working on this, check the status of openstacksdk (and OSC) first. Please avoid implementing all logic in horizon directly. If so, consider contributing openstacksdk first.