Rally does not clean the security_group when the nova_boot testing is finished

[Sun Jun]

When I am going to run the nova_boot_server benchmark which is configured as below:
{
    "NovaServers.boot_server": [
        {
            "args": {
                "flavor": {
                    "name": "m1.tiny"
                },
                "image": {
                    "name": "cirros"
                },
                "nics":[{
                    "net-id":"ccd6ce3e-eeba-47b6-a104-63c2ca853aa9"
                }]
            },
            "runner": {
                "type": "constant",
                "times": 10,
                "concurrency": 5
            },
            "context": {
                "users": {
                    "tenants": 10,
                    "users_per_tenant": 1
                },
                "quotas": {
                    "nova": {
                        "instances": 5000,
                        "cores": 50
                    }
                }
            }
        }
    ]
}

and before the testing , I type the command "neutron security-group-list" , the results are:
# neutron security-group-list
+--------------------------------------+---------+-------------+
| id | name | description |
+--------------------------------------+---------+-------------+
| b4fdfac6-4d95-4ec0-85a6-a0c657844920 | default | default |
| ba3bb113-9d47-4d7b-9831-2e724927fb0c | default | default |
+--------------------------------------+---------+-------------+

but when I finish the rally test, the command "neutron security-group-list" shows :
+--------------------------------------+---------+-------------+
| id | name | description |
+--------------------------------------+---------+-------------+
| 0204725e-da34-44ee-b9ea-d93d7ca22630 | default | default |
| 062df76e-3c81-4fd7-9b1a-572f19e39ff9 | default | default |
| 07c0cb44-e444-49ad-a16f-d6ecdee93fd6 | default | default |
| 21f96045-aab9-4ed9-8234-e5ffa7a32c9a | default | default |
| 414c2ab1-3f54-4d97-8015-dca3d646d338 | default | default |
| 6b01f2bc-838e-49a8-86a6-08562b169075 | default | default |
| 8a543f76-77ac-4d13-a540-93f90c60d029 | default | default |
| b4fdfac6-4d95-4ec0-85a6-a0c657844920 | default | default |
| ba3bb113-9d47-4d7b-9831-2e724927fb0c | default | default |
| cf731999-973f-40e8-a597-46ba3b1c6271 | default | default |
| f9c769b6-c488-43eb-823a-91e96799d6ea | default | default |
| fd8fb38a-b984-4e9b-86f1-431273885a2c | default | default |
+--------------------------------------+---------+-------------+

It seems that Rally create neutron a security group for each tenant, but it forget to clean them up.

[Sergey Skripnick (eyerediskin)]

Thanks for reporting. I'll try to reproduce and fix.

[Sergey Skripnick (eyerediskin)]

Just tried rally from master and fresh devstack with neutron, and dont see any extra neutron security groups. Could you please provide more info? Which rally and cloud was used?

[Sun Jun (sanjeo)]

I also ran rally from admin user, but i did not use an all-in-one setup.

rally version:0.0.1 (clone from github on Jul 18)
openstack: 2013.2.3-1.el6 @openstack-havana

[Sergey Skripnick (eyerediskin)]

Reproduced with scenario VMTasks.boot_runcommand_delete

[Dr. Jens Harbott (j-harbott)]

There is a pretty old bug report regarding this, with no real progress made in almost three years:

https://bugs.launchpad.net/keystone/+bug/967832

Would be nice to have some workaround for it in rally until this is solved globally.

[Boris Pavlovic (boris-42)]

Jens,

Actually Rally is cleaning up all resources. We have special mechanism for this. And as far as I know Rally is deleting all security groups now

[Boris Pavlovic (boris-42)]

Jens,

By the way could you reproduce this locally?

[Dr. Jens Harbott (j-harbott)]

Hi Boris,

sorry for the late reply, I didn't subscribe myself to the bug, so I saw your question only now.

Yes, I can reproduce it. I also looked into the code a bit and it seems that there are three issues here:

1. rally.benchmark.context.cleanup.manager tries to delete the secgroups for all users. However, if there are multiple users per tenant, they share the same secgroup, so after the first deletion the other ones produce errors. These however are only seen when running in debug mode, so this is a minor issue.

2. After rally.benchmark.context.cleanup.manager has finished, rally.benchmark.context.secgroup tries to clean up the secgroups once more, resulting in errors that are also shown in normal output. So either the cleanup in rally.benchmark.context.secgroup is obsolete now, or it should run before rally.benchmark.context.cleanup.manager and also remove the corresponding attributes from the context.

These two issues are more or less cosmetical, now comes the major one:

3. While rally does very thoroughly clean up the secgroups that are explicitly created, neutron is playing tricks and also creates a security group called 'default' for every tenant. These are not removed on tenant deletion automatically, so we should add some code into rally in order to do this.

[Dr. Jens Harbott (j-harbott)]

@Sergey: Are you still working on this? Otherwise please unassign yourself and let someone else take over.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/179428

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/179428
Committed: https://git.openstack.org/cgit/openstack/rally/commit/?id=977afacf33a71b6e520a214bba81975367663ab9
Submitter: Jenkins
Branch: master

commit 977afacf33a71b6e520a214bba81975367663ab9
Author: Jens Rosenboom <email address hidden>
Date: Fri May 1 19:08:38 2015 +0200

    Fix security groups cleanup

    In addition to the security groups that are explicitly created, Neutron
    automatically creates a security group called "default" for each tenant.
    We have to clean these up after destroying networks using the admin
    context.

    Co-Authored-By: Chris St. Pierre <email address hidden>
    Change-Id: Ib8da1c608a3355cd2e08a7bc1a5aaa86444d1c39
    Closes-Bug: 1355629