juju2 not cleaning up nova secgroups with openstack provider

[Brad Marshall]

I've been testing out juju2 with a Mitaka openstack provider, and have found that it doesn't always clean up its secgroups.

$ juju2 destroy-controller local.cs
WARNING! This command will destroy the "local.cs" controller.
This includes all machines, services, data and other resources.

Continue [y/N]? y
Destroying controller "local.cs"
WARNING cannot delete security group "juju-643cc133-3ba4-4df3-8888-af56dec32725-0". Used by another model?
WARNING cannot delete security group "juju-643cc133-3ba4-4df3-8888-af56dec32725-1". Used by another model?
WARNING cannot delete security group "juju-643cc133-3ba4-4df3-8888-af56dec32725-2". Used by another model?
WARNING cannot delete security group "juju-643cc133-3ba4-4df3-8888-af56dec32725". Used by another model?
WARNING cannot delete security group "juju-643cc133-3ba4-4df3-8888-af56dec32725-0". Used by another model?
WARNING cannot delete security group "juju-643cc133-3ba4-4df3-8888-af56dec32725-1". Used by another model?
WARNING cannot delete security group "juju-643cc133-3ba4-4df3-8888-af56dec32725-2". Used by another model?
jujumanage@corsola:~$ nova secgroup-list
+--------------------------------------+---------------------------------------------+------------------------+
| Id | Name | Description |
+--------------------------------------+---------------------------------------------+------------------------+
| a07f9c66-1d0d-4aae-9a2d-23cc4a4e387d | default | Default security group |
| 842ec71c-e4c9-4a9e-8d05-dea114b41566 | juju-643cc133-3ba4-4df3-8888-af56dec32725 | juju group |
| 1219a123-345d-4cdf-b640-abf94529e5b6 | juju-643cc133-3ba4-4df3-8888-af56dec32725-0 | juju group |
| fe9d9633-055f-44bc-84c5-ac33ae12c3a7 | juju-643cc133-3ba4-4df3-8888-af56dec32725-1 | juju group |
| 810ce19e-ecd0-4f99-a16d-9bc1f00645bd | juju-643cc133-3ba4-4df3-8888-af56dec32725-2 | juju group |
| e6aa995b-ed49-42a9-ab40-f5ed57b42f37 | juju-e5316ff5-8331-496a-8ba2-d1a4b911ca4a | juju group |
| 78b893c4-7987-4b3e-b69d-97d1a0843075 | juju-e5316ff5-8331-496a-8ba2-d1a4b911ca4a-0 | juju group |
+--------------------------------------+---------------------------------------------+------------------------+

This accumulation of secgroup can easily lead to running out of quota on Openstack - easy enough to fix, but would be good not to have to.

This was with juju2 2.0-beta3-trusty-amd64.

Please let me know if you need any more information.

[Curtis Hovey (sinzui)]

Juju CI saw this too with beta3, but the issue was fixed for CI during the development of beta4. Are you still seeing this? If so which version of openstack are you using?

[Brad Marshall (brad-marshall)]

Interesting, I am still seeing it with juju 2.0 beta4. This is an Openstack Mitaka deployment running on Trusty. Please let me know if you need any further information.

$ juju-2.0 bootstrap cs canonistack --config network=<net-id>
Creating Juju controller "local.cs" on canonistack/bootstack-canonistack-bos01
Bootstrapping model "admin"
Starting new instance for initial controller
Launching instance
 - 9ffc1d4d-1678-44f3-a9eb-13b9d3ac5dbb
Installing Juju agent on bootstrap instance
Preparing for Juju GUI 2.1.2 release installation
Waiting for address
Attempting to connect to 10.48.128.20:22
Logging to /var/log/cloud-init-output.log on remote host
Running apt-get update
Running apt-get upgrade
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: cloud-utils
Installing package: cloud-image-utils
Installing package: tmux
Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/agent/2.0-beta4/juju-2.0-beta4-trusty-amd64.tgz]>
Bootstrapping Juju machine agent
Starting Juju machine agent (jujud-machine-0)
Bootstrap agent installed
Waiting for API to become available: upgrade in progress (upgrade in progress)
Waiting for API to become available: upgrade in progress (upgrade in progress)
Bootstrap complete, local.cs now available.

$ nova secgroup-list
+--------------------------------------+---------------------------------------------+------------------------+
| Id | Name | Description |
+--------------------------------------+---------------------------------------------+------------------------+
| a07f9c66-1d0d-4aae-9a2d-23cc4a4e387d | default | Default security group |
| 5a397032-0988-4f4f-8c8b-bfe7a736658b | juju-f7a339df-5bb2-41c5-8fcf-5131a6610272 | juju group |
| 3f7e430f-732e-42d1-bdbf-98044f56e990 | juju-f7a339df-5bb2-41c5-8fcf-5131a6610272-0 | juju group |
+--------------------------------------+---------------------------------------------+------------------------+

$ juju-2.0 destroy-controller local.cs
WARNING! This command will destroy the "local.cs" controller.
This includes all machines, services, data and other resources.

Continue [y/N]? y
Destroying controller
Waiting for hosted model resources to be reclaimed
All hosted models reclaimed, cleaning up controller machines
WARNING cannot delete security group "juju-f7a339df-5bb2-41c5-8fcf-5131a6610272-0". Used by another model?
WARNING cannot delete security group "juju-f7a339df-5bb2-41c5-8fcf-5131a6610272". Used by another model?
WARNING cannot delete security group "juju-f7a339df-5bb2-41c5-8fcf-5131a6610272-0". Used by another model?

$ nova secgroup-list
+--------------------------------------+---------------------------------------------+------------------------+
| Id | Name | Description |
+-------------...

[Jason Hobbs (jason-hobbs)]

Possibly related to, bug #1509635, in v-pil, where models are added and removed frequently, we're seeing juju spam nova-cloud-controller constantly to remove security groups, and that's failing, and may be causing openstack to fail altogether too, as eventually we can't allocate any instances due to nova cloud controller not being able to communicate with neutron server.

[Anastasia (anastasia-macmood)]

The logic of the controller destruction will try to destroy an instance and its related resources sequentially.
For each instance, we get all resources that this instance needed, i.e. these resources were tagged by this instance at some stage. Of course, some of the resources may have been tagged by more than one instance. They will have several "instance" tags. For these shared resources, whenever we destroy an instance, we will try to attempt to delete them as well. However, we'd get the message originally reported "cannot delete security group ... Used by another model". Eventually, only one instance tag will remain on the shared resource and it will be deleted with that last instance. For environments where resources are shared by many models and instances, this process of getting down to one last "instance" may take a while.

Current status of this bug:
1. CI is not seeing it since beta 4. (if the failure is till there, maybe a functional test needs to be designed? I am adding an 'eda' tag for QA to consider if there is a gap. I suspect the number of units/models will play a role in reproducing this scenario.);
2. Bug #1509635 from comment #3, has been fix released in beta18 last week;
3. There was another bug # 1620415 filed by QA. However as per comment #1 on it, the groups are deleted successfully after a while and we are not reporting it.

Since this was last seen 14 betas back, is there a possibility for you to reproduce it with newer beta? If the failure is there, we would really appreciate logs from controller machine.

[Heather Lanigan (hmlanigan)]

I've hit this with juju-2.0-beta18, during a bootstrap failure. Bug 1625830.