juju 2 doesn't remove openstack security groups

[Francis Ginther]

This was most recently observed with 2.0-beta18-xenial-amd64. The cloud is provided by a landscape autopilot deploy of openstack version 'mitaka'.

I noticed this with 'destroy-model' and 'kill-controller', but it may happen when removing services and machines too. In my testing, I do see that 'kill-controller' attempts to remove security groups, but is just unsuccessful:

buntu@juju2-b18:~/inner$ juju kill-controller -y osa-controller
Destroying controller "osa-controller"
Waiting for resources to be reclaimed
All hosted models reclaimed, cleaning up controller machines
WARNING cannot delete security group "juju-b5ee8749-31c5-48a4-8b2c-a53e45d2c9bd-dcad8c29-df6a-46f8-8094-bb0094a1af3b
-0". Used by another model?
ERROR health ping timed out after 30s
WARNING cannot delete security group "juju-b5ee8749-31c5-48a4-8b2c-a53e45d2c9bd-dcad8c29-df6a-46f8-8094-bb0094a1af3b
". Used by another model?
WARNING cannot delete security group "juju-b5ee8749-31c5-48a4-8b2c-a53e45d2c9bd-dcad8c29-df6a-46f8-8094-bb0094a1af3b
-0". Used by another model?
WARNING cannot delete security group "juju-b5ee8749-31c5-48a4-8b2c-a53e45d2c9bd-275b148e-9e0b-49f3-8f71-5441fe95dbd9
". Used by another model?
WARNING cannot delete security group "juju-b5ee8749-31c5-48a4-8b2c-a53e45d2c9bd-275b148e-9e0b-49f3-8f71-5441fe95dbd9
-0". Used by another model?
WARNING cannot delete security group "juju-b5ee8749-31c5-48a4-8b2c-a53e45d2c9bd-dcad8c29-df6a-46f8-8094-bb0094a1af3b
". Used by another model?
WARNING cannot delete security group "juju-b5ee8749-31c5-48a4-8b2c-a53e45d2c9bd-dcad8c29-df6a-46f8-8094-bb0094a1af3b
-0". Used by another model?

A short time later (less then a minute), the security groups could be removed with 'openstack security group delete'. I only see these messages with 'kill-controller'. The 'destroy-model' command didn't appear to attempt to remove any security groups.

This becomes a problem once a couple of deploys have been performed and removed and I start to run out of openstack security groups.

Attached is a script log showing 'bootstrap', 'deploy', 'destroy-model' and 'kill-controller' and the security groups in between.

[Curtis Hovey (sinzui)]

Which version of Openstack. Juju appears to clean up in some, but not all openstacks.

[Francis Ginther (fginther)]

The openstack version is use is 'liberty'.

[Francis Ginther (fginther)]

Correction, the openstack version is mitaka.

[Antti Rahikainen (anrah)]

Security group deletion also broken when using OpenStack Newton.

[Martijn Evers (mevers)]

Here too for Mitaka and 2.0.0-trusty-amd64.

[Liam Young (gnuoy)]

I think I've got to the bottom of this, its a bug in goose so I assume it affects Juju 1.X and 2.X at the moment.

When Juju requests the deletion of a security group it hits the API with these parameters:

sendRateLimitedRequest method: DELETE
sendRateLimitedRequest URL: http://10.245.161.158:8774/v2/a65e1673ca784ec3b3a4b76c95a254b0/os-security-groups/3e4ff483-4b12-4a07-b466-ed93a146a56a
sendRateLimitedRequest headers: map[User-Agent:[goose (0.1.0)] X-Auth-Token:[54baab4248574e5c862de354100021fc] Content-Type:[application/octet-stream] Accept:[application/octet-stream]]
sendRateLimitedRequest reqData: []

(Output above doesn't match the uuids below because there from different test runs but you get the picture)

If I take those parameters and use curl I get:

$ curl -g -i -X DELETE http://10.245.161.158:8774/v2/a65e1673ca784ec3b3a4b76c95a254b0/os-security-groups/592c0b2e-ce86-4c6a-9778-faf8c89cdf61 -H "User-Agent: goose (0.1.0)" -H "Content-Type:[application/octet-stream]
" -H "Accept: application/octet-stream" -H "X-Auth-Token: b11ada0df6a342e1a667e579f6e6f153"
HTTP/1.1 415 Unsupported Media Type
Content-Type: application/json; charset=UTF-8
Content-Length: 70
X-Compute-Request-Id: req-d19a46a2-736f-479d-a0b4-156ffa4da8de
Date: Wed, 23 Nov 2016 12:15:02 GMT

{"badMediaType": {"message": "Unsupported Content-Type", "code": 415}

So the delete, fails and Juju keeps on trying. The problem is the sending of the "Content-Type:[application/octet-stream]" header, which isn't needed at all since I don't think DELETEs ever have a payload?

I have patched juju 1.X locally ( https://pastebin.canonical.com/171596/ ) to remove the Content-Type header from Delete requests for JsonRequest and BinaryRequest. This fixes the issue for me.

[Liam Young (gnuoy)]

Sorry, patch was reversed https://pastebin.canonical.com/171602/

[Anastasia (anastasia-macmood)]

Changed to "critical" for 1.25.9 as we only fix "critical"s in 1.25.x series.

[James Page (james-page)]

potentially related bug 1645298

[Mario Splivalo (mariosplivalo)]

I have built patched juju 1.25.6 based on Liam's comment (https://pastebin.canonical.com/171602/):

https://launchpad.net/~mariosplivalo/+archive/ubuntu/lp1625624

I've tested this against our testing Openstack (Mitaka), and security groups get deleted. Moreover, 'juju destroy-environment' takes cca 20 seconds (compared to cca 15 minutes when juju is failing to delete security groups, retrying along the way).

[Liam Young (gnuoy)]

Pull request with fix:

https://github.com/go-goose/goose/pull/37

[Anastasia (anastasia-macmood)]

PR against 1.25: https://github.com/juju/juju/pull/6638

PR against 2.0: https://github.com/juju/juju/pull/6640

[Dean Maniatis (dean-maniatis)]

I'm facing the same problem when using a private openstack mitaka with juju 2.0.1. In addition of not being able to remove security groups during removal of a service or even when destroying controller I've also noticed that it creates 2 security groups per service. Is that expected?

[Martin Packman (gz)]

And final dep bump for develop/2.1 branch:

<https://github.com/juju/juju/pull/6651>

[Nicholas Skaggs (nskaggs)]

These changes are in the proposed ppa / stream. Confirmation that this doesn't occur in the proposed client is appreciated. See https://launchpad.net/~juju/+archive/ubuntu/1.25-proposed, and the ppa description for details on setting your environment to use the proposed agents to go with the binary.

Francis, thank you in advance for verifying.

[Francis Ginther (fginther)]

@nskaggs, verified with juju 1.25.9-0ubuntu1~14.04.2~juju1 that openstack security groups were removed with environment-destroy. This was tested with mitaka.

Have not tested with any version of juju 2 yet. Will attempt that next.

[Francis Ginther (fginther)]

@nskaggs, is there a 2.0.3 package available for testing?

[Dean Maniatis (dean-maniatis)]

The fix committed doesn't seem to address fully openstack Mitaka. I've upgraded existing bootstrapped controller to 2.1-beta3.1, I added a new model and deployed a couple services. This yielded +3 new security groups (+2 on the first deployed charm and +1 on the second). Upon removal of both charms only 2 security groups were removed instead of 3 and I had to remove the last one by hand.

[Anastasia (anastasia-macmood)]

It looks like the fix was applied to Juju 1.25.x and Juju 2.0.x but was not forward-ported to 2.1.x and 2.2.x. We'll get right on it!
Thank you for reporting it, Dean!

[Anastasia (anastasia-macmood)]

After closer examination, it looks like fixes has been forward-ported.

The extra security group that you are seeing probably belongs to the model. It was created with deployment of first application. If you re-try - i.e. (a) create model, (b) deploy application, (c) destroy application, (d) destroy model - you should not have any security groups related to that model or its applications.