juju leaves security groups behind in aws

[Curtis Hovey]

Juju CI is will reach its limit of 100 security groups in AWS now that so many tests are performed. There are never more than 30 groups being used at any one time, but destroy environment doesn't remove the unused security groups.

CI stalls when the limit is reached. We are writing a proc to clean up every hour. Juju should just clean up.

[Dimiter Naydenov (dimitern)]

Possible fallout due to this - bug 1254402.

[Curtis Hovey (sinzui)]

There are two failing parties in this issue.

1. Juju should delete aws secgroups with the env is destroyed. It does not (bug 1385276)
2. Bundle Tester doesn't cleanup substrates when it is done, preventing Bundle Tester from doing another test in the same substrate.

Juju CI is also affected, many of its test scripts use substrate.py library to setup and teardown substrates to ensure resources are not left behind. substrate.AWSAccount has specific rules to delete sec groups.

juju-ci-tools has a helper script that can cleanup the resources created for an env. It can be added a job like this

export JUJU_HOME=$HOME/cloud-city
$HOME/juju-ci-tools/clean_resources.py -v $name_of_env_from_environments_yaml

[Francis Ginther (fginther)]

Landscape CI also sees this problem when destroying juju environments on openstack clouds. Work around is to add a clean up step to the CI scripts.

[Anastasia (anastasia-macmood)]

@Curtis - we have done changes in ec2 cleanup, specifically for security groups, for eg. https://bugs.launchpad.net/juju-core/+bug/1537620

Is this still an issue?

[james beedy (jamesbeedy)]

I'm hitting this on 2.x. <- @rharding

[Kevin W Monroe (kwmonroe)]

Me too, in both 2.1.2 and 2.2-beta3.

[John A Meinel (jameinel)]

Is this reaching the limit because of the number of machines that you are
running? Or because you are destroying models and it is leaving security
groups behind. I've certainly seen a fair bit in --debug mode where we sit
waiting to consider the destroy to actually be finished while we poll
trying to remove the final security groups. (You can't delete the security
groups until AWS considers all of the machines fully gone.)

If you ^C at exactly the right time, I could certainly see that failing and
the controller really is gone. It may be that you could run it again if we
don't remove the client records until everything is gone-gone.

John
=:->

On Wed, May 3, 2017 at 1:39 PM, Kevin W Monroe <email address hidden>
wrote:

> Me too, in both 2.1.2 and 2.2-beta3.
>
> --
> You received this bug notification because you are subscribed to juju-
> core.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1385276
>
> Title:
> juju leaves security groups behind in aws
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1385276/+subscriptions
>

[Kevin W Monroe (kwmonroe)]

@jam, this is leaving sec groups behind when destroying a model. Repro:

$ juju version
2.2-beta3-xenial-amd64

$ juju add-model foo aws/us-west-1
Using credential 'default' cached in controller
Added 'foo' model on aws/us-west-1 with credential 'default' for user 'admin'

$ juju deploy ubuntu -n 10
Located charm "cs:ubuntu-10".
Deploying charm "cs:ubuntu-10".

## wait for deployment to finish, take a look at the aws console, see 10 new instances/volumes and 11 new secgroups.

$ juju destroy-model foo
...
http://paste.ubuntu.com/24555130/
...

## check aws console, see 10 fewer instances/volumes, but only 1 less secgroup.

The +11 / -1 related to security groups seems strange to me. To put real numbers behind it, I had 3 secgroups before deployment, 14 after i added the ubuntu units, and 13 after destroying the model.

The result is that I have to go to aws every couple weeks and delete secgroups once i hit my limit of 500.

[Kevin W Monroe (kwmonroe)]

Just found something interesting... My controller is on aws/us-west-1. With a model in the same region, I hit this bug.

When adding a new model to a different region (e.g. controller in us-west-1, model in us-east-1), security groups are correctly removed when I destroy that model.

@jamesbeedy, will you confirm that destroying models in the same controller region leaves the security groups behind, while destroying models in a separate region correctly removes them?

[james beedy (jamesbeedy)]

Wow, guess I missed this one :)

@kwmonroe I am still hitting this bug in one way or another. I seem to hit this even when my controller lives in the same region as the model I'm operating in.

Using juju 2.8.6, remove-application operation will clear the machines out of the model as well as terminate them in aws. Unfortunately, the security groups are left behind.

I'm not seeing any pertinent logs anywhere, although I would be happy to provide any info that may help us get to the bottom of this.