AWS SecurityGroup's consistently blocking CI runs

[Charles Butler]

For the last week CI on AWS has been blocked due to juju not cleaning up the security groups. This is known side-effect behavior of the AWS provider, however this consistently blocks deployments on AWS in a fairly short amount of time.

As an example: http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/166/console

[Curtis Hovey (sinzui)]

There are two failing parties in this issue.

1. Juju should delete aws secgroups with the env is destroyed. It does not (bug 1385276)
2. Bundle Tester doesn't cleanup substrates when it is done, preventing Bundle Tester from doing another test in the same substrate.

Juju CI is also affected, many of its test scripts use substrate.py library to setup and teardown substrates to ensure resources are not left behind. substrate.AWSAccount has specific rules to delete sec groups.

juju-ci-tools has a helper script that can cleanup the resources created for an env. It can be added a job like this

export JUJU_HOME=$HOME/cloud-city
$HOME/juju-ci-tools/clean_resources.py -v $name_of_env_from_environments_yaml