juju using Openstack provider does not remove security groups on remove-machine after a failed provisioning

[Gareth Woolridge]

When using the Openstack provider on juju 2.8.9 controller/model we hit an Openstack quota issue which caused add-unit to fail with instances stuck in error state.

We removed the machines eg juju remove-machine --force but observed that openstack security groups pertaining to those instances were not removed. This meant after bumping instance quota and trying again we hit the same issue due to being at secgroup quota limit.

It was necessary to remove the security groups manually with openstack security group delete ....

Expected outcome: juju should tidyup after itself wrt secgroups.

[John A Meinel (jameinel)]

Is this happening if you also just 'juju remove-machine' without '--force' ?
What is the underlying error state, is the issue that the machine never came up, or it did come up but then ended up hanging after a while with an error.
Certainly we shouldn't be leaking security groups as a general case. And we should be trying to clean them up even with '--force'. It does mean that if we fail to clean them up, likely '--force' will still progress the removal of the rest of the machine information because you asked us to.

However, if any 'juju remove-machine' leaks a security group, that is definitely something we should be fixing. I also wonder if 'force' makes us progress faster, and that leaves us hitting something like "cannot remove security group because it is still in use".

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]

[Haw Loeung (hloeung)]

We're still seeing this, even with 2.9.21.

We're using --force in this environment that I'm looking at (the prometheus check one). It was recently switched to using --force since the juju controller issues post-upgrade and was leaking resources causing us to manually clean them up. Resources being juju applications, units, and OpenStack instances.

[John A Meinel (jameinel)]

So you are using --force, which says "even if you get errors during cleanup, continue to remove the machine from the model", which is somewhat likely to leave things like security groups behind (if the initial request to clean them up had failed).