python_libjuju destroy model with --force doesn't clean-up instances when using openstack provider

[Alex Kavanagh]

When destroy models (particularly when using --force) on the openstack provider, juju leaves instances running / doesn't clean up after itself.

This is so repeatable that we are implementing a work-around in zaza: https://github.com/openstack-charmers/zaza/pull/411
(It reaps instances on - almost - every single CI run).

We think this might be due to errored units or something similar. We don't really care whether a unit errors out as the model is being destroyed. We'd really like a --just-kill-it type option which literally just removes the instances / storage / network ports, and doesn't try to run any hooks to tear down the model.

[Alex Kavanagh (ajkavanagh)]

As an additional note, I've noticed that the problem might be that if "--destroy-storage" (and the equivalent to python_libjuju destroy_model()) isn't used then the instances don't get reaped. If it is used, then the instances are reaped. This isn't definitive, just an observation.

[Ian Booth (wallyworld)]

The --force option will still see Juju try to destroy machines/units etc cleanly, running hooks etc, but will fall back to a "remove all the things anyway" scenario after a timeout.

If you really do want to destroy everything, then --destroy-storage is definitely recommended, otherwise Juju will attempt to detach storage (if possible) from any machine it was mounted on, and retain that storage for subsequent import into a different model so it can be reused.

[Alex Kavanagh (ajkavanagh)]

> If you really do want to destroy everything, then --destroy-storage is definitely recommended, otherwise Juju will attempt to detach storage (if possible) from any machine it was mounted on, and retain that storage for subsequent import into a different model so it can be reused.

Ah, a little light went on in my head; on our CI we've had loads of "orphaned" volumes that we manually clean up, and we weren't using the --destroy-storage option (via libjuju) in zaza. This is much improved now, so that explains that scenario that we were seeing.

We have now added that as the default (along with --force). We've added a 'reaper' to clean up any remaining instances and (anecdotally) it seems improved, but I've had a couple of reports where the reaping code has cleaned up a few instances.

One other oddity is that it seems the libjuju destroy call can return before all the instances have been reaped. It may be that OpenStack is returning before the instance is reaped. Again, our patch works around this as they then (mostly) disappear asynchronously after the call has completed.

It would still be nice to have a 'kill' which doesn't even try to run hooks, though.

[John A Meinel (jameinel)]

I think there are 2 issues at play here:
1) Destroy as an API call has never been blocking (the request is telling the controller that it should tear down, not wait for it to clean up). I do believe there is a request that you can do to monitor the model to see it tearing down (since that is what the Juju client uses). So for pylibjuju we would want to add that support.

2) Being able to tear down everything in a model without worrying about unit hooks. I'd rather split that out into something else.

[Canonical Juju QA Bot (juju-qa-bot)]

This Medium-priority bug has not been updated in 60 days, so we're marking it Low importance. If you believe this is incorrect, please update the importance.