kill-controller fails with "ERROR getting controller environ: unable to get bootstrap information from client store or API" when controller is no longer running (on OpenStack)

[Jason Hobbs]

I'm trying to kill a controller that shows up in my controllers list, but juju fails to do so:

http://paste.ubuntu.com/24086765/

It's an OpenStack controller, and in this case, the machines that were running OpenStack are no longer around, so juju can't reach the controller or OpenStack at all. It's easy to reproduce by adding an openstack controller in your controllers.yaml that has IPs that can't be reached.

We rely on kill-controller always working and if it doesn't, it hangs up or automation.

Here is my controllers.yaml:
https://pastebin.canonical.com/180985/

This is with 2.1.0-0ubuntu1~16.04.1~juju1

[Anastasia (anastasia-macmood)]

@Jason Hobbs,
Could you please attach 'juju status --format=yaml' for that controller? [ci-oil-slave4-openstack-RegionOne]

As well as controller logs?

Anything in addition you can share about this openstack? mitaka? v3 auth?...

Thank you for the report!

[Jason Hobbs (jason-hobbs)]

Anastasia,

juju status just hangs because the controller is no longer around, and there are no controller logs for the same reason. It appears the client errors out when it can't reach the controller, instead of continuing and removing the local information about the controller.

[Tim Penhey (thumper)]

If running in a terminal, kill-controller should ask whether to continue when it gets an error attempting to deal with the killing of the environment directly.

A command line option should also be able to be specified to effectively skip the ask and just continue (like the --yes), perhaps something like --clean-broken.

[Jason Hobbs (jason-hobbs)]

Tim, this breaks our CI automation so it is a high priority for us to get this fixed.

[John A Meinel (jameinel)]

I think we want kill-controller to exit non-zero if it can't actually find something to kill. With human interaction, it is reasonable to ask a human to validate their assumptions and make sure they know what they're doing with the controller.

For scripting, you should be able to do:

  juju kill-controller -y foo || juju unregister -y foo

Which falls back to just removing the controller from your local registry no matter what.

[Jason Hobbs (jason-hobbs)]

Ok I agree - unregister is fine.