enable-ha should not promote dying machines

[John A Meinel]

Now that you can remove-machine, we should make sure 'juju enable-ha' doesn't try to promote a machine that it thinks of as dying.

Specifically, I wanted to make sure you can get rid of controller machines that fail to start. And doing:

juju bootstrap lxd
juju enable-ha
lxc stop juju-XXXXX-1 # before it finishes starting
juju status
juju show-controller

can see that 0 and 2 want to participate, but 2 isn't ha-enabled because there isn't a 3rd voter.
And you are able to:
juju remove-machine 1

And it transitions out of being part of the output of "juju show-controller".

Note that machine 1 does not disappear entirely, because ultimately it is the Machiner on machine 1's job to call EnsureDead. Since the machine never runs, it can't ack that it is ready to go away.

But calling
$ juju enable-ha -n3
maintaining machines: 0, 2
promoting machines: 1

Is trying to restore the machine that we already flagged for death.

We can work around this with

$ juju remove-machine --force

Will move machine 1 all the way to dead and then get reaped. And even kill the container in LXD. And then 'juju enable-ha -n3' does the right thing:
$ juju enable-ha -n3
maintaining machines: 0, 2
adding machines: 3

I have the feeling that 'promoting machine' is actually a poor action now. We don't have machines sitting around that don't want to vote. Either they want to actively participate but can't yet, or are dying and on the way out.

Note that there is a workaround.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.