ensure-availability can add more state servers while ones it just started haven't come up yet

[Roger Peppe]

I was just playing with HA, and saw the following behaviour.
Before running the commands, I'd had an environment with
3 environment managers (0, 3 and 4), of which I had destroyed the instance
of 0. I was expecting it to remove the EnvironManager status
of that one (machine 0) but definitely did not expect
to see it add two more machines... (6 and 7)

http://paste.ubuntu.com/7252280/

(Note that I'd tricked out juju status with the ability to show voting status)

[Andrew Wilkins (axwalk)]

I don't think there's a bug here; note that machine agents 3 and 4 are down after the ensure-availability call, and that their wants-vote=false.

Unless there really is a bug, those machine agents were down when you called ensure-availability, and it did the right thing and took away their voting rights and created two replacement machines.

Having said that, this isn't ideal. Machines 3 and 4 were down temporarily; it would be good to (a) have ensure-availability say what it has done (and why), and (b) improve the availability heuristic. Perhaps we need an uptime for agents as well as current availability.

[John A Meinel (jameinel)]

So I think we do have a bug here that needs addressing. If you do:

juju bootstrap
...
juju ensure-availability && juju ensure-availability

The first one will try to spin up 2 new machines to become state servers. Immediately running ensure-availability again will say "I want to have 3 available state servers" and will notice that 2 of them are down.

I think we need a way to say "I just kicked off 2 machines and I'm expecting them to come up soon, don't spin up 2 more until these have been given a chance".
I don't know what that timeout is, but otherwise we just give users a big gun to shoot themselves in the foot if they run "juju ensure-availability" too many times in a row.