worker/provisioner: handle stopped/suspended instances

[Dave Cheney]

Definitions (apply as makes sense for the provider):

* shutdown - shutdown/suspended/stunned/temporarily unreachable/rebooting/paused/offline
* dead - terminated/deleted

The provisioner should be able to handle instances (physical/virtual) that are shutdown outside jujus' control. These machines are not terminated, but are coming back at some point in the future. More importantly they can return at any point without warning, so the environment has to be ready to accept them again without damage.

The _why_ someone would do such a thing is outside the scope of this ticket, but it is a use case that Juju should support.

Some things that fall out of this

1. an instance returned from environs.AllInstances() may not be running, consumers of this data will need to cater for this
2. environs.Instance needs to provide a way to indicate its status 'starting/running', or 'paused/offline', maybe this is a filter passed to environs.AllInstance()
3. There is probably some work that charm writers need to do to cater for machines which have relations, but are broken at the moment, this is outside what I can scope.
4. At some point in the future 'smart' charms may invoke add-unit to replace the capacity of their fallen comrade, but this is an addition, not a replacement. This is also outside the scope of this work.

[William Reade (fwereade)]

Why did we not match the python behaviour in the first place? I'm not sure the above proposal is any improvement...

[Dave Cheney (dave-cheney)]

This is not my issue, it was reported by Gustavo, I just transposed it onto LP.

[William Reade (fwereade)]

Closing invalid: what actually happens currently is that the environment only reports active instances, but the provisioner does not take "missing" instances into account and just assumes they're probably ok. The user can observe a machine agent to be "down" via juju status and take action from there, but we shouldn't be trying to be too smart here -- the python feature went unremarked for months and months, and was only ever detected as a bug -- the user terminated an instance out-of-band, and was surprised that juju spun up a new machine to replace (it despite the machine not doing anything).