unit-remains in goal state after being flagged for removal

[Stuart Bishop]

Juju units may seem themselves in goal state, even after they have been flagged as removal. The unit being able to tell that it is doomed is my prime use case for charm goal state.

With postgresql/0 currently in an error state:

$ juju run --unit=postgresql/0 -- goal-state --format=yaml
units:
  postgresql/0:
    status: maintenance
    since: 2018-06-20 07:42:58Z
relations: {}

$ juju add-unit postgresql

$ juju run --unit=postgresql/0 -- goal-state --format=yaml
units:
  postgresql/0:
    status: maintenance
    since: 2018-06-20 07:42:58Z
  postgresql/1:
    status: waiting
    since: 2018-06-20 09:00:11Z
relations: {}

$ juju remove-unit postgresql/0
removing unit postgresql/0

$ juju run --unit=postgresql/0 -- goal-state --format=yaml
units:
  postgresql/0:
    status: maintenance
    since: 2018-06-20 07:42:58Z
  postgresql/1:
    status: waiting
    since: 2018-06-20 09:00:11Z
relations: {}

Running 'juju debug-hooks postgresql/0' and resolving postgresql/0, I see that postgresql/0 remains in goal state throughout install & leader-settings-changed, at which point it disappears and never knew that it was dying.

Similarly, in a separate test during a 'replication-relation-departed' hook (the peer relation) after destroying postgresql/1, unit postgresql/2 sees both postgresql/1 and postgresql/2 in the goal state, despite the fact that the peer relation is being departed means that one or the other is being destroyed. This is exactly the situation I need to know which unit is being destroyed and which unit is going to remain. The postgresql/1 unit similarly saw both units in the goal state throughout its tear down, never knowing that it was being torn down (and even remained leader throughout, making detrimental decisions about the cluster from a position of ignorance, which is Bug #1532085)

I have no use cases for goal-state unless this issue can be fixed, by which I mean that units should never appear in goal-state once the remove-unit or remove-application command has been accepted.

[Ian Booth (wallyworld)]

To confirm the fix here - the "status" of a unit will show as "dying" once remove-application has been run and the unit transitions to running its departed hooks etc so it can leave scope. Until then, while the unit is still alive, "status" will show the workload status, "maintenance", "waiting", "alive" etc. And when the unit becomes "dead" after all hooks have been run and it has left scope, it won't show up at all.

[Stuart Bishop (stub)]

https://github.com/juju/charm-helpers/blob/cf2dd13a04a712378358b80e2e3ff48ec9385552/charmhelpers/core/hookenv.py#L1389 is the charm-helpers wrapper around this functionality.