[RFE] Impossible to cleanly remove a unit from a relation

[Stuart Bishop]

A relation-departed hook cannot be used by a charm to perform cleanup, as the remote service may have already run its relation-departed hook and revoked access. From the documentation, "this should be used to remove all references to the remote unit, because there's no guarantee that it's still part of the system".

The situation is worse for a peer relation. In addition to the above catch-22, the unit running the relation-departed hook has no idea if it is the unit leaving the service or if it is the remote unit leaving the service.

So as a concrete example, it is impossible for the Cassandra charm to automatically decommission a node before it is removed. The peer-relation-departed hook cannot decommission the node because the charm has no idea which unit is actually being dropped. And even if it did, the decommissioning process would fail as it takes time and the other units in the cluster will have revoked its access before it completed. Instead, the operator is required to manually decommission nodes before dropping the unit. Failing to do this requires lengthy cleanup operations, and data stored at replication factor 1 will be lost.

Before the relation-departed hooks are run, another hook needs to be run on the departing unit to provide it with the opportunity it needs. relation-departing seems the obvious choice.

[Mario Splivalo (mariosplivalo)]

Hi.

This is also issue for the percona-cluster charm which can't safely remove the unit - it should shut down mysql prior removing the unit but it can't do so as there is no way to tell on which endpoint of the relation the -departed hook is running (if it's on the unit to be removed then mysql should be stopped, but not on the other one!).

Introducing a hook that would fire before -departed, only on unit that is about to depart relation, would solve this issue.
(Here is the percona-cluster related bug: https://bugs.launchpad.net/charms/+source/percona-cluster/+bug/1514472)

[Cheryl Jennings (cherylj)]

We're tracking this request for consideration in future development cycles.

Copying some discussion from the mailing list to the bug (from axw):
Comment from the sidelines: we have something similar in storage now, with "storage-detaching" hook. This runs before storage is detached, so that charms can stop using the storage before it's ripped out beneath them.

With that in mind, can we please call this "-relation-departing"?

[Stuart Bishop (stub)]

Should a network partition between the controller and the departing node block the unit's departure until the -relation-departing hook can be run? 'Yes' would be a good answer if 'destroy-unit --force' will force the issue and skip running the -relation-departing hook if necessary.

[Mario Splivalo (mariosplivalo)]

Hello, Anastasia.

Is there another mechanism inside juju that will allow for clean removing of a unit - something that would provide similar means as wanted '-relation-departing' hook?

[Anastasia (anastasia-macmood)]

@Mario Splivalo,
I have closed this bug as "juju-core" project is only tracking Juju 1.x. Our current Juju 1 release is 1.25 and it is only open for Critical bugs.

I will add this as wish list item to our Juju 2 launchpad project!

Thank you for your feedback!

[Gabriel Samfira (gabriel-samfira)]

This has creped up in one of our use cases as well. Essentially, any charm that deploys a cluster will have this issue at some point.

Just adding my +1 to this being resolved.

[Marco Ceppi (marcoceppi)]

I think this could be solved by simply having a new environment variable based to denote lifecycle. One possible solution is a `JUJU_UNIT_DYING` where only set when the unit is on it's way out. Another possibility is a `JUJU_UNIT_LIFECYCLE` environment variable which has an "alive" and "dying" value where the opportunity to add more LIFECYCLE labels

[Stuart Bishop (stub)]

An environment variable detailing the lifecycle would solve a related issue (the second paragraph in the original bug report), and would certainly help or solve many situations.

However, this particular issue is that when a unit's -departed hooks are run it may find that the related units have already departed the relation and have cut off all access. A Cassandra node has no opportunity to decommission itself cleanly, because it is no longer able to communicate with the rest of the cluster and migrate its data to the remaining nodes. Its particularly important if data is being stored without redundancy (replication factor == 1), because in this case the data is lost. Without the extra hook, removing a node from a cluster is the same as a failure and requires the cluster to be repaired. With the extra hook, the node may decommission itself cleanly and we don't need to repair the cluster and we never have a period of time with reduced data redundancy.

[John A Meinel (jameinel)]

this feels like bug #1417874

[Anastasia (anastasia-macmood)]

@John,
It certainly is :D

[Tim Penhey (thumper)]

To be honest, I don't think adding synchronisation of hooks across units is something we are likely to add, and this is what is really needed for the clean removal of a clustered unit.

Instead, I think this would be much more suited to an action. One that explicitly removes one unit from the cluster, and that this action should be run and completed before the removal of the unit.

[Mario Splivalo (mariosplivalo)]

The problem would still remain: when unit is being removed, juju fires the -departed hook, but we don't know, when inside the hook, is the hook running on the unit that's being removed, or on the unit that's remaining.

If you have two units deployed, say, percona-cluster. When you remove one of the units (say, percona-cluster/0) here is what happens:

1a. -relation-departed hook is run in percona-cluster/0
1b. -relation-departed hook is run in percona-cluster/1

2.-relation-broken hook is run in percona-cluster/0

Now, this creates the issue. When -relation-departed hook is run on both units there is no way for the hook code to know if it's running on the unit that's parting or on the unit that's remaining. Therefore it can't politely stop mysqld service. Currently if you just 'juju remove-unit', and get rid of percona-cluster/1, percona-cluster/0's mysqld will switch to 'degraded' state and will not allow querying data.

A "-relation-departing" hook would solve that issue (or the environment variable, or any other mechanism that would allow hook that is being run to know if it's run on departing unit or not).

(In this particular example with percona-cluster the workaround is quite simple - operator would, prior issuing 'juju remove-unit', ssh into the unit that is to be removed and manually issue a clean shutdown of mysql service. After mysql stops there operator can utilize 'juju remove-unit' to get rid of the unit completely. However, having a separate hook would, imho, greatly simplify unit removal).

[Dmitrii Shcherbakov (dmitriis)]

Haven't noticed this bug. Here is the doc bug I created about similar matters https://github.com/juju/docs/issues/2357

[Achilleas Anagnostopoulos (achilleasa)]

PR https://github.com/juju/juju/pull/11356 targets the develop branch and exposes the departing unit when invoking xyz-relation-departed hooks via a new envvar called JUJU_DEPARTING_UNIT. Charms can compare this value to JUJU_REMOTE_UNIT to see if they are the ones going away.