juju emitting `relation_broken` while there are remaining units in the relation

[Paulo Machado]

Juju version: 2.9.32 (initially detected on 2.9.29)
LXD cloud

Given an app A with 3 units related to and app B,
when scaling in app A with `juju remove-unit A/0`
the unit A/0 will receive a `relation_broken` event.

This behavior seems to contradict the documentation,
where is said the event will be triggered:
> once all the units have been removed
since app A is active with two units and still related
to app B.

This tests where done against in development mysql-operator
(https://github.com/canonical/mysql-operator/tree/feature/shared_db_refactor)
using `shared-db` relation.

[Juan M. Tirado (tiradojm)]

Could you please point to the section of the documentation where this is described?

[Paulo Machado (paulomachado)]

Sure!

https://juju.is/docs/sdk/relations#heading--event-types

at the Description of `<name>_relation_broken`

[Joseph Phillips (manadart)]

This is not accidental behaviour - it is quite explicit in the logic, that a dying unit gets a relation-broken hook invocation for relations it is participating in.

We need to update the documentation to this effect.

This behaviour is long-standing. I've traced it as far back as 8 years ago, so the potential for breakage in the ecosystem from changing this is significant.

[Carl Csaposs (carlcsaposs)]

Would it be possible to add information to the event to determine if the relation is actually breaking?

Many of our charms need to perform cleanup (e.g. delete a user) if the relation is breaking. To do so currently, each of our charms has to add a workaround that stores state during the relation departed event if the departing unit == unit and then check that state during the relation breaking event. Example: https://github.com/canonical/postgresql-operator/blob/e13b871022bb6cff44cfb3ca83427c0e9d03f606/src/relations/db.py#L193

[Joseph Phillips (manadart)]

Deliberate behaviour and work-arounds aside, I am of the view that a departing unit should *not* actually get a relation broken event.

The life-cycle of the relation itself is an application level concern, and a departing unit should not have the option of acting on that concern if it was not affected at the application level.

[Carl Csaposs (carlcsaposs)]

Related bug report (no workaround possible) for subordinate charms: https://bugs.launchpad.net/juju/+bug/2025676

[Ian Booth (wallyworld)]

If you had asked me before this bug was raised, I would have said the juju behaviour would have been to only run relation broken after all counterpart units had left the relation scope. It seems that's not the case and to me that's a bug. Furthermore, the hook behaviour was implemented before we had unit leadership. And I do agree that a broken relation is an application level concern, so the relation broken hook should I think only run on the leader unit.

Regardless of compatibility concerns, if we agree the current behaviour is broken, we should fix it instead of making the problem worse by doing some work around. We can always make it opt in using a charm "assumes" directive and make it default in juju 4.

[John A Meinel (jameinel)]

All units get 'relation-created' so I think all units should get 'relation-broken'. But I do understand the concern around "the relation isn't really going away, you're just being removed"

[Paulo Machado (paulomachado)]

Does a unit receives a 'relation-created' if the said unit is created after the relation was established or just pre existing units will receive?

[Carl Csaposs (carlcsaposs)]

Yes, a unit receives relation-created if the unit starts after the relation is established.

This is helpful, for example, with the mysql <-> tls-certificates-operator relation. Each unit needs its own TLS certificate, so each unit requests a certificate on relation-created.

But for that example, getting a relation-broken event during single unit teardown as opposed to a stop or remove event isn't necessary. Off the top of my head, I can't think of an example where getting a relation-broken event during single unit teardown would be helpful

[Paulo Machado (paulomachado)]

Is it feasible to include a new field in the relation-broken with the reasoning for the break?
So when ops render the event sent to the charm, it's possible to know that the relation is breaking for the unit only.
This would keep the created-broken symmetry and delegate any special treatment to the charm.

[Mia Altieri (miaaltieri)]

Is it feasible to include a new field in the relation-broken with the reasoning for the break?
So when ops render the event sent to the charm, it's possible to know that the relation is breaking for the unit only.
This would keep the created-broken symmetry and delegate any special treatment to the charm.

^ this would be very helpful for my own charm as well

[Teodora Mihoc (tmihoc)]

Updated the doc to clarify.