[2.7.4] relation-changed events for container-scoped app relation data may fire for primary units before they see their own subordinate

Bug #1866828 reported by Dmitrii Shcherbakov
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
juju
Triaged
High
Unassigned

Bug Description

* Two apps, one primary, one subordinate;
* Both have two units;
* A leader subordinate writes app relation data to a container-scoped relation for all primaries to read;
* The first primary observes the leader subordinate (relation-joined) which then writes some data to the app relation data bag;
* The second primary comes up but gets a relation-changed event for the app relation data before it observes (relation-joined) its own subordinate - relation-list returns no units.

See https://github.com/canonical/operator/issues/175 for more information and outputs.

summary: [2.7.4] relation-changed events for container-scoped app relation data
- may fire for primary units before they see their subordinate
+ may fire for primary units before they see their own subordinate
Revision history for this message
John A Meinel (jameinel) wrote :

I think this changes if you have relation-created, because then you know about the application and application data before you have to know about a single unit.

The potential patch in Juju is attached.

Essentially, you just look at application changes after you look for new units.

Changed in juju:
status: New → Triaged
importance: Undecided → High
Revision history for this message
John A Meinel (jameinel) wrote :

We'll certainly want to talk through whether we want to get the relation-application changed event after relation-created before/after the individual unit joined/changed events.

Revision history for this message
John A Meinel (jameinel) wrote :

Given container scope of the subordinate relationship, I'm curious how you're able to set application level data. (If you are container scope then you are very much talking 1-unit to 1-unit, not app-to-app.)

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

An example that can be used for testing:

https://github.com/dshcherb/charm-apache-httpd#charm-apache-httpd

juju deploy cs:~dmitriis/apache-httpd-0
juju deploy cs:~dmitriis/dummy-vhost-0
juju relate apache-httpd dummy-vhost
juju add-unit apache-httpd

More specifically:

https://github.com/dshcherb/charm-apache-httpd/blob/0e57286b3b9b4304201f5d8e815b874f72b3214d/src/charm.py#L173

https://github.com/dshcherb/charm-dummy-vhost/blob/4875bbb1298645f4c65f7169277066d4e5b83579/src/charm.py#L85-L89

Revision history for this message
John A Meinel (jameinel) wrote : Re: [Bug 1866828] Re: [2.7.4] relation-changed events for container-scoped app relation data may fire for primary units before they see their own subordinate

If I read that correctly:

dummy-vhost is subordinate to apache
each unit of dummy-vhost does need to configure its /var/www/dummy-vhost
data

only the leader of dummy-vhost passes the relation data to apache.

However, is it correct for Apache/1 to have the config from vhost_rdata if
its associated dummy-vhost/1 hasn't actually started up yet? (If
dummy-vhost/1 hasn't had the opportunity to fire install and start yet,
then it doesn't actually have /var/www/dummy-vhost on machine-1 for
apache/1 to serve that data, so it shouldn't (?) be configured with it yet?

Thoughts?

On Mon, Mar 16, 2020 at 2:15 PM Dmitrii Shcherbakov <
<email address hidden>> wrote:

> An example that can be used for testing:
>
> https://github.com/dshcherb/charm-apache-httpd#charm-apache-httpd
>
> juju deploy cs:~dmitriis/apache-httpd-0
> juju deploy cs:~dmitriis/dummy-vhost-0
> juju relate apache-httpd dummy-vhost
> juju add-unit apache-httpd
>
>
> More specifically:
>
> https://github.com/dshcherb/charm-apache-
> httpd/blob/0e57286b3b9b4304201f5d8e815b874f72b3214d/src/charm.py#L173
>
> https://github.com/dshcherb/charm-dummy-
> vhost/blob/4875bbb1298645f4c65f7169277066d4e5b83579/src/charm.py#L85-L89
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1866828
>
> Title:
> [2.7.4] relation-changed events for container-scoped app relation data
> may fire for primary units before they see their own subordinate
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1866828/+subscriptions
>

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

> However, is it correct for Apache/1 to have the config from vhost_rdata if
its associated dummy-vhost/1 hasn't actually started up yet?

I don't think it is. It definitely feels like a use-case for unit data bags with a container-scoped relation. I think this example is useful for demonstration purposes only.

We could just say that this is a valid sequence of events:

1) relation-created;
2) relation-changed (app data);
3) relation-joined;
4) relation-changed (remote unit data).

And then charms and the framework itself would have to handle that situation gracefully for subordinates.

It is quite specific to container-scoped relations because it seems like for global-scoped ones this sequence should always be the case:

1) relation-created;
2) relation-joined;
3) relation-changed (app or unit data).

I may be wrong in my assumption because the relation-created feature isn't there yet.

What do you think?

Revision history for this message
John A Meinel (jameinel) wrote :
Download full text (3.4 KiB)

I do think that you won't typically have application data until the first
unit of an app sets it. Which is why you wouldn't see an early app data
changed. Whether you would see app changed before or after a unit joined
would be ordering, I think. If an app is already deployed and you launch a
new app and relate it. Then relation-created is going to fire on the
running units, while the new machine is still starting.

That doesn't exist today. Which means the unit doesn't know the relation
exists until a unit joins it.

An example probably helps:

Today:

mysql/0 up and running
juju deploy wordpress
juju relate wordpress mysql

* wait for wordpress/0 to start
* relation-joined fires on both wordpress and mysql "simultaneously"
* relation-changed is hard coded to fire immediately after -joined
* each one sees the relation and can potentially set app data or unit data

Future
* after 'juju relate' I would expect relation-created to fire on mysql
while waiting for wordpress/0
* wordpress/0 will also see relation-created before it gets to start
* once start, relation-joined will fire simultaneously for both

at some point you would see relation-changed app due anything set during
the relation-changed event

similarly 'juju add-unit wordpress'

* at some point wordpress/1 will be made aware of the application data from
mysql. should it be before/after relation-joined?

If we go back to the discussion we had today, one interesting feature of
deferring app relation changed is that it could conceivably happen after
you've been told about all the currently running units of app. So code that
doesn't want to spin for each unit could wait until it sees the first app
changed before handling any of those events (just like deferring until
start to know that you've seen most of the things you're related to)

We can never guarantee that you've seen *everything* (concurrency means
that some of them haven't finished starting yet). But you could fast path
a lot of the one by ones until you get there.

John
=:->

On Mon, Mar 16, 2020, 17:15 Dmitrii Shcherbakov <email address hidden>
wrote:

> > However, is it correct for Apache/1 to have the config from vhost_rdata
> if
> its associated dummy-vhost/1 hasn't actually started up yet?
>
> I don't think it is. It definitely feels like a use-case for unit data
> bags with a container-scoped relation. I think this example is useful
> for demonstration purposes only.
>
> We could just say that this is a valid sequence of events:
>
> 1) relation-created;
> 2) relation-changed (app data);
> 3) relation-joined;
> 4) relation-changed (remote unit data).
>
> And then charms and the framework itself would have to handle that
> situation gracefully for subordinates.
>
> It is quite specific to container-scoped relations because it seems like
> for global-scoped ones this sequence should always be the case:
>
> 1) relation-created;
> 2) relation-joined;
> 3) relation-changed (app or unit data).
>
> I may be wrong in my assumption because the relation-created feature
> isn't there yet.
>
> What do you think?
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
>...

Read more...

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

> * at some point wordpress/1 will be made aware of the application data from mysql. should it be before/after relation-joined?

I managed to get a relation-changed for the app data before relation-joined for global-scoped relations as well:

https://github.com/canonical/operator/issues/175#issuecomment-606169657

I suppose we should just retrieve a remote app differently to what we do in the framework today.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.