Inability to add cross model relation

[Steven Parker]

We have two models.

  lma -- offers graylog as cross model relation
  snap proxy - holds filebeat and consumes lma's graylog offer

Adding the relation from snap-store to lma for graylog <->filebeats does not result in the services configuring correctly via the charms.

--

SNAP PROXY MODEL ADDING RELATION TO GRAYLOG OFFER FROM LMA MODEL

juju status --relations

Model Controller Cloud/Region Version SLA Timestamp
snap-store-proxy foundations-maas maas_cloud 2.9.18 unsupported 02:00:03Z

SAAS Status Store URL
graylog active foundations-maas admin/lma.graylog-beats
nagios terminated foundations-maas admin/lma.nagios-monitors
...
App Version Status Scale Charm Store Channel Rev OS Message
filebeat 5.6.16 waiting 4 filebeat charmstore stable 24 ubuntu Waiting for: elasticsearch, logstash or kafka.
...
  filebeat/0* waiting idle 10.145.253.36 Waiting for: elasticsearch, logstash or kafka.

---
juju add-relation filebeat graylog <<--- adding relation here
---
sleep 120 Actually same status after 10 minutes
----
juju status --relations

Model Controller Cloud/Region Version SLA Timestamp
snap-store-proxy foundations-maas maas_cloud 2.9.18 unsupported 02:02:47Z

SAAS Status Store URL
graylog active foundations-maas admin/lma.graylog-beats
nagios terminated foundations-maas admin/lma.nagios-monitors
...
filebeat 5.6.16 waiting 4 filebeat charmstore stable 24 ubuntu Waiting for: elasticsearch, logstash or kafka.
...
graylog:beats filebeat:logstash elastic-beats regular

---
relation shows up but filebeat does not link up with graylog correctly.
Still in state:
     filebeat/1 waiting idle 10.145.253.37 Waiting for: elasticsearch, logstash or kafka.
---
LMA MODEL PROVIDING GRAYLOG OFFER

juju status -m lma --relations
Model Controller Cloud/Region Version SLA Timestamp
lma foundations-maas maas_cloud 2.9.18 unsupported 02:12:53Z
...
elasticsearch 5.6.16 active 3 elasticsearch charmstore stable 51 ubuntu Unit is ready

...
Offer Application Charm Rev Connected Endpoint Interface Role
grafana grafana grafana 51 3/3 dashboards grafana-dashboard requirer
graylog-beats graylog graylog 50 1/1 beats elastic-beats provider <<<<--- 1/1 relations ?

-----
Think we should see more relations here as snapstore model has 4 filebeat units

 juju offers -m lma
Offer User Relation id Status Endpoint Interface Role Ingress subnets
grafana admin 219 joined dashboards grafana-dashboard requirer
                   admin 223 joined dashboards grafana-dashboard requirer
                   admin 224 joined dashboards grafana-dashboard requirer
graylog-beats admin 227 joined beats elastic-beats provider 10.145.253.37/32,10.145.253.38/32,10.145.253.39/32,10.145.253.36/32

[Steven Parker (sbparke)]

This bug occurred on the same juju controller as bug https://bugs.launchpad.net/juju/+bug/1952796

Relevant artifacts for this bug...

Juju controller SOS reports:
https://private-fileshare.canonical.com/~sbparke/sosreport-juju-1-Juju-CMR-Controller-0-2021-11-30-npxesvi.tar.xz
https://private-fileshare.canonical.com/~sbparke/sosreport-juju-3-Juju-CMR-Controller-1-2021-11-30-olgynuc.tar.xz
https://private-fileshare.canonical.com/~sbparke/sosreport-juju-2-Juju-CMR-Controller-2-2021-11-30-lahomhg.tar.xz

Crash Dumps for snap-store and LMA models:
https://private-fileshare.canonical.com/~sbparke/juju-crashdump-59a7cfdd-f754-4641-aebd-26c072ca29eb-lmaModel.tar.xz
https://private-fileshare.canonical.com/~sbparke/juju-crashdump-29162c9a-5244-45b6-ab1e-07927bd614b8-snap-storeModel.tar.xz

Controller database:
https://private-fileshare.canonical.com/~sbparke/sanitizedMongoDB.tgz

[Ian Booth (wallyworld)]

Quick comment on the relation count

Offer Application Charm Rev Connected Endpoint Interface Role
grafana grafana grafana 51 3/3 dashboards grafana-dashboard requirer
graylog-beats graylog graylog 50 1/1 beats elastic-beats provider <<<<--- 1/1 relations ?

When you relate a consuming application to the offer, there is only the one relation (aka connection) displayed; this is independent of how many units may be involved. It's no different to in-model relations in "juju status --relations" - only one item is shown per relation even if many units are involved.

Depending on what the exact issue is, it could be related to exactly what relation data is passed between the offering and consuming charm. We've seen in the past that charms can select an inappropriate address for example. We'd need to know exactly what's misconfigured, how it's misconfigured, where the config comes from, what's missing or incorrect etc etc to narrow down is it a juju issue, is it a charm issue etc etc.

[Steven Parker (sbparke)]

Ok that's good to know.

We are rebuilding this deployment but I will try and replicate in the lab and provide the requested data.

[Steven Parker (sbparke)]

Ok I have been able to replicate this in the lab with nagios and nrpe cross model relations.

The proper hook sequence seems to be:
  relation-created
  relation-joined
  relation-changed

It seems that relation-joined is not being fired for cross model relations when deleting and adding relations after deployment. It does seem to fire on initial deployment of the charms.

Below you can see the deployment, followed by the remove-relation and add-relation actions.

Here is an nrpe cross model relation remove/add action:

fgrep relation unit-nrpe-0.log.back.CrossModel
2021-12-05 04:55:14 INFO juju.worker.uniter.operation runhook.go:155 skipped "general-info-relation-created" hook (missing)
2021-12-05 04:55:14 INFO juju.worker.uniter.operation runhook.go:155 skipped "monitors-relation-created" hook (missing)
2021-12-05 04:55:31 INFO juju.worker.uniter.operation runhook.go:152 ran "monitors-relation-joined" hook (via explicit, bespoke hook script)
2021-12-05 04:55:36 INFO juju.worker.uniter.operation runhook.go:152 ran "monitors-relation-changed" hook (via explicit, bespoke hook script)
2021-12-05 04:55:42 INFO juju.worker.uniter.operation runhook.go:152 ran "general-info-relation-joined" hook (via explicit, bespoke hook script)
2021-12-05 04:55:47 INFO juju.worker.uniter.operation runhook.go:152 ran "general-info-relation-changed" hook (via explicit, bespoke hook script)
2021-12-05 04:56:57 INFO juju.worker.uniter.operation runhook.go:155 skipped "monitors-relation-departed" hook (missing)
2021-12-05 04:56:58 INFO juju.worker.uniter.operation runhook.go:155 skipped "monitors-relation-broken" hook (missing)
2021-12-05 04:57:45 INFO juju.worker.uniter.operation runhook.go:155 skipped "monitors-relation-created" hook (missing)

--- Missing relation-joined hook ---

Here is the same action within the same model:

fgrep relation unit-nrpe-0.log.back.SameModel | fgrep -v WARNING
2021-12-05 06:34:39 INFO juju.worker.uniter.operation runhook.go:155 skipped "nrpe-external-master-relation-created" hook (missing)
2021-12-05 06:35:04 INFO juju.worker.uniter.operation runhook.go:152 ran "nrpe-external-master-relation-joined" hook (via explicit, bespoke hook script)
2021-12-05 06:35:14 INFO juju.worker.uniter.operation runhook.go:152 ran "nrpe-external-master-relation-changed" hook (via explicit, bespoke hook script)
2021-12-05 06:35:58 INFO juju.worker.uniter.operation runhook.go:155 skipped "monitors-relation-created" hook (missing)
2021-12-05 06:36:05 INFO juju.worker.uniter.operation runhook.go:152 ran "monitors-relation-joined" hook (via explicit, bespoke hook script)
2021-12-05 06:36:13 INFO juju.worker.uniter.operation runhook.go:152 ran "monitors-relation-changed" hook (via explicit, bespoke hook script)
2021-12-05 06:38:01 INFO juju.worker.uniter.operation runhook.go:155 skipped "monitors-relation-departed" hook (missing)
2021-12-05 06:38:04 INFO juju.worker.uniter.operation runhook.go:155 skipped "monitors-relation-broken" hook (missing)
2021-12-05 06:39:14 INFO juju.worker.uniter.operation runhook.go:155 skipped "monitors-relation-created" hook (missing)
2021-12-05 06:39:21 INFO juju.worker.uniter.operation runhook.go:152 ran "monitors-relation-joi...

[Steven Parker (sbparke)]

I have replicated this bug with nrpe/nagios and graylog/filebeats. I do not think this is an issue with charm code.
It seems that juju is not triggering the relation-joined hook for cross model relations.

Please find more files for reference under private-fileshare.canonical.com:~sbparke/Bug-1952796-dump.tgz

tar tf Bug-1952796-dump.tgz
dump/
dump/nrpe-test/
dump/nrpe-test/unit-nrpe-0.log.back.SameModel <--- nrpe logs from remove/add relation in same model
dump/nrpe-test/client.yaml <--- client bundle
dump/nrpe-test/unit-nrpe-0.log.back.CrossModel <--- remove/add relation in cross model
dump/nrpe-test/lma.yaml <-- lma bundle

dump/filebeat-test/
dump/filebeat-test/client.yaml <<-- client bundle
dump/filebeat-test/unit-graylog-0.log.backup <--- graylog unit logs
dump/filebeat-test/controller-logsink.log.back <-- juju controller
dump/filebeat-test/unit-filebeat-0.log.backup <--- filebeat unit
dump/filebeat-test/ControllerDBDump.yaml <<-- juju controller DB dump
dump/filebeat-test/controller-machine-0.log.back
dump/filebeat-test/graylog.yaml <<--- lma bundle
dump/filebeat-test/unit-graylog-1.log.back
dump/filebeat-test/graylog-machine-1.log.back
dump/filebeat-test/unit-graylog-2.log.back

What I'm seeing is the lack of the relation-joined hooks being fired when adding and removing cross model relations. The hook does seem to fire within the same model.

Hopefully you can deploy the client and lma bundles and then remove and add cross model relations to see the issue in your lab.

[Steven Parker (sbparke)]

Please replicate with the following commands:

From the files under dump/filebeat-test provided above you will find the bundles used below

Steps:

# deploy graylog
juju deploy ./graylog.yaml

# wait for model to settle

# deploy the client bundle
juju deploy ./client.yaml

# wait for model to settle
# filebeat should be in ready state

#remove
juju remove-relation filebeat graylog

# wait for model to settle
# filebeat will go into waiting state

# add relation
juju add-relation filebeat graylog

# filebeat never comes out of wait state
# tail logs for filebeat agent

[Ian Booth (wallyworld)]

This PR allows the steps above to complete successfully. Tested on AWS.

https://github.com/juju/juju/pull/13559