peer relation events not firing on a particular model

[Tom Haddon]

We have a model we've just deployed on juju 3.1.7 where peer relation events for a particular relation don't seem to be firing:

```
prod-lego-certs-k8s@is-bastion-ps6:~$ juju debug-log --no-tail --replay | grep secret-storage
prod-lego-certs-k8s@is-bastion-ps6:~$ juju ssh httprequest-lego-provider/0
# bash
root@httprequest-lego-provider-0:/var/lib/juju# grep -b2 peers agents/unit-httprequest-lego-provider-0/charm/metadata.yaml
1311- '
1315-name: httprequest-lego-provider
1347:peers:
1354- secret-storage:
1372- interface: secret-storage
root@httprequest-lego-provider-0:/var/lib/juju#
exit
#
```

[Joseph Phillips (manadart)]

Can you post the output of show-application for it?

[Joseph Phillips (manadart)]

And the whole metadata.yaml please. I checked the source for that charm on Github, but the peer relation does not seem to be defined there.

[John A Meinel (jameinel)]

so `juju download httprequest-lego-provider --channel edge` gives us r19 and the metadata.yaml does have:

...

name: httprequest-lego-provider
peers:
  secret-storage:
    interface: secret-storage

...

[Tom Haddon (mthaddon)]

Sure, here's the output of `show-application` and the contents of `metadata.yaml` https://pastebin.ubuntu.com/p/Knv8VvprnG/.

[Caner Derici (cderici)]

Small update with some relevant events from the debug-log:

We get ingress-relation-created and logging-relation-created
unit-httprequest-lego-provider-0: 2024-03-27 08:37:22 DEBUG unit.httprequest-lego-provider/0.juju-log ingress:2: Emitting Juju event ingress_relation_created.
unit-httprequest-lego-provider-0: 2024-03-27 08:37:22 DEBUG unit.httprequest-lego-provider/0.juju-log logging:3: Emitting Juju event logging_relation_created.

Downloading promtail fails because the proxy is not set up:
unit-httprequest-lego-provider-0: 2024-03-27 08:39:33 ERROR unit.httprequest-lego-provider/0.juju-log logging:3: Uncaught exception while in charm code: ......... Connection timed out
unit-httprequest-lego-provider-0: 2024-03-27 08:39:33 ERROR juju.worker.uniter.operation hook "logging-relation-created" (via hook dispatching script: dispatch) failed: exit status 1

The unit is in error state waiting for resolution for this:
unit-httprequest-lego-provider-0: 2024-03-27 08:39:38 INFO juju.worker.uniter awaiting error resolution for "relation-created" hook
unit-httprequest-lego-provider-0: 2024-03-27 08:39:38 DEBUG unit.httprequest-lego-provider/0.juju-log logging:3: ops 2.11.0 up and running.
unit-httprequest-lego-provider-0: 2024-03-27 08:39:38 DEBUG unit.httprequest-lego-provider/0.juju-log logging:3: Emitting Juju event logging_relation_created.
unit-httprequest-lego-provider-0: 2024-03-27 08:41:54 ERROR unit.httprequest-lego-provider/0.juju-log logging:3: Uncaught exception while in charm code: ......... Connection timed out
unit-httprequest-lego-provider-0: 2024-03-27 08:41:54 ERROR juju.worker.uniter.operation hook "logging-relation-created" (via hook dispatching script: dispatch) failed: exit status 1
unit-httprequest-lego-provider-0: 2024-03-27 08:41:54 INFO juju.worker.uniter awaiting error resolution for "relation-created" hook

.......

[Caner Derici (cderici)]

....................... (all for unit-httprequest-lego-provider-0)

Then the proxy is set to model config, which triggers a restart for the pod:
2024-03-27 08:41:55 INFO juju.worker.caasunitterminationworker terminating due to SIGTERM
2024-03-27 08:41:55 INFO juju.worker.uniter unit "httprequest-lego-provider/0" shutting down: agent should be terminated

2024-03-27 08:41:56 INFO juju.worker.uniter unit "httprequest-lego-provider/0" started
2024-03-27 08:41:56 INFO juju.worker.uniter hooks are retried true
2024-03-27 08:41:57 INFO juju.worker.uniter awaiting error resolution for "relation-created" hook
2024-03-27 08:42:02 DEBUG unit.httprequest-lego-provider/0.juju-log logging:3: Emitting Juju event logging_relation_created.

Promtail gets in place:
2024-03-27 08:42:04 INFO unit.httprequest-lego-provider/0.juju-log logging:3: Promtail binary zip file has been downloaded and stored in: /tmp/promtail-static-amd64.gz
2024-03-27 08:42:04 DEBUG unit.httprequest-lego-provider/0.juju-log logging:3: Promtail binary file has been downloaded.
2024-03-27 08:42:05 DEBUG unit.httprequest-lego-provider/0.juju-log logging:3: The promtail binary file has been pushed to the workload container.

We progress to leader elected, things seem to be working:
2024-03-27 08:42:05 INFO juju.worker.uniter.operation ran "logging-relation-created" hook (via hook dispatching script: dispatch)
2024-03-27 08:42:05 INFO juju.worker.uniter found queued "leader-elected" hook
2024-03-27 08:42:06 DEBUG unit.httprequest-lego-provider/0.juju-log ops 2.11.0 up and running.
2024-03-27 08:42:06 DEBUG unit.httprequest-lego-provider/0.juju-log Emitting Juju event leader_elected.
2024-03-27 08:42:06 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch)

2024-03-27 08:42:07 WARNING unit.httprequest-lego-provider/0.juju-log Incorrect relation data from the data provider: {'endpoints': '10.141.66.76:5432'}

2024-03-27 08:42:09 DEBUG unit.httprequest-lego-provider/0.juju-log Legacy hooks/start exited with status 0.
2024-03-27 08:42:09 DEBUG unit.httprequest-lego-provider/0.juju-log Emitting Juju event start.
2024-03-27 08:42:10 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-changed" hook (via hook dispatching script: dispatch)
2024-03-27 08:42:11 INFO juju.worker.uniter.operation ran "postgresql-relation-changed" hook (via hook dispatching script: dispatch)
2024-03-27 08:42:11 INFO juju.worker.uniter.operation ran "postgresql-relation-joined" hook (via hook dispatching script: dispatch)
2024-03-27 08:42:12 INFO juju.worker.uniter.operation ran "ingress-relation-joined" hook (via hook dispatching script: dispatch)
2024-03-27 08:42:13 INFO juju.worker.uniter.operation ran "logging-relation-changed" hook (via hook dispatching script: dispatch)

[Tom Haddon (mthaddon)]

I tried to create a small charm to test, but I wasn't able to reproduce the issue. The charm is available at https://github.com/mthaddon/peer-relation-error-test. The plan was to have a peer relation and another relation, have the other relation go into error. However, the peer relation fired even though the charm went into error on the other relation. In any case, hopefully this test charm can be adapted to try and reproduce the issue once we know more about it.