The remote end of a cross-model relation fails intermittently. The workload of that unit continues to function in isolation, but events are no longer propagated to the charm. Once this happens, it may be resolved using the following workaround:
As this is an intermittent error I'm not really able to reproduce it in a controlled fashion, but I've attached a dump of my terminal from last time it happened.
The remote end of a cross-model relation fails intermittently. The workload of that unit continues to function in isolation, but events are no longer propagated to the charm. Once this happens, it may be resolved using the following workaround:
```
juju remove-saas someoffer
juju remove-offer -m lma someoffer --force # not always needed but doesn't hurt
juju offer ...
juju consume
juju relate ...
```
As this is an intermittent error I'm not really able to reproduce it in a controlled fashion, but I've attached a dump of my terminal from last time it happened.
```
❯ juju version
2.9.22-ubuntu-amd64
```
Running on microk8s.
```
❯ juju remove-relation spring-music prometheus
❯ juju debug-log -m spring music-0: 19:53:22 INFO juju.worker. uniter. operation ran "metrics- endpoint- relation- created" hook (via hook dispatching script: dispatch) music-0: 19:57:41 INFO juju.util.exec run result: exit status 1 music-scrape- config- 0: 19:58:37 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-0: 19:58:38 INFO juju.worker. uniter. operation ran "metrics- endpoint- relation- broken" hook (via hook dispatching script: dispatch) music-0: 19:58:39 INFO juju.worker. uniter. operation ran "metrics- endpoint- relation- created" hook (via hook dispatching script: dispatch) music-0: 19:58:58 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-0: 20:03:56 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-scrape- config- 0: 20:04:10 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-scrape- config- 0: 20:09:20 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-0: 20:09:50 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-0: 20:10:48 INFO juju.worker. uniter. operation ran "metrics- endpoint- relation- broken" hook (via hook dispatching script: dispatch)
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
^C
❯ juju add-relation spring-music prometheus
❯ juju debug-log -m spring music-0: 19:57:41 INFO juju.util.exec run result: exit status 1 music-scrape- config- 0: 19:58:37 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-0: 19:58:38 INFO juju.worker. uniter. operation ran "metrics- endpoint- relation- broken" hook (via hook dispatching script: dispatch) music-0: 19:58:39 INFO juju.worker. uniter. operation ran "metrics- endpoint- relation- created" hook (via hook dispatching script: dispatch) music-0: 19:58:58 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-0: 20:03:56 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-scrape- config- 0: 20:04:10 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-scrape- config- 0: 20:09:20 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-0: 20:09:50 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-0: 20:10:48 INFO juju.worker. uniter. operation ran "metrics- endpoint- relation- broken" hook (via hook dispatching script: dispatch) music-0: 20:11:07 INFO juju.worker. uniter. operation ran "metrics- endpoint- relation- created" hook (via hook dispatching script: dispatch)
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
^C
❯ juju debug-log -m controller connection agent disconnected: unit-prometheus-1 for bccb7cc3- a40a-474e- 83c3-20b8f0b09a 80 connection agent login: unit-prometheus-1 for bccb7cc3- a40a-474e- 83c3-20b8f0b09a 80 common setting password for "application- prometheus" uniter resolving "": lookup : no such host uniter resolving "": lookup : no such host connection agent login: application- prometheus for bccb7cc3- a40a-474e- 83c3-20b8f0b09a 80 connection agent disconnected: application- prometheus for bccb7cc3- a40a-474e- 83c3-20b8f0b09a 80 connection agent login: unit-prometheus-0 for bccb7cc3- a40a-474e- 83c3-20b8f0b09a 80 common error stopping *apiserver. pingTimeout resource: ping timeout connection agent disconnected: unit-prometheus-0 for bccb7cc3- a40a-474e- 83c3-20b8f0b09a 80
controller-0: 20:07:47 INFO juju.apiserver.
controller-0: 20:07:48 INFO juju.apiserver.
controller-0: 20:07:50 INFO juju.apiserver.
controller-0: 20:07:54 ERROR juju.apiserver.
controller-0: 20:07:55 ERROR juju.apiserver.
controller-0: 20:08:02 INFO juju.apiserver.
controller-0: 20:08:02 INFO juju.apiserver.
controller-0: 20:08:03 INFO juju.apiserver.
controller-0: 20:10:07 ERROR juju.apiserver.
controller-0: 20:10:07 INFO juju.apiserver.
^C
❯ juju debug-log -m spring music-scrape- config- 0: 19:58:37 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-0: 19:58:38 INFO juju.worker. uniter. operation ran "metrics- endpoint- relation- broken" hook (via hook dispatching script: dispatch) music-0: 19:58:39 INFO juju.worker. uniter. operation ran "metrics- endpoint- relation- created" hook (via hook dispatching script: dispatch) music-0: 19:58:58 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-0: 20:03:56 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-scrape- config- 0: 20:04:10 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-scrape- config- 0: 20:09:20 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-0: 20:09:50 INFO juju.worker. uniter. operation ran "update-status" hook (via hook dispatching script: dispatch) music-0: 20:10:48 INFO juju.worker. uniter. operation ran "metrics- endpoint- relation- broken" hook (via hook dispatching script: dispatch) music-0: 20:11:07 INFO juju.worker. uniter. operation ran "metrics- endpoint- relation- created" hook (via hook dispatching script: dispatch)
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
unit-spring-
^C
```