| 2021-12-16 10:30:46 |
Simon Aronsson |
description |
The remote end of a cross-model relation fails intermittently. The workload of that unit continues to function in isolation, but events are no longer propagated to the charm. Once this happens, it may be resolved using the following workaround:
```
juju remove-saas someoffer
juju remove-offer -m lma someoffer --force # not always needed but doesn't hurt
juju offer ...
juju consume
juju relate ...
```
As this is an intermittent error I'm not really able to reproduce it in a controlled fashion, but I've attached a dump of my terminal from last time it happened.
```
❯ juju version
2.9.22-ubuntu-amd64
```
Running on microk8s.
```
❯ juju remove-relation spring-music prometheus
❯ juju debug-log -m spring
unit-spring-music-0: 19:53:22 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-created" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:57:41 INFO juju.util.exec run result: exit status 1
unit-spring-music-scrape-config-0: 19:58:37 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:38 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-broken" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:39 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-created" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:58 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:03:56 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-scrape-config-0: 20:04:10 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-scrape-config-0: 20:09:20 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:09:50 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:10:48 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-broken" hook (via hook dispatching script: dispatch)
^C
❯ juju add-relation spring-music prometheus
❯ juju debug-log -m spring
unit-spring-music-0: 19:57:41 INFO juju.util.exec run result: exit status 1
unit-spring-music-scrape-config-0: 19:58:37 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:38 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-broken" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:39 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-created" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:58 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:03:56 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-scrape-config-0: 20:04:10 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-scrape-config-0: 20:09:20 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:09:50 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:10:48 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-broken" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:11:07 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-created" hook (via hook dispatching script: dispatch)
^C
❯ juju debug-log -m controller
controller-0: 20:07:47 INFO juju.apiserver.connection agent disconnected: unit-prometheus-1 for bccb7cc3-a40a-474e-83c3-20b8f0b09a80
controller-0: 20:07:48 INFO juju.apiserver.connection agent login: unit-prometheus-1 for bccb7cc3-a40a-474e-83c3-20b8f0b09a80
controller-0: 20:07:50 INFO juju.apiserver.common setting password for "application-prometheus"
controller-0: 20:07:54 ERROR juju.apiserver.uniter resolving "": lookup : no such host
controller-0: 20:07:55 ERROR juju.apiserver.uniter resolving "": lookup : no such host
controller-0: 20:08:02 INFO juju.apiserver.connection agent login: application-prometheus for bccb7cc3-a40a-474e-83c3-20b8f0b09a80
controller-0: 20:08:02 INFO juju.apiserver.connection agent disconnected: application-prometheus for bccb7cc3-a40a-474e-83c3-20b8f0b09a80
controller-0: 20:08:03 INFO juju.apiserver.connection agent login: unit-prometheus-0 for bccb7cc3-a40a-474e-83c3-20b8f0b09a80
controller-0: 20:10:07 ERROR juju.apiserver.common error stopping *apiserver.pingTimeout resource: ping timeout
controller-0: 20:10:07 INFO juju.apiserver.connection agent disconnected: unit-prometheus-0 for bccb7cc3-a40a-474e-83c3-20b8f0b09a80
^C
❯ juju debug-log -m spring
unit-spring-music-scrape-config-0: 19:58:37 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:38 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-broken" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:39 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-created" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:58 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:03:56 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-scrape-config-0: 20:04:10 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-scrape-config-0: 20:09:20 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:09:50 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:10:48 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-broken" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:11:07 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-created" hook (via hook dispatching script: dispatch)
^C
``` |
The remote end of a cross-model relation fails intermittently. The workload of that unit continues to function in isolation, but events are no longer propagated to the charm.
As this is an intermittent error I'm not really able to reproduce it in a controlled fashion, but I've attached a dump of my terminal from last time it happened.
```
❯ juju version
2.9.22-ubuntu-amd64
```
Running on microk8s.
```
❯ juju remove-relation spring-music prometheus
❯ juju debug-log -m spring
unit-spring-music-0: 19:53:22 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-created" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:57:41 INFO juju.util.exec run result: exit status 1
unit-spring-music-scrape-config-0: 19:58:37 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:38 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-broken" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:39 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-created" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:58 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:03:56 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-scrape-config-0: 20:04:10 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-scrape-config-0: 20:09:20 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:09:50 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:10:48 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-broken" hook (via hook dispatching script: dispatch)
^C
❯ juju add-relation spring-music prometheus
❯ juju debug-log -m spring
unit-spring-music-0: 19:57:41 INFO juju.util.exec run result: exit status 1
unit-spring-music-scrape-config-0: 19:58:37 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:38 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-broken" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:39 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-created" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:58 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:03:56 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-scrape-config-0: 20:04:10 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-scrape-config-0: 20:09:20 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:09:50 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:10:48 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-broken" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:11:07 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-created" hook (via hook dispatching script: dispatch)
^C
❯ juju debug-log -m controller
controller-0: 20:07:47 INFO juju.apiserver.connection agent disconnected: unit-prometheus-1 for bccb7cc3-a40a-474e-83c3-20b8f0b09a80
controller-0: 20:07:48 INFO juju.apiserver.connection agent login: unit-prometheus-1 for bccb7cc3-a40a-474e-83c3-20b8f0b09a80
controller-0: 20:07:50 INFO juju.apiserver.common setting password for "application-prometheus"
controller-0: 20:07:54 ERROR juju.apiserver.uniter resolving "": lookup : no such host
controller-0: 20:07:55 ERROR juju.apiserver.uniter resolving "": lookup : no such host
controller-0: 20:08:02 INFO juju.apiserver.connection agent login: application-prometheus for bccb7cc3-a40a-474e-83c3-20b8f0b09a80
controller-0: 20:08:02 INFO juju.apiserver.connection agent disconnected: application-prometheus for bccb7cc3-a40a-474e-83c3-20b8f0b09a80
controller-0: 20:08:03 INFO juju.apiserver.connection agent login: unit-prometheus-0 for bccb7cc3-a40a-474e-83c3-20b8f0b09a80
controller-0: 20:10:07 ERROR juju.apiserver.common error stopping *apiserver.pingTimeout resource: ping timeout
controller-0: 20:10:07 INFO juju.apiserver.connection agent disconnected: unit-prometheus-0 for bccb7cc3-a40a-474e-83c3-20b8f0b09a80
^C
❯ juju debug-log -m spring
unit-spring-music-scrape-config-0: 19:58:37 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:38 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-broken" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:39 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-created" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 19:58:58 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:03:56 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-scrape-config-0: 20:04:10 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-scrape-config-0: 20:09:20 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:09:50 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:10:48 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-broken" hook (via hook dispatching script: dispatch)
unit-spring-music-0: 20:11:07 INFO juju.worker.uniter.operation ran "metrics-endpoint-relation-created" hook (via hook dispatching script: dispatch)
^C
```
Once this happens, it may be resolved using the following workaround:
```
juju remove-saas someoffer
juju remove-offer -m lma someoffer --force # not always needed but doesn't hurt
juju offer ...
juju consume
juju relate ...
``` |
|
