Error in consuming secrets if one of offers is removed with --force flag

[Hemanth Nakkina]

Application A on model A has multiple CMR with applications on model B.
One of the offer on model B is forcefully removed. (Simulating terraform-provider-juju behavior if offer is deleted with integrations)
Application A went into error state unable to get secrets (The secret still exists on model B).

Other observations:
Observed <relation>-departed hook called on Application A but the <relation> is not the destroyed offer one. Seems like rest of the remote integrations are also getting departed which is not expected.

Reproducer:
Environment: juju 3.5/stable + microk8s 1.28-strict/stable (metallb enabled)

juju add-model keystone
juju deploy ch:traefik-k8s traefik --channel latest/stable --trust
juju deploy ch:mysql-k8s mysql --channel 8.0/stable
juju deploy ch:rabbitmq-k8s rabbitmq --channel 3.12/edge --config minimum-replicas=1 --trust
juju deploy ch:keystone-k8s keystone --channel 2024.1/edge --base ubuntu@24.04
juju integrate keystone:database mysql:database
juju integrate keystone:ingress-public traefik:ingress
juju offer mysql:database
juju offer keystone:identity-service
juju offer rabbitmq:amqp
juju offer traefik:ingress

# Wait for all units in keystone model settle and active

juju add-model glance
juju deploy ch:glance-k8s glance --channel 2024.1/edge --base ubuntu@24.04 --storage local-repository=10G
juju integrate glance:database admin/keystone.mysql
juju integrate glance:identity-service admin/keystone.keystone
juju integrate glance:ingress-public admin/keystone.traefik
juju integrate glance:amqp admin/keystone.rabbitmq

# Wait for all units in glance model settle and active

# Force remove one of the offer
juju switch keystone
juju remove-offer rabbitmq --force

#### Logs and other outputs:

# Model glance status

$ juju status -m glance
Model Controller Cloud/Region Version SLA Timestamp
glance microk8s-localhost microk8s/localhost 3.5.4 unsupported 03:09:39Z

SAAS Status Store URL
keystone active microk8s-localhost admin/keystone.keystone
mysql active microk8s-localhost admin/keystone.mysql
rabbitmq terminated microk8s-localhost admin/keystone.rabbitmq
traefik error microk8s-localhost admin/keystone.traefik

App Version Status Scale Charm Channel Rev Address Exposed Message
glance error 1 glance-k8s 2024.1/edge 115 10.152.183.40 no hook failed: "ingress-public-relation-departed"

Unit Workload Agent Address Ports Message
glance/0* error idle 10.1.230.45 hook failed: "ingress-public-relation-departed"

# Model keystone status

$ juju status -m keystone
Model Controller Cloud/Region Version SLA Timestamp
keystone microk8s-localhost microk8s/localhost 3.5.4 unsupported 03:09:53Z

App Version Status Scale Charm Channel Rev Address Exposed Message
keystone active 1 keystone-k8s 2024.1/edge 211 10.152.183.116 no
mysql 8.0.37-0ubuntu0.22.04.3 active 1 mysql-k8s 8.0/stable 180 10.152.183.235 no
rabbitmq 3.12.1 active 1 rabbitmq-k8s 3.12/edge 37 10.121.193.31 no
traefik v2.11.0 error 1 traefik-k8s latest/stable 194 10.152.183.236 no hook failed: "ingress-relation-departed"

Unit Workload Agent Address Ports Message
keystone/0* active idle 10.1.230.42
mysql/0* active idle 10.1.230.37 Primary
rabbitmq/0* active idle 10.1.230.39
traefik/0* error idle 10.1.230.35 hook failed: "ingress-relation-departed"

Offer Application Charm Rev Connected Endpoint Interface Role
keystone keystone keystone-k8s 211 0/0 identity-service keystone provider
mysql mysql mysql-k8s 180 1/1 database mysql_client provider
traefik traefik traefik-k8s 194 0/0 ingress ingress provider

# Unit glance logs

unit-glance-0: 03:06:56 WARNING unit.glance/0.juju-log Secret secret://e62a15e7-8448-49d1-824e-9c7218800c50/csbhbv7mp25c74pvb8a0 not found
unit-glance-0: 03:06:57 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)
unit-glance-0: 03:06:58 ERROR unit.glance/0.juju-log ingress-public:3: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-glance-0/charm/venv/ops/model.py", line 3319, in _run
    result = subprocess.run(args, **kwargs) # type: ignore
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-glance-0/secret-get', 'secret://e62a15e7-8448-49d1-824e-9c7218800c50/csbhbv7mp25c74pvb8a0', '--format=json')' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-glance-0/charm/./src/charm.py", line 602, in <module>
    ops.main(GlanceOperatorCharm)
  File "/var/lib/juju/agents/unit-glance-0/charm/venv/ops/__init__.py", line 343, in __call__
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/venv/ops/_main.py", line 541, in main
    manager = _Manager(charm_class, use_juju_for_storage=use_juju_for_storage)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/venv/ops/_main.py", line 424, in __init__
    self.charm = self._make_charm(self.framework, self.dispatcher)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/venv/ops/_main.py", line 427, in _make_charm
    charm = self._charm_class(framework)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 402, in wrap_init
    original_init(self, framework, *args, **kwargs)
  File "/var/lib/juju/agents/unit-glance-0/charm/./src/charm.py", line 298, in __init__
    super().__init__(*args, **kwargs)
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/ops_sunbeam/charm.py", line 798, in __init__
    super().__init__(framework)
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/ops_sunbeam/charm.py", line 597, in __init__
    super().__init__(framework)
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/ops_sunbeam/charm.py", line 102, in __init__
    self.relation_handlers = self.get_relation_handlers()
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 677, in wrapped_function
    return callable(*args, **kwargs) # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/./src/charm.py", line 367, in get_relation_handlers
    handlers = super().get_relation_handlers()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/ops_sunbeam/charm.py", line 833, in get_relation_handlers
    self.id_svc = sunbeam_rhandlers.IdentityServiceRequiresHandler(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/ops_sunbeam/relation_handlers.py", line 544, in __init__
    super().__init__(charm, relation_name, callback_f, mandatory)
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/ops_sunbeam/relation_handlers.py", line 113, in __init__
    self.set_status(self.status)
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 677, in wrapped_function
    return callable(*args, **kwargs) # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 677, in wrapped_function
    return callable(*args, **kwargs) # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/ops_sunbeam/relation_handlers.py", line 127, in set_status
    elif self.ready:
         ^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/ops_sunbeam/relation_handlers.py", line 589, in ready
    return bool(self.interface.service_password)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/charms/keystone_k8s/v1/identity_service.py", line 295, in service_password
    credentials = self.charm.model.get_secret(id=credentials_id)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/venv/ops/model.py", line 307, in get_secret
    content = self._backend.secret_get(id=id, label=label)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/venv/ops/model.py", line 3691, in secret_get
    result = self._run('secret-get', *args, return_output=True, use_json=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/venv/ops/model.py", line 3321, in _run
    raise ModelError(e.stderr) from e
ops.model.ModelError: ERROR permission denied

unit-glance-0: 03:06:58 ERROR juju.worker.uniter.operation hook "ingress-public-relation-departed" (via hook dispatching script: dispatch) failed: exit status 1
unit-glance-0: 03:06:58 INFO juju.worker.uniter awaiting error resolution for "relation-departed" hook
unit-glance-0: 03:07:03 INFO juju.worker.uniter awaiting error resolution for "relation-departed" hook
unit-glance-0: 03:07:04 ERROR unit.glance/0.juju-log ingress-public:3: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-glance-0/charm/venv/ops/model.py", line 3319, in _run
    result = subprocess.run(args, **kwargs) # type: ignore
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-glance-0/secret-get', 'secret://e62a15e7-8448-49d1-824e-9c7218800c50/csbhbv7mp25c74pvb8a0', '--format=json')' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-glance-0/charm/./src/charm.py", line 602, in <module>
    ops.main(GlanceOperatorCharm)
  File "/var/lib/juju/agents/unit-glance-0/charm/venv/ops/__init__.py", line 343, in __call__
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/venv/ops/_main.py", line 541, in main
    manager = _Manager(charm_class, use_juju_for_storage=use_juju_for_storage)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/venv/ops/_main.py", line 424, in __init__
    self.charm = self._make_charm(self.framework, self.dispatcher)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/venv/ops/_main.py", line 427, in _make_charm
    charm = self._charm_class(framework)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 402, in wrap_init
    original_init(self, framework, *args, **kwargs)
  File "/var/lib/juju/agents/unit-glance-0/charm/./src/charm.py", line 298, in __init__
    super().__init__(*args, **kwargs)
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/ops_sunbeam/charm.py", line 798, in __init__
    super().__init__(framework)
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/ops_sunbeam/charm.py", line 597, in __init__
    super().__init__(framework)
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/ops_sunbeam/charm.py", line 102, in __init__
    self.relation_handlers = self.get_relation_handlers()
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 677, in wrapped_function
    return callable(*args, **kwargs) # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/./src/charm.py", line 367, in get_relation_handlers
    handlers = super().get_relation_handlers()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/ops_sunbeam/charm.py", line 833, in get_relation_handlers
    self.id_svc = sunbeam_rhandlers.IdentityServiceRequiresHandler(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/ops_sunbeam/relation_handlers.py", line 544, in __init__
    super().__init__(charm, relation_name, callback_f, mandatory)
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/ops_sunbeam/relation_handlers.py", line 113, in __init__
    self.set_status(self.status)
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 677, in wrapped_function
    return callable(*args, **kwargs) # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 677, in wrapped_function
    return callable(*args, **kwargs) # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/ops_sunbeam/relation_handlers.py", line 127, in set_status
    elif self.ready:
         ^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/ops_sunbeam/relation_handlers.py", line 589, in ready
    return bool(self.interface.service_password)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/lib/charms/keystone_k8s/v1/identity_service.py", line 295, in service_password
    credentials = self.charm.model.get_secret(id=credentials_id)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/venv/ops/model.py", line 307, in get_secret
    content = self._backend.secret_get(id=id, label=label)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/venv/ops/model.py", line 3691, in secret_get
    result = self._run('secret-get', *args, return_output=True, use_json=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-glance-0/charm/venv/ops/model.py", line 3321, in _run
    raise ModelError(e.stderr) from e
ops.model.ModelError: ERROR permission denied

unit-glance-0: 03:07:04 ERROR juju.worker.uniter.operation hook "ingress-public-relation-departed" (via hook dispatching script: dispatch) failed: exit status 1

# Controller logs on model glance

juju@controller-0:/var/log/juju/models$ cat admin-glance-cbb4d9.log
2024-10-22 03:01:18 INFO juju.worker.apicaller connect.go:163 [cbb4d9] "controller-0" successfully connected to "localhost:17070"
2024-10-22 03:01:18 INFO juju.worker.logforwarder logforwarder.go:105 config change - log forwarding not enabled
2024-10-22 03:01:18 INFO juju.worker.logger logger.go:120 logger worker started
2024-10-22 03:01:18 INFO juju.worker.pruner.action worker.go:98 pruner config: max age: 336h0m0s, max collection size 5120M for glance (cbb4d908-5415-4e29-854c-8fb2f6a780b9)
2024-10-22 03:01:18 INFO juju.worker.pruner.statushistory worker.go:98 pruner config: max age: 336h0m0s, max collection size 5120M for glance (cbb4d908-5415-4e29-854c-8fb2f6a780b9)
2024-10-22 03:01:50 INFO juju.worker.caasapplicationprovisioner.runner runner.go:592 start "glance"
2024-10-22 03:01:50 INFO juju.worker.caasapplicationprovisioner.glance ops.go:716 scaling application "glance" to desired scale 1
2024-10-22 03:01:50 INFO juju.worker.caasapplicationprovisioner.glance ops.go:646 scaling application "glance" to desired scale 1
2024-10-22 03:01:50 INFO juju.worker.caasapplicationprovisioner.glance ops.go:716 scaling application "glance" to desired scale 1
2024-10-22 03:01:53 INFO juju.worker.caasapplicationprovisioner.glance ops.go:716 scaling application "glance" to desired scale 1
2024-10-22 03:01:59 INFO juju.worker.caasapplicationprovisioner.glance ops.go:716 scaling application "glance" to desired scale 1
2024-10-22 03:03:03 INFO juju.worker.remoterelations runner.go:592 start "mysql"
2024-10-22 03:03:03 INFO juju.worker.remoterelations runner.go:592 start "keystone"
2024-10-22 03:03:04 INFO juju.worker.remoterelations runner.go:592 start "traefik"
2024-10-22 03:03:05 INFO juju.worker.remoterelations runner.go:592 start "rabbitmq"
2024-10-22 03:06:54 ERROR juju.worker.remoterelations remoterelationsworker.go:70 error in remote relations worker for relation glance:database mysql:database: connection is shut down
2024-10-22 03:06:54 ERROR juju.worker.remoterelations relationunitsworker.go:86 error in relation units worker for glance:database mysql:database: connection is shut down
2024-10-22 03:06:54 ERROR juju.worker.remoterelations remoterelationsworker.go:70 error in remote relations worker for relation glance:database mysql:database: connection is shut down
2024-10-22 03:06:54 ERROR juju.worker.remoterelations relationunitsworker.go:86 error in relation units worker for glance:database mysql:database: connection is shut down
2024-10-22 03:06:54 ERROR juju.worker.remoterelations remoteapplicationworker.go:93 error in remote application worker for mysql: consuming secrets change []watcher.SecretRevisionChange{csbhbrvmp25c74pvb89g/1} from remote model e62a15e7-8448-49d1-824e-9c7218800c50: secret consumers for secret "secret:csbhbrvmp25c74pvb89g" not found
2024-10-22 03:06:54 INFO juju.worker.remoterelations runner.go:623 stopped "mysql", err: consuming secrets change []watcher.SecretRevisionChange{csbhbrvmp25c74pvb89g/1} from remote model e62a15e7-8448-49d1-824e-9c7218800c50: secret consumers for secret "secret:csbhbrvmp25c74pvb89g" not found
2024-10-22 03:06:54 INFO juju.worker.remoterelations runner.go:498 non-fatal error "mysql": consuming secrets change []watcher.SecretRevisionChange{csbhbrvmp25c74pvb89g/1} from remote model e62a15e7-8448-49d1-824e-9c7218800c50: secret consumers for secret "secret:csbhbrvmp25c74pvb89g" not found
2024-10-22 03:06:54 ERROR juju.worker.remoterelations runner.go:506 exited "mysql": consuming secrets change []watcher.SecretRevisionChange{csbhbrvmp25c74pvb89g/1} from remote model e62a15e7-8448-49d1-824e-9c7218800c50: secret consumers for secret "secret:csbhbrvmp25c74pvb89g" not found
2024-10-22 03:06:54 INFO juju.worker.remoterelations runner.go:582 restarting "mysql" in 15s
2024-10-22 03:06:54 INFO juju.worker.remoterelations runner.go:623 stopped "rabbitmq", err: <nil>
2024-10-22 03:07:09 INFO juju.worker.remoterelations runner.go:592 start "mysql"

# Controller logs on keystone model

juju@controller-0:/var/log/juju/models$ cat admin-keystone-e62a15.log
2024-10-22 02:56:59 INFO juju.worker.apicaller connect.go:163 [e62a15] "controller-0" successfully connected to "localhost:17070"
2024-10-22 02:57:00 INFO juju.worker.logforwarder logforwarder.go:105 config change - log forwarding not enabled
2024-10-22 02:57:00 INFO juju.worker.logger logger.go:120 logger worker started
2024-10-22 02:57:00 INFO juju.worker.pruner.action worker.go:98 pruner config: max age: 336h0m0s, max collection size 5120M for keystone (e62a15e7-8448-49d1-824e-9c7218800c50)
2024-10-22 02:57:00 INFO juju.worker.pruner.statushistory worker.go:98 pruner config: max age: 336h0m0s, max collection size 5120M for keystone (e62a15e7-8448-49d1-824e-9c7218800c50)
2024-10-22 02:57:02 INFO juju.worker.caasapplicationprovisioner.runner runner.go:592 start "traefik"
2024-10-22 02:57:02 INFO juju.worker.caasapplicationprovisioner.traefik ops.go:716 scaling application "traefik" to desired scale 1
2024-10-22 02:57:04 INFO juju.worker.caasapplicationprovisioner.runner runner.go:592 start "mysql"
2024-10-22 02:57:04 INFO juju.worker.caasapplicationprovisioner.mysql ops.go:716 scaling application "mysql" to desired scale 1
2024-10-22 02:57:04 INFO juju.worker.caasapplicationprovisioner.mysql ops.go:646 scaling application "mysql" to desired scale 1
2024-10-22 02:57:05 INFO juju.worker.caasapplicationprovisioner.traefik ops.go:716 scaling application "traefik" to desired scale 1
2024-10-22 02:57:05 INFO juju.worker.caasapplicationprovisioner.runner runner.go:592 start "rabbitmq"
2024-10-22 02:57:05 INFO juju.worker.caasapplicationprovisioner.rabbitmq ops.go:716 scaling application "rabbitmq" to desired scale 1
2024-10-22 02:57:05 INFO juju.worker.caasapplicationprovisioner.rabbitmq ops.go:646 scaling application "rabbitmq" to desired scale 1
2024-10-22 02:57:07 INFO juju.worker.caasapplicationprovisioner.runner runner.go:592 start "keystone"
2024-10-22 02:57:07 INFO juju.worker.caasapplicationprovisioner.mysql ops.go:716 scaling application "mysql" to desired scale 1
2024-10-22 02:57:08 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:57:08 INFO juju.worker.caasapplicationprovisioner.rabbitmq ops.go:716 scaling application "rabbitmq" to desired scale 1
2024-10-22 02:57:11 INFO juju.worker.caasapplicationprovisioner.mysql ops.go:716 scaling application "mysql" to desired scale 1
2024-10-22 02:57:12 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:57:12 INFO juju.worker.caasapplicationprovisioner.rabbitmq ops.go:716 scaling application "rabbitmq" to desired scale 1
2024-10-22 02:57:14 INFO juju.worker.caasapplicationprovisioner.traefik ops.go:716 scaling application "traefik" to desired scale 1
2024-10-22 02:57:14 INFO juju.worker.caasapplicationprovisioner.mysql ops.go:716 scaling application "mysql" to desired scale 1
2024-10-22 02:57:15 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:57:15 INFO juju.worker.caasapplicationprovisioner.rabbitmq ops.go:716 scaling application "rabbitmq" to desired scale 1
2024-10-22 02:57:17 INFO juju.worker.caasapplicationprovisioner.mysql ops.go:716 scaling application "mysql" to desired scale 1
2024-10-22 02:57:18 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:57:18 INFO juju.worker.caasapplicationprovisioner.rabbitmq ops.go:716 scaling application "rabbitmq" to desired scale 1
2024-10-22 02:57:20 INFO juju.worker.caasapplicationprovisioner.mysql ops.go:716 scaling application "mysql" to desired scale 1
2024-10-22 02:57:21 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:57:21 INFO juju.worker.caasapplicationprovisioner.rabbitmq ops.go:716 scaling application "rabbitmq" to desired scale 1
2024-10-22 02:57:23 INFO juju.worker.caasapplicationprovisioner.mysql ops.go:716 scaling application "mysql" to desired scale 1
2024-10-22 02:57:24 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:57:24 INFO juju.worker.caasapplicationprovisioner.rabbitmq ops.go:716 scaling application "rabbitmq" to desired scale 1
2024-10-22 02:57:26 INFO juju.worker.caasapplicationprovisioner.mysql ops.go:716 scaling application "mysql" to desired scale 1
2024-10-22 02:57:27 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:57:27 INFO juju.worker.caasapplicationprovisioner.rabbitmq ops.go:716 scaling application "rabbitmq" to desired scale 1
2024-10-22 02:57:29 INFO juju.worker.caasapplicationprovisioner.mysql ops.go:716 scaling application "mysql" to desired scale 1
2024-10-22 02:57:30 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:57:30 INFO juju.worker.caasapplicationprovisioner.rabbitmq ops.go:716 scaling application "rabbitmq" to desired scale 1
2024-10-22 02:57:32 INFO juju.worker.caasapplicationprovisioner.mysql ops.go:716 scaling application "mysql" to desired scale 1
2024-10-22 02:57:33 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:57:33 INFO juju.worker.caasapplicationprovisioner.rabbitmq ops.go:716 scaling application "rabbitmq" to desired scale 1
2024-10-22 02:57:35 INFO juju.worker.caasapplicationprovisioner.mysql ops.go:716 scaling application "mysql" to desired scale 1
2024-10-22 02:57:36 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:57:36 INFO juju.worker.caasapplicationprovisioner.rabbitmq ops.go:716 scaling application "rabbitmq" to desired scale 1
2024-10-22 02:57:38 INFO juju.worker.caasapplicationprovisioner.mysql ops.go:716 scaling application "mysql" to desired scale 1
2024-10-22 02:57:39 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:57:39 INFO juju.worker.caasapplicationprovisioner.rabbitmq ops.go:716 scaling application "rabbitmq" to desired scale 1
2024-10-22 02:57:41 INFO juju.worker.caasapplicationprovisioner.mysql ops.go:716 scaling application "mysql" to desired scale 1
2024-10-22 02:57:42 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:57:42 INFO juju.worker.caasapplicationprovisioner.rabbitmq ops.go:716 scaling application "rabbitmq" to desired scale 1
2024-10-22 02:57:45 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:57:45 INFO juju.worker.caasapplicationprovisioner.rabbitmq ops.go:716 scaling application "rabbitmq" to desired scale 1
2024-10-22 02:57:48 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:57:48 INFO juju.worker.caasapplicationprovisioner.mysql ops.go:716 scaling application "mysql" to desired scale 1
2024-10-22 02:57:51 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:57:54 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:57:54 INFO juju.worker.caasapplicationprovisioner.rabbitmq ops.go:716 scaling application "rabbitmq" to desired scale 1
2024-10-22 02:57:57 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:58:00 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:58:03 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:58:06 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:58:09 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:58:09 INFO juju.worker.caasapplicationprovisioner.runner runner.go:623 stopped "keystone", err: more than 20 retries ensuring scale: setting scale to 1 for "keystone": statefulset "keystone" not found
2024-10-22 02:58:09 INFO juju.worker.caasapplicationprovisioner.runner runner.go:498 non-fatal error "keystone": more than 20 retries ensuring scale: setting scale to 1 for "keystone": statefulset "keystone" not found
2024-10-22 02:58:09 ERROR juju.worker.caasapplicationprovisioner.runner runner.go:506 exited "keystone": more than 20 retries ensuring scale: setting scale to 1 for "keystone": statefulset "keystone" not found
2024-10-22 02:58:09 INFO juju.worker.caasapplicationprovisioner.runner runner.go:582 restarting "keystone" in 3s
2024-10-22 02:58:12 INFO juju.worker.caasapplicationprovisioner.runner runner.go:592 start "keystone"
2024-10-22 02:58:12 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:646 scaling application "keystone" to desired scale 1
2024-10-22 02:58:12 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 02:58:12 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:646 scaling application "keystone" to desired scale 1
2024-10-22 02:58:24 INFO juju.worker.caasapplicationprovisioner.keystone ops.go:716 scaling application "keystone" to desired scale 1
2024-10-22 03:03:03 INFO juju.worker.remoterelations runner.go:592 start "remote-5cb15ce16499421f8451aa03947a5e81"
2024-10-22 03:06:54 INFO juju.worker.remoterelations runner.go:623 stopped "remote-5cb15ce16499421f8451aa03947a5e81", err: <nil>
2024-10-22 03:06:54 INFO juju.worker.remoterelations runner.go:592 start "remote-5cb15ce16499421f8451aa03947a5e81"

[Hemanth Nakkina (hemanth-n)]

Relevant info from above outputs:

* Offer removed is rabbitmq. So the integration that should be departed is glance:amqp
* In Unit glance logs, we can observe the relation departed is "ingress-public"
  unit-glance-0: 03:07:04 ERROR juju.worker.uniter.operation hook "ingress-public-relation-departed" (via hook dispatching script: dispatch) failed: exit status 1
* In Controller logs on model glance, we see another remote relation "mysql" is shutdown
  2024-10-22 03:06:54 ERROR juju.worker.remoterelations remoterelationsworker.go:70 error in remote relations worker for relation glance:database mysql:database: connection is shut down

[John A Meinel (jameinel)]

That makes it sound that while the symptom is the secret not being accessible, the actual failure is that we are killing the mysql relation and not just the amqp relation.

That said, this seems suspicious to me:
```
2024-10-22 03:03:03 INFO juju.worker.remoterelations runner.go:592 start "remote-5cb15ce16499421f8451aa03947a5e81"
2024-10-22 03:06:54 INFO juju.worker.remoterelations runner.go:623 stopped "remote-5cb15ce16499421f8451aa03947a5e81", err: <nil>
2024-10-22 03:06:54 INFO juju.worker.remoterelations runner.go:592 start "remote-5cb15ce16499421f8451aa03947a5e81"
```

It appears that something that temporarily stops a remote relation runner, and then restores it.

[Heather Lanigan (hmlanigan)]

@Hemanth Nakkina, please provide a link to the terraform plan you've referenced.

[Alastair Flynn (aflynn50)]

Assigned this high since this is causing seemingly unconnected relations to be departed which is not expected behaviour.

[Joseph Phillips (manadart)]

2024-10-22 03:06:54 ERROR juju.worker.remoterelations remoterelationsworker.go:70 error in remote relations worker for relation glance:database mysql:database: connection is shut down

This is probably due to the remote worker bouncing:
2024-10-22 03:06:54 INFO juju.worker.remoterelations runner.go:623 stopped "remote-5cb15ce16499421f8451aa03947a5e81", err: <nil>
2024-10-22 03:06:54 INFO juju.worker.remoterelations runner.go:592 start "remote-5cb15ce16499421f8451aa03947a5e81"

I expect that upon restarting, the normal function of this CMR will resume. The relation is not departed.

This part at least appears to be a red herring.