Juju secrets not found in scale down event after removing application

[Shayan]

Juju version: 3.1.5-genericlinux-amd64
lxd version: 5.0.2
Juju agent version: 3.1.5

The MySQL charm relies on the scale down events to correctly shut down, which in turn requires access to the secrets in these events. We run into the following exception (which indicates secrets are not found) after running `juju remove-application mysql` but not after `juju remove-unit mysql/1`. When are juju secrets removed after a `remove-application` call?

Steps to reproduce:
```
$ git clone <email address hidden>:shayancanonical/juju-secrets-operator.git
$ charmcraft pack

$ juju add-model dev
$ juju deploy -n 1 ./secrets-test_ubuntu-22.04-amd64.charm secrets
$ juju run secrets/0 --wait=0s set-secret key=test-key value=test-value
Running operation 1 with 1 task
  - task 2 on unit-secrets-0

Waiting for task 2...
$ juju run secrets/0 --wait=0s get-secrets
Running operation 3 with 1 task
  - task 4 on unit-secrets-0

Waiting for task 4...
secrets:
  test-key: test-value

$ juju remove-application secrets
will remove application secrets
- will remove unit secrets/0
- will remove storage database/0
```

Error trace:
```
machine-0: 14:20:41 INFO juju.downloader download complete ("local:jammy/secrets-test-0")
machine-0: 14:20:41 INFO juju.downloader download verified ("local:jammy/secrets-test-0")
machine-0: 14:20:44 INFO juju.container.lxd Availability zone will be empty for this container manager
machine-0: 14:20:45 INFO juju.worker.kvmprovisioner machine-0 does not support kvm container
unit-secrets-0: 14:20:50 INFO juju.worker.uniter hooks are retried true
unit-secrets-0: 14:20:51 INFO juju.worker.uniter.operation ran "database-storage-attached" hook (via hook dispatching script: dispatch)
unit-secrets-0: 14:20:51 INFO juju.worker.uniter.storage initial storage attachments ready
unit-secrets-0: 14:20:51 INFO unit.secrets/0.juju-log Running legacy hooks/install.
unit-secrets-0: 14:20:51 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch)
unit-secrets-0: 14:20:52 INFO juju.worker.uniter.operation ran "charm-peer-relation-created" hook (via hook dispatching script: dispatch)
unit-secrets-0: 14:20:52 INFO juju.worker.uniter found queued "leader-elected" hook
unit-secrets-0: 14:20:52 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch)
unit-secrets-0: 14:20:53 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch)
unit-secrets-0: 14:20:53 INFO juju.worker.uniter found queued "start" hook
unit-secrets-0: 14:20:53 INFO unit.secrets/0.juju-log Running legacy hooks/start.
unit-secrets-0: 14:20:53 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch)
unit-secrets-0: 14:21:18 INFO unit.secrets/0.juju-log Added secret secret://01698792-b233-4d10-8a4f-a9c09975edb0/cj4kq7kp9net54v7m7qg to {'test-key': 'test-value'}
unit-secrets-0: 14:21:19 INFO juju.worker.uniter.operation ran "charm-peer-relation-changed" hook (via hook dispatching script: dispatch)
unit-secrets-0: 14:21:27 INFO unit.secrets/0.juju-log Retrieved secret secret://01698792-b233-4d10-8a4f-a9c09975edb0/cj4kq7kp9net54v7m7qg with content {'test-key': 'test-value'}
machine-0: 14:21:41 INFO juju.worker.deployer checking unit "secrets/0"
unit-secrets-0: 14:21:41 WARNING juju.worker.uniter.operation we should run a leader-deposed hook here, but we can't yet
unit-secrets-0: 14:21:41 INFO unit.secrets/0.juju-log Starting storage detaching event
unit-secrets-0: 14:21:41 ERROR unit.secrets/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/model.py", line 2693, in _run
    result = subprocess.run(args, **kwargs) # type: ignore
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-secrets-0/secret-get', 'secret://01698792-b233-4d10-8a4f-a9c09975edb0/cj4kq7kp9net54v7m7qg', '--format=json')' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/model.py", line 3049, in secret_get
    result = self._run('secret-get', *args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/model.py", line 2695, in _run
    raise ModelError(e.stderr)
ops.model.ModelError: ERROR secret "secret://01698792-b233-4d10-8a4f-a9c09975edb0/cj4kq7kp9net54v7m7qg" not found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/model.py", line 269, in get_secret
    content = self._backend.secret_get(id=id, label=label)
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/model.py", line 3052, in secret_get
    raise SecretNotFoundError() from e
ops.model.SecretNotFoundError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/model.py", line 2693, in _run
    result = subprocess.run(args, **kwargs) # type: ignore
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-secrets-0/secret-info-get', 'secret://01698792-b233-4d10-8a4f-a9c09975edb0/cj4kq7kp9net54v7m7qg', '--format=json')' returned non-zero exit statu
s 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/model.py", line 3059, in _run_for_secret
    return self._run(*args, return_output=return_output, use_json=use_json)
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/model.py", line 2695, in _run
    raise ModelError(e.stderr)
ops.model.ModelError: ERROR secret "cj4kq7kp9net54v7m7qg" not found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-secrets-0/charm/./src/charm.py", line 115, in <module>
    ops.main(SecretsTestCharm)
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/main.py", line 454, in __call__
    return main(charm_class, use_juju_for_storage=use_juju_for_storage)
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/framework.py", line 344, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/framework.py", line 833, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/framework.py", line 922, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-secrets-0/charm/./src/charm.py", line 110, in _on_database_storage_detaching
    secrets = self.get_secrets()
  File "/var/lib/juju/agents/unit-secrets-0/charm/./src/charm.py", line 79, in get_secrets
    secret = self.model.get_secret(id=secret_id)
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/model.py", line 274, in get_secret
    info = self._backend.secret_info_get(id=id, label=label)
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/model.py", line 3073, in secret_info_get
    result = self._run_for_secret('secret-info-get', *args, return_output=True, use_json=True)
  File "/var/lib/juju/agents/unit-secrets-0/charm/venv/ops/model.py", line 3062, in _run_for_secret
    raise SecretNotFoundError() from e
ops.model.SecretNotFoundError
unit-secrets-0: 14:21:41 ERROR juju.worker.uniter.operation hook "database-storage-detaching" (via hook dispatching script: dispatch) failed: exit status 1
```

[Ian Booth (wallyworld)]

Secrets are currently removed when an application has finished the transition to life=Dying. This is also when things like status history are removed.

Based on this bug, we should defer secret removal until the after the application has transitioned to Dead.

[Ian Booth (wallyworld)]

https://github.com/juju/juju/pull/16034