Peer relation disappears too early on application removal

[Leon]

When I `juju remove-application` (with or without `--destroy-storage`), it seems that the peer relation disappears too quickly - before other relation events were processed:

See logs in thread: https://github.com/canonical/grafana-k8s-operator/issues/138

In particular:
During hook "grafana-source-relation-departed", I get:

    data = self._charm.peers.data[self._charm.app].get(key, "")
AttributeError: 'NoneType' object has no attribute 'data'

Shouldn't peer relations be removed last?

[Joseph Phillips (manadart)]

Is the model completely quiesced at point of the removal?

[Leon (sed-i)]

So I saw this after `juju destroy-model` but also after only `juju remove-application` (logs in the github issue).

[Leon (sed-i)]

This is still an issue on Juju 3.2.0.

Repro:

1. Relate prometheus to grafana
2. Remove grafana

unit-graf-0: 22:45:28.955 ERROR unit.graf/0.juju-log grafana-source:81: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 1280, in <module>
    main(GrafanaCharm, use_juju_for_storage=True)
  File "/var/lib/juju/agents/unit-graf-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-graf-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-graf-0/charm/venv/ops/framework.py", line 344, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-graf-0/charm/venv/ops/framework.py", line 833, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-graf-0/charm/venv/ops/framework.py", line 922, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-graf-0/charm/lib/charms/grafana_k8s/v0/grafana_source.py", line 615, in _on_grafana_source_relation_departed
    removed_source = self._remove_source_from_datastore(event)
  File "/var/lib/juju/agents/unit-graf-0/charm/lib/charms/grafana_k8s/v0/grafana_source.py", line 630, in _remove_source_from_datastore
    stored_sources = self.get_peer_data("sources")
  File "/var/lib/juju/agents/unit-graf-0/charm/lib/charms/grafana_k8s/v0/grafana_source.py", line 733, in get_peer_data
    data = self._charm.peers.data[self._charm.app].get(key, "") # type: ignore[attr-defined]
AttributeError: 'NoneType' object has no attribute 'data'
unit-graf-0: 22:45:29.204 ERROR juju.worker.uniter.operation hook "grafana-source-relation-departed" (via hook dispatching script: dispatch) failed: exit status 1

[Leon (sed-i)]

On relation-departed of a regular relation, the peer relation is no longer there.

[Chi Wai CHAN (raychan96)]

This is also affecting sunbeam observability plugin [1]: we can't proper disable the plugin (i.e. destroy the COS model [2]) without `--force`. The observability plugin is simply the COS lite bundle written in terraform plan [2], and is deployed with terraform with terraform-provider-juju on juju 3.1.6 controller.

Reproduce:

1. Deploy COS via [2] using `terraform apply` with terraform-provider-juju
2. Destroy COS with `terraform destroy`
3. Check the `juju status` and `juju debug-log`, and it should show exactly the error mentioned comment #3, and the destruction is kept re-trying with the same error.

@Joseph is it possible that you can take a look at it?

[1] https://microstack.run/docs/observability
[2] https://github.com/canonical/snap-openstack/blob/main/sunbeam-python/sunbeam/plugins/observability/etc/deploy-cos/main.tf