Removing an application in Juju 2.9.28 may leave a model in a zombie state with an application stuck in active state with scale 0/1

[Ryan Barry]

In some cases, removing an application leaves a Juju model in an inconsistent state where an application is "stuck" after being removed, and the model cannot be destroyed, even with --force (nor can the application be removed)

When this occurs, no meaningful messages come through either the controller logs or the model logs, even at maximum logging levels.

The "dead" application lives on. It continues to have a valid IP address in the `cloudservices` collection in Mongo even though the state of the application is dead and there are no pods left, continues to have settings even after this, and the refcount never drops to zero.

Controller logs, model logs, and a dump of the juju database are attached.

Reproduction is somewhat of a race, but is more or less reproducible from here, for example:
https://github.com/canonical/loki-k8s-operator/pull/150

The tests will pass, but run a couple of times (or maybe only once) and look for models stuck in "Destroying..."

Juju 2.9.29 does resolve this, but the concern is that we are relying on behavior not guaranteed, because the symptoms do not otherwise match any known bugs.

[Harry Pidcock (hpidcock)]

https://github.com/juju/juju/pull/13973

[Ben Hoyt (benhoyt)]

We believe this will be address by the fix in 13973 (specifically the race condition in commit https://github.com/juju/juju/pull/13973/commits/5ebc199b2de89872ad053f2a82d46b0447a5faf0) which will be included in Juju 2.9.30.