Allow admin remove-unit to K8s pods

[Haw Loeung]

Hi,

Every so often, we have units stuck in error or terminating state. Some of which the K8s pods have already disappeared and Juju fails to remove the unit (see ~Juju 2022-02-10). We should have a hammer to remove-unit --force to clean these up.

[Harry Pidcock (hpidcock)]

Please change back to New if you are still seeing this. Some `juju debug-log` from the controller and model would be helpful if it does occur again.

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]

[Haw Loeung (hloeung)]

I think having a "remove-unit --force" option would still be useful though so reopening this.

[John A Meinel (jameinel)]

The issue on K8s is that you cannot remove a pod "in the middle" of a stateful-set. You can only scale down the stateful set and then scale it back up again.
I think the functionality is certainly useful, and being 'stateful' means that it is explicitly important to be able to target the state of a given one. (If it has corrupted its data in the persistent volume that it is using then it needs to be replaced with a different unit.)

Its something that should go into the potential rework for a Juju pod controller (rather than just using stateful sets).

In the short time, though, because of K8s limitations you can't really just `juju remove-unit app/1` for a deployment.