Unable to upgrade-charm pod_spec charm to sidecar charm

[Stone Preston]

I am running into the same issue as this bug: https://bugs.launchpad.net/juju/+bug/1928778 that was fixed. Also related are https://bugs.launchpad.net/juju/+bug/1881740 and https://bugs.launchpad.net/juju/+bug/1977582

When trying to upgrade from a podspec k8s charm to a sidecar charm, the old unit gets stuck in a lost state and cannot be removed.

Note that I am using CK revision 814 (which is 1.22), and upgrading the coredns charm from rev 10 to rev 18 (latest/stable on charmhub)

Here are some psuedo commands to replicate the issue:

juju add-model k8s-22

juju deploy cs:charmed-kubernetes-814 --trust --debug

juju scp kubernetes-master/0:config ~/.kube/config

juju config kubernetes-master dns-provider=none

juju add-k8s my-k8s-cloud --controller my-controller --skip-storage

juju add-model my-model my-k8s-cloud

juju deploy cs:~containers/coredns-10

juju offer coredns:dns-provider
juju consume -m k8s-22 coredns-model.coredns
juju relate -m k8s-22 coredns kubernetes-master

juju trust coredns --scope=cluster # trust is necessary in the sidecar charm
juju upgrade-charm coredns --switch coredns --channel=stable

units look like this after the upgrade:
Unit Workload Agent Address Ports Message
coredns/0* waiting idle 10.1.95.66 Waiting for leadership
coredns/2 unknown lost 10.1.95.65 53/UDP,53/TCP,9153/TCP agent lost, see 'juju show-status-log coredns/2'

coredns/2 seems impossible to get rid of