Unable to redeploy a k8s application with the same name as before

[Tom Haddon]

We have a juju k8s model (2.9.32) attached to an openstack HA controller in which we had a number of applications deployed, including two versions of https://charmhub.io/redis-k8s - named in this model as `redis-cache` and `redis-broker`.

The initial deployment was using pod-spec charms, and we wanted to switch to sidecar charms (which are published in the edge channel of this charm). In local testing, we found that we couldn't just run a charm upgrade, but needed to remove the application and then redeploy.

However, when we came to do this on our staging instance, we found ourselves unable to redeploy the applications. The units are stuck in "installing agent" status and there are no pods deployed in kubernetes.

We have the same issue for both versions of the redis-k8s applications that we've deployed (redis-cache and redis-broker). We can deploy them fine if we use a different application name (e.g. redis-cache2 or redis-broker2).

We have done some live debugging with Joe Phillips (thx!) and I'll attach some engine reports from the controllers for reference.

We're not able to reproduce this yet in another model, but are happy to provide any information or access to the model via a screenshare as needed to help debug.

[Tom Haddon (mthaddon)]

It turned out there was a stale kubernetes service with the name we were trying to deploy with. The reason was that we had a problem deleting the service due to the following error https://paste.ubuntu.com/p/q4XyRRHmWM/. This was fixed by adding the "load-balancer_observer" role to the relevant account, and then the service deletion completed. We could then redeploy the application with the same name.

As discussed with Ian Booth, we think Juju should include a more useful error message explaining why it can't provision the new application (e.g. because there's a k8s service with a conflicting name).