Juju 2.8 not removing custom Resource

[Kenneth Koski]

I have a charm (istio-pilot) that is creating a VirtualService custom resource for each charm that gets related to it over the service-mesh interface. It creates the VirtualService resources via its own pod spec.

The issue is that when the application is removed, the VirtualService isn't cleaned up. It seems that if the pod spec changes to remove a customResource, Juju doesn't detect the lack of that item compared to previous pod spec calls.

[Kenneth Koski (knkski)]

As an example:

$ juju status
...
App Version Status Scale Charm Store Rev OS Address Notes
argo-ui argoui:v2.3.0 active 1 argo-ui local 0 kubernetes 10.152.183.32
istio-ingressgateway docker.io/istio/proxyv2:1.5.0 active 1 istio-ingressgateway local 0 kubernetes 10.64.140.43
istio-pilot docker.io/istio/pilot:1.5.0 active 1 istio-pilot local 0 kubernetes 10.152.183.150
...

$ microk8s kubectl get virtualservices -A
NAMESPACE NAME GATEWAYS HOSTS AGE
kubeflow argo-ui [kubeflow-gateway] [*] 16m

$ juju remove-application argo-ui
removing application argo-ui

$ juju status
...
App Version Status Scale Charm Store Rev OS Address Notes
istio-ingressgateway docker.io/istio/proxyv2:1.5.0 active 1 istio-ingressgateway local 0 kubernetes 10.64.140.43
istio-pilot docker.io/istio/pilot:1.5.0 active 1 istio-pilot local 0 kubernetes 10.152.183.150
...

$ microk8s kubectl get virtualservices -A
NAMESPACE NAME GATEWAYS HOSTS AGE
kubeflow argo-ui [kubeflow-gateway] [*] 18m

[Yang Kelvin Liu (kelvin.liu)]

Hi Ken
Currently logic is all the CR of `virtualservices` will be removed if `istio-pilot` gets removed.
Because the CRD of `virtualservices` was created by `istio-pilot`.
And in the podspec, we don't diff to get deletion of resources then delete the missing resources in new podspec. I think `kubectl apply` doesn't do this as well.

[Pen Gale (pengale)]

Moving to 3.0.0 milestone, as this involves changes that would be better applied to the new container charms, rather than the existing framework.

[Juan M. Tirado (tiradojm)]

This is an old bug. Reopen if required.