multi-cloud juju controller folder not being removed from vsphere

[Stone Preston]

When using a multi-cloud juju controller running in k8s with a vsphere cloud added to it, the controller folder is left behind in vsphere after destroying the controller (so you end up with an empty controller folder left hanging around)

This does not happen with a standard juju controller running directly in vsphere. In that case, the entire controller folder is removed completely.

To reproduce:
Assuming you have a controller available already, add a model to hold the kubernetes deployment:
juju add-model --config enable-os-refresh-update=false --config enable-os-upgrade=false --config logging-config='<root>=DEBUG' --config datastore=vsanDatastore --config primary-network=YOUR_VLAN my-k8s vsphere/Boston

Deploy CK core 1.25:
juju deploy kubernetes-core --overlay vsphere-overlay.yaml --trust --debug --channel 1.25/stable

The yaml overlay I have been using for reference:
description: Charmed Kubernetes overlay to add native vSphere support.
applications:
  kubernetes-control-plane:
    options:
      allow-privileged: "true"
  containerd:
    options:
      http_proxy: http://squid.internal:3128
      https_proxy: http://squid.internal:3128
      no_proxy: localhost,127.0.0.1,::1,10.246.154.0/24,10.152.183.0/24,192.168.0.0/16
  vsphere-integrator:
    charm: vsphere-integrator
    channel: edge
    num_units: 1
    trust: true
    options:
      datastore: vsanDatastore
      folder: k8s-crew-root
relations:
  - ['vsphere-integrator', 'kubernetes-control-plane']
  - ['vsphere-integrator', 'kubernetes-worker']

Once the model is active/idle, copy the kubeconfig:
juju scp kubernetes-control-plane/0:config ~/.kube/config

Then create a storage class:
kubectl apply -f vsphere-storageclass.yaml

The storage class yaml looks like this:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: mystorage
provisioner: kubernetes.io/vsphere-volume
parameters:
  diskformat: zeroedthick

Now add-k8s:
juju add-k8s vs-repro-k8s --storage mystorage

Bootstrap the controller:
juju bootstrap vs-repro-k8s

Once the boostrap is finished, add a vsphere cloud to the controller using --force
juju add-cloud --c=vs-repro-k8s --force
select vsphere as the cloud type and fill in the relevant cloud information as prompted

Add cloud creds:
juju add-credential your-cloud-added-above --region your-region
fill in the relevant information

Set model defaults:
juju model-defaults vsphere-in-k8s juju-http-proxy=http://squid.internal:3128 apt-http-proxy=http://squid.internal:3128 snap-http-proxy=http://squid.internal:3128 juju-https-proxy=http://squid.internal:3128 apt-https-proxy=http://squid.internal:3128 snap-https-proxy=http://squid.internal:3128 apt-no-proxy=localhost,127.0.0.1,ppa.launchpad.net,launchpad.net juju-no-proxy=localhost,127.0.0.1,0.0.0.0,ppa.launchpad.net,launchpad.net,10.0.8.0/24,10.246.154.0/24

Add a model:
juju add-model --credential your-cred-added-above --config logging-config='<root>=DEBUG' --config datastore=vsanDatastore --config primary-network=YOUR_VLAN vs-repro-model your-cloud-added-above/your-region

Once the model has been added, the controller folder and model subfolder should be visible in the vs UI.

You can delete the model:
juju destroy-model vs-repro-model

and see that the model folder has now been removed.

you can destroy the controller:
juju destroy-controller --destroy-all-models --destroy-storage vs-repro-k8s

However, the controller folder will be left behind after the controller destruction is complete (controller namespace fully removed)

This behavior is not observed for non-multi cloud controllers (ie if you bootstrap a controller in vsphere directly instead of using the one in k8s, then destroy it, the folder is completely removed as expected)