[Azure] Juju does not clean up the compute resources after the controller destroy

[Vladimir Grevtsev]

= Problem description
Juju (v2.9.29) can create, but can't cleanup its own VM/VNET resources upon controller removal.

= Steps to reproduce

ubuntu@aws:~$ juju bootstrap azure/westeurope --config resource-group-name=JujuController --config=logging-config="<root>=DEBUG" --no-default-model azure-controller

Creating Juju controller "azure-controller" on azure/westeurope
Looking for packaged Juju agent version 2.9.29 for amd64
Located Juju agent version 2.9.29-ubuntu-amd64 at https://jujuagents.blob.core.windows.net/juju-agents/agents/agent/2.9.29/juju-2.9.29-linux-amd64.tgz
Launching controller instance(s) on azure/westeurope...
 - machine-0 (arch=amd64 mem=3.5G cores=1)
Installing Juju agent on bootstrap instance
Fetching Juju Dashboard 0.8.1
Waiting for address
Attempting to connect to 52.232.109.238:22
Attempting to connect to 192.168.16.4:22
Connected to 52.232.109.238
Running machine configuration script...
Bootstrap agent now started
Contacting Juju controller at 192.168.16.4 to verify accessibility...

Bootstrap complete, controller "azure-controller" is now available
Controller machines are in the "controller" model

Now you can run
 juju add-model <model-name>
to create a new model to deploy workloads.

# check the resources are in-place
ubuntu@aws:~$ az resource list --resource-group JujuController -o table
Name ResourceGroup Location Type Status
--------------------- --------------- ---------- --------------------------------------- --------
juju-controller JujuController westeurope Microsoft.Compute/availabilitySets
machine-0 JujuController westeurope Microsoft.Compute/virtualMachines
machine-0-primary JujuController westeurope Microsoft.Network/networkInterfaces
juju-internal-nsg JujuController westeurope Microsoft.Network/networkSecurityGroups
machine-0-public-ip JujuController westeurope Microsoft.Network/publicIPAddresses
juju-internal-network JujuController westeurope Microsoft.Network/virtualNetworks

ubuntu@aws:~$ juju destroy-controller azure-controller
WARNING! This command will destroy the "azure-controller" controller.
This includes all machines, applications, data and other resources.

Continue? (y/N):y
Destroying controller
Waiting for hosted model resources to be reclaimed
All hosted models reclaimed, cleaning up controller machines

# but resources are still there!
ubuntu@aws:~$ sleep 300; az resource list --resource-group JujuController -o table
Name ResourceGroup Location Type Status
--------------------- --------------- ---------- --------------------------------------- --------
juju-controller JujuController westeurope Microsoft.Compute/availabilitySets
machine-0 JujuController westeurope Microsoft.Compute/virtualMachines
machine-0-primary JujuController westeurope Microsoft.Network/networkInterfaces
juju-internal-nsg JujuController westeurope Microsoft.Network/networkSecurityGroups
machine-0-public-ip JujuController westeurope Microsoft.Network/publicIPAddresses
juju-internal-network JujuController westeurope Microsoft.Network/virtualNetworks

[John A Meinel (jameinel)]

I'm pretty sure Ian can give more detail here. I thought that if you force
the ResourceGroup, then Juju intentionally doesn't clean them up. Because
you're likely sharing that group with other resources consumed by other
applications (that was how others have used it in the past).
If you *don't* supply `--resource-group` then juju should create a UUID
based resource group and tear it down when the controller is torn down.

On Mon, May 16, 2022 at 4:40 PM Vladimir Grevtsev <
<email address hidden>> wrote:

> Public bug reported:
>
> = Problem description
> Juju (v2.9.29) can create, but can't cleanup its own VM/VNET resources
> upon controller removal.
>
> = Steps to reproduce
>
> ubuntu@aws:~$ juju bootstrap azure/westeurope --config resource-group-
> name=JujuController --config=logging-config="<root>=DEBUG" --no-default-
> model azure-controller
>
> Creating Juju controller "azure-controller" on azure/westeurope
> Looking for packaged Juju agent version 2.9.29 for amd64
> Located Juju agent version 2.9.29-ubuntu-amd64 at
> https://jujuagents.blob.core.windows.net/juju-agents/agents/agent/2.9.29/juju-2.9.29-linux-amd64.tgz
> Launching controller instance(s) on azure/westeurope...
> - machine-0 (arch=amd64 mem=3.5G cores=1)
> Installing Juju agent on bootstrap instance
> Fetching Juju Dashboard 0.8.1
> Waiting for address
> Attempting to connect to 52.232.109.238:22
> Attempting to connect to 192.168.16.4:22
> Connected to 52.232.109.238
> Running machine configuration script...
> Bootstrap agent now started
> Contacting Juju controller at 192.168.16.4 to verify accessibility...
>
> Bootstrap complete, controller "azure-controller" is now available
> Controller machines are in the "controller" model
>
> Now you can run
> juju add-model <model-name>
> to create a new model to deploy workloads.
>
> # check the resources are in-place
> ubuntu@aws:~$ az resource list --resource-group JujuController -o table
> Name ResourceGroup Location Type
> Status
> --------------------- --------------- ----------
> --------------------------------------- --------
> juju-controller JujuController westeurope
> Microsoft.Compute/availabilitySets
> machine-0 JujuController westeurope
> Microsoft.Compute/virtualMachines
> machine-0-primary JujuController westeurope
> Microsoft.Network/networkInterfaces
> juju-internal-nsg JujuController westeurope
> Microsoft.Network/networkSecurityGroups
> machine-0-public-ip JujuController westeurope
> Microsoft.Network/publicIPAddresses
> juju-internal-network JujuController westeurope
> Microsoft.Network/virtualNetworks
>
> ubuntu@aws:~$ juju destroy-controller azure-controller
> WARNING! This command will destroy the "azure-controller" controller.
> This includes all machines, applications, data and other resources.
>
> Continue? (y/N):y
> Destroying controller
> Waiting for hosted model resources to be reclaimed
> All hosted models reclaimed, cleaning up controller machines
>
> # but resources are still there!
> ubuntu@aws:~$ sleep 300; az resource list --resource-group JujuController
> -o table
> Name ...

[Ian Booth (wallyworld)]

This is currently a known limitation with BYO resource groups and destroy controller

https://discourse.charmhub.io/t/juju-2-9-new-azure-features-disk-encryption-byo-vnets-public-ip-constraint/4030

Juju must not delete the resource group, just remove the relevant contents applicable to just the model(s) being removed. We'll need to look into what's needed to remove the current limitation. Until then, you need to manually remove any remaining resources.

[Vladimir Grevtsev (vlgrevtsev)]

Hi Ian,

Can we have this at least documented in the docs (not in some Discourse post) as a known limitation for the time being?