juju destroy-model of the k8s model hangs

[Vladimir Grevtsev]

Juju 2.8-rc1, snap 2.8.1+2.8-8e75b38

tl;dr: Juju cannot clean up a created k8s resources if something goes wrong during the pod deployment process.

So, I did a "juju add-k8s o7k-k8s" before, then:

ubuntu@OrangeBox84:~/charm-k8s-prometheus$ juju add-model k8s3 o7k-k8s
Added 'k8s3' model on o7k-k8s/default with credential 'o7k-k8s' for user 'admin'
ubuntu@OrangeBox84:~/charm-k8s-prometheus$ juju deploy . --resource prometheus-image=prom/prometheus:latest
Deploying charm "local:kubernetes/prometheus-0".

# ok so far
ubuntu@OrangeBox84:~/charm-k8s-prometheus$ j status
Model Controller Cloud/Region Version SLA Timestamp
k8s3 foundations-maas o7k-k8s/default 2.8-rc1 unsupported 19:16:31Z

App Version Status Scale Charm Store Rev OS Address Notes
prometheus allocating 0/1 prometheus local 0 kubernetes Successfully assigned k8s3/prometheus-operator-0 to juju-6d7577-k8s-9

Unit Workload Agent Address Ports Message
prometheus/0 waiting allocating agent initializing

# but after some time unit is failing:

ubuntu@OrangeBox84:~/charm-k8s-prometheus$ j status
Model Controller Cloud/Region Version SLA Timestamp
k8s3 foundations-maas o7k-k8s/default 2.8-rc1 unsupported 19:36:04Z

App Version Status Scale Charm Store Rev OS Address Notes
prometheus allocating 0/1 prometheus local 0 kubernetes Error: ImagePullBackOff

Unit Workload Agent Address Ports Message
prometheus/0 waiting allocating agent initializing

# so I'm trying to remove the model...
ubuntu@OrangeBox84:~/charm-k8s-prometheus$ j destroy-model test-k8s-workload --destroy-storage
WARNING! This command will destroy the "test-k8s-workload" model.
This includes all containers, applications, data and other resources.

Continue [y/N]? y
Destroying model
Waiting for model to be removed, 1 application(s), 1 volume(s), 1 filesystems(s)....
Waiting for model to be removed, 1 application(s)...............................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
...............................................................ERROR timeout after 30m0s timeout

# and k8s resources were not cleaned up...
ubuntu@OrangeBox84:~/charm-k8s-prometheus$ kubectl get pods -n k8s3
NAME READY STATUS RESTARTS AGE
prometheus-operator-0 0/1 ImagePullBackOff 0 23m

# even from the previous deployment
ubuntu@OrangeBox84:~/charm-k8s-prometheus$ kubectl get pods -n test-k8s-workload
NAME READY STATUS RESTARTS AGE
prometheus-operator-0 0/1 ImagePullBackOff 0 131m

[Ian Booth (wallyworld)]

When something goes wrong and agents aren't running to destroy things cleanly, the --force option is the option to use.

juju destroy-model --destroy-storage --force

Can you please try and and re-open if it doesn't work. it may take at least a minute for it to allow things to try and complete normally before it forces the removal.

[Vladimir Grevtsev (vlgrevtsev)]

Well, --force makes difference sometimes, but not this time:

ubuntu@ubuntu:~/Canonical/charm-k8s-prometheus$ juju bootstrap microk8s mk8s
Creating Juju controller "mk8s" on microk8s/localhost
Creating k8s resources for controller "controller-mk8s"
Downloading images
Starting controller pod
Bootstrap agent now started
Contacting Juju controller at 10.152.183.180 to verify accessibility...

Bootstrap complete, controller "mk8s" is now available in namespace "controller-mk8s"

Now you can run
 juju add-model <model-name>
to create a new model to deploy k8s workloads.
ubuntu@ubuntu:~/Canonical/charm-k8s-prometheus$ juju deploy . --resource prometheus-image=prom/prometheus:latest
Deploying charm "local:kubernetes/prometheus-0".
ubuntu@ubuntu:~/Canonical/charm-k8s-prometheus$ juju status
Model Controller Cloud/Region Version SLA Timestamp
controller mk8s microk8s/localhost 2.8.1 unsupported 17:39:06+03:00

App Version Status Scale Charm Store Rev OS Address Notes
prometheus error 1 prometheus local 0 kubernetes hook failed: "start"

Unit Workload Agent Address Ports Message
prometheus/0* error idle hook failed: "start"

ubuntu@ubuntu:~/Canonical/charm-k8s-prometheus$ juju destroy-model controller
WARNING! This command will destroy the "controller" model.
This includes all containers, applications, data and other resources.

Continue [y/N]? y
Destroying modelERROR cannot destroy model "controller"

The model has persistent storage remaining:
 1 volume and 1 filesystem

To destroy the storage, run the destroy-model
command again with the "--destroy-storage" option.

To release the storage from Juju's management
without destroying it, use the "--release-storage"
option instead. The storage can then be imported
into another Juju model.

ubuntu@ubuntu:~/Canonical/charm-k8s-prometheus$ juju destroy-model controller --destroy-storage --force
WARNING! This command will destroy the "controller" model.
This includes all containers, applications, data and other resources.

Continue [y/N]? y
Destroying model
Waiting for model to be removed, 1 application(s), 1 volume(s), 1 filesystems(s)....
Waiting for model to be removed, 1 application(s), 1 volume(s)....
Waiting for model to be removed, 1 application(s)...............................
.................
Waiting for model to be removed.................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
............................................................

[Ian Booth (wallyworld)]

Can you share the charm so we can try to reproduce?
The charms we've been using don't seem to produce the problem.

[Vladimir Grevtsev (vlgrevtsev)]

Ok, so for the moment I'm not able to reproduce it. I will mark this issue as Invalid for the moment and reopen in case it will start to fire again.

Thanks, Ian!