destroy-controller fails on microk8s

[Tom Haddon]

I'm running the following versions:

$ snap list microk8s
Name Version Rev Tracking Publisher Notes
microk8s v1.17.3 1214 stable canonical✓ classic
$ snap list juju
Name Version Rev Tracking Publisher Notes
juju 2.8-beta1+develop-4cd46ad 10782 edge canonical✓ classic

I've created a k8s controller, and deployed a charm into the model. I then saw a config-changed hook error, so decided to destroy the controller (I'm iterating on developing a charm locally). However, it's still running approximately 15 minutes later:

$ juju status -m is-pd-bot
Model Controller Cloud/Region Version SLA Timestamp
is-pd-bot micro microk8s/localhost 2.8-beta1 unsupported 13:33:45+01:00

App Version Status Scale Charm Store Rev OS Address Notes
pd-bot error 1 pd-bot local 0 kubernetes hook failed: "config-changed"

Unit Workload Agent Address Ports Message
pd-bot/0* error idle hook failed: "config-changed"

$ juju destroy-model is-pd-bot
WARNING! This command will destroy the "is-pd-bot" model.
This includes all containers, applications, data and other resources.

Continue [y/N]? y
Destroying model
Waiting for model to be removed, 1 application(s)...............................
................................................................................
................................................................................
................................................................................
................................................................................
...........................................

Checking microk8s the operator pod is still running:

$ microk8s.kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
container-registry registry-d7d7c8bc9-twt24 1/1 Running 4 2d
controller-micro controller-0 2/2 Running 1 25m
is-pd-bot pd-bot-operator-0 1/1 Running 0 15m
kube-system coredns-7b67f9f8c-85qg9 1/1 Running 5 2d2h
kube-system hostpath-provisioner-7b9cb5cdb4-spxrg 1/1 Running 4 2d

[Tom Haddon (mthaddon)]

Running juju status in a separate window I see "attempt 1 to destroy model failed (will retry): model not empty, found 1 application (model not empty)", fwiw.

[Tom Haddon (mthaddon)]

And now the command completed "ERROR timeout after 30m0s timeout" and juju status' notes still reports "attempt 1 to destroy model failed (will retry): model not empty, found 1 application (model not empty)"

[Tom Haddon (mthaddon)]

Eventually this model was deleted. I was making live changes to the pod to debug my config-changed errors, and I *think* when I actually fixed those to the point where the pod wasn't erroring any more, it was immediately terminated.

I span up another model got that to the point were the charm was running without any errors, and I was able to destroy the model without problems this time.

So perhaps this blocks on a failed hook?

[Ian Booth (wallyworld)]

juju will block if there's any hook errors etc as it tries to allow units to terminate gracefully, running the cleanup steps such as terminating relations etc.

If there's an error, you need to use --force to tell juju to try cleanly, but if that fails, continue anyway.

I'll mark this as Invalid on that basis - please feel free to re-open if necessary.

[Tom Haddon (mthaddon)]

Could the "note" field say something more useful? Currently it says "attempt 1 to destroy model failed (will retry): model not empty, found 1 application (model not empty)" but this wasn't obvious to me that it was blocking on a failed hook.

[Ian Booth (wallyworld)]

We can look to add a bit of extra info to the model destroy progress messages. It's not totally trivial since the api used by the destroy command only feeds back back counts of remaining apps, machines etc. So we'll need to rev the api version and change data formats. juju status will of course surface any model related errors so that's one place to look in the meantime.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.