Can't destroy kubernetes model

[Kenneth Koski]

I am trying to destroy a model deployed on top of microk8s. When I attempt to destroy it, I get these two applications hanging around (out of a larger bundle), preventing model destruction:

Model Controller Cloud/Region Version SLA Timestamp Notes
kubeflow uk8s microk8s/localhost 2.7.6 unsupported 17:32:58-05:00 attempt 22 to destroy model failed (will retry): model not empty, found 2 applications (model not empty)

App Version Status Scale Charm Store Rev OS Address Notes
dex-auth active 0/1 dex-auth local 0 kubernetes 10.152.183.181
oidc-gatekeeper active 0/1 oidc-gatekeeper local 0 kubernetes 10.152.183.158

Relation provider Requirer Interface Type Message
oidc-gatekeeper:oidc-client dex-auth:oidc-client oidc-client regular

Tailing logs from the controller shows this message repeated quite a bit:

88c65c78-b955-4b25-8a8b-cf670baddca2: controller-0 2020-06-24 22:31:19 ERROR juju.worker.dependency engine.go:671 "caas-unit-provisioner" manifold worker returned unexpected error: cannot set scale for application "oidc-gatekeeper" to 1: application is not found or not alive
2020-06-24 22:31:55 ERROR juju.worker.dependency engine.go:671 "caas-unit-provisioner" manifold worker returned unexpected error: cannot set scale for application "oidc-gatekeeper" to 1: application is not found or not alive

[Ian Booth (wallyworld)]

I think this might be fixed in 2.8

Do you still see the issue using 2.8.1?

[Kenneth Koski (knkski)]

I've run into the same issue on 2.8.1. Those same two charms are refusing to go away, even if I try `juju remove-application --force --no-wait dex-auth oidc-gatekeeper`. I've attached the output from `juju debug-log`.

[Ian Booth (wallyworld)]

We also need the juju dump-model output to see what's stuck.
Also kubectl get all,pvc,pv

[Kenneth Koski (knkski)]

Get all in regular format

[Kenneth Koski (knkski)]

get all in YAML format

[Kenneth Koski (knkski)]

juju dump-model

[Kenneth Koski (knkski)]

And juju status:

$ juju status --relations
Model Controller Cloud/Region Version SLA Timestamp Notes
kubeflow uk8s microk8s/localhost 2.7.7 unsupported 08:51:04-05:00 attempt 36 to destroy model failed (will retry): model not empty, found 2 applications (model not empty)

App Version Status Scale Charm Store Rev OS Address Notes
dex-auth active 0/1 dex-auth local 0 kubernetes 10.152.183.12
oidc-gatekeeper active 0/1 oidc-gatekeeper local 0 kubernetes 10.152.183.51

Relation provider Requirer Interface Type Message
oidc-gatekeeper:oidc-client dex-auth:oidc-client oidc-client regular

[Kenneth Koski (knkski)]

microk8s inspection report