Destroying model fails initially, but works a second time

[Kenneth Koski]

I tried to destroy a model, and it initially failed:

$ juju destroy-model kubeflow --yes --destroy-storage --force
Destroying model
Waiting for model to be removed, 6 application(s), 2 volume(s), 2 filesystems(s)....
Waiting for model to be removed, 2 volume(s), 2 filesystems(s)................^C

If I wait longer, I just get more dots. However, cancelling the first operation and rerunning it works:

$ juju destroy-model kubeflow --yes --destroy-storage --force
Destroying model
Waiting for model to be removed, 2 volume(s), 2 filesystems(s)...
Waiting for model to be removed..............
Model destroyed.

juju debug-log shows a bunch of errors like this for each application during the failed model destruction, and no errors during the successful destruction:

application-foo: 10:30:05 ERROR juju.worker.dependency "operator" manifold worker returned unexpected error: failed to initialize caasoperator for "foo": application "foo" not found

[Kenneth Koski (knkski)]

Here's the YAML for the K8s namespace that's getting destroyed by this command:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    juju.io/controller: ...
    juju.io/model: ...
  creationTimestamp: "2019-08-07T21:07:11Z"
  name: kubeflow
  resourceVersion: "2208"
  selfLink: /api/v1/namespaces/kubeflow
  uid: ...
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

Here's the output from `juju status`:

Model Controller Cloud/Region Version SLA Timestamp Notes
kubeflow cdkkf cdkkf/us-east-1 2.6.7 unsupported 16:26:02-05:00 attempt 31 to destroy model failed (will retry): model not empty, found 2 volumes, 2 filesystems (model not empty)

Model "admin/kubeflow" is empty.

Here's the resources left in the namespace:

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/database-2ee3af8e-katib-db-0 Bound pvc-2818d654-2ff9-4ef2-9ea3-7da578a1c178 1Gi RWO juju-operator-storage 14m
persistentvolumeclaim/database-d6c8b0ed-mariadb-0 Bound pvc-225cd47c-e8a0-4ebb-93d6-e9eadd088dee 1Gi RWO juju-operator-storage 14m

NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-225cd47c-e8a0-4ebb-93d6-e9eadd088dee 1Gi RWO Delete Bound kubeflow/database-d6c8b0ed-mariadb-0 juju-operator-storage 14m
persistentvolume/pvc-2818d654-2ff9-4ef2-9ea3-7da578a1c178 1Gi RWO Delete Bound kubeflow/database-2ee3af8e-katib-db-0 juju-operator-storage 14m

I tried deleting the PV/PVCs, which did not make the deletion process work, and didn't destroy the Juju storage resources:

storage:
  database/0:
    kind: filesystem
    life: alive
    status:
      current: detached
      since: 07 Aug 2019 16:18:17-05:00
    persistent: false
  database/1:
    kind: filesystem
    life: alive
    status:
      current: detached
      since: 07 Aug 2019 16:18:16-05:00
    persistent: false
filesystems:
  "0":
    provider-id: 2818d654-2ff9-4ef2-9ea3-7da578a1c178
    volume: "0"
    storage: database/0
    attachments: null
    pool: kubernetes
    size: 32
    life: alive
    status:
      current: detached
      since: 07 Aug 2019 16:18:17-05:00
  "1":
    provider-id: 225cd47c-e8a0-4ebb-93d6-e9eadd088dee
    volume: "1"
    storage: database/1
    attachments: null
    pool: kubernetes
    size: 31
    life: alive
    status:
      current: detached
      since: 07 Aug 2019 16:18:16-05:00
volumes:
  "0":
    provider-id: pvc-2818d654-2ff9-4ef2-9ea3-7da578a1c178
    storage: database/0
    pool: kubernetes
    size: 856
    persistent: false
    life: alive
    status:
      current: detached
      since: 07 Aug 2019 16:18:17-05:00
  "1":
    provider-id: pvc-225cd47c-e8a0-4ebb-93d6-e9eadd088dee
    storage: database/1
    pool: kubernetes
    size: 855
    persistent: false
    life: alive
    status:
      c...

[Yang Kelvin Liu (kelvin.liu)]

a PR to fix it https://github.com/juju/juju/pull/10521, and will be released in 2.6.7