destroy-model does not finish, storage active

[Marian Gasparovic]

While testing maas-anvil I encountered this issue, it happens every single time

anvil deployed, juju status

Model Controller Cloud/Region Version SLA Timestamp
controller anvil-controller maas/default 3.4.3 unsupported 12:08:42Z

App Version Status Scale Charm Channel Rev Exposed Message
controller active 1 juju-controller 3.4/stable 101 no
haproxy active 3 haproxy latest/stable 75 no Unit is ready
maas-agent 35359 active 3 maas-agent latest/edge 9 no
maas-region 35359 active 3 maas-region latest/edge 9 no
postgresql 14.10 active 3 postgresql 14/stable 363 no

Unit Workload Agent Machine Public address Ports Message
controller/0* active idle 0 10.244.40.30
haproxy/0* active idle 0 10.244.40.30 80-81/tcp Unit is ready
haproxy/1 active idle 1 10.244.40.31 80-81/tcp Unit is ready
haproxy/2 active idle 2 10.244.40.32 80-81/tcp Unit is ready
maas-agent/0* active idle 0 10.244.40.30
maas-agent/1 active idle 1 10.244.40.31
maas-agent/2 active idle 2 10.244.40.32
maas-region/0* active idle 0 10.244.40.30 53,3128,5239-5247,5250-5274,5280-5284,5443,8000/tcp 53,67,69,123,323,5241-5247/udp
maas-region/1 active idle 1 10.244.40.31 53,3128,5239-5247,5250-5274,5280-5284,5443,8000/tcp 53,67,69,123,323,5241-5247/udp
maas-region/2 active idle 2 10.244.40.32 53,3128,5239-5247,5250-5274,5280-5284,5443,8000/tcp 53,67,69,123,323,5241-5247/udp
postgresql/0* active idle 0 10.244.40.30 5432/tcp Primary
postgresql/1 active idle 1 10.244.40.31 5432/tcp
postgresql/2 active idle 2 10.244.40.32 5432/tcp

Machine State Address Inst id Base AZ Message
0 started 10.244.40.30 manual: ubuntu@22.04 Manually provisioned machine
1 started 10.244.40.31 manual:10.244.40.31 ubuntu@22.04 Manually provisioned machine
2 started 10.244.40.32 manual:10.244.40.32 ubuntu@22.04 Manually provisioned machine

model-destroy

$ juju destroy-model admin/controller
WARNING This command will destroy the "admin/controller" model and all its resources. It cannot be stopped.

 - 2 machines will be destroyed
  - machine list: "0 (manual:)" "1 (manual:10.244.40.31)" "2 (manual:10.244.40.32)"
 - 5 applications will be removed
  - application list: "controller" "haproxy" "maas-agent" "maas-region" "postgresql"
 - 3 filesystems and 0 volume will be destroyed

To continue, enter the name of the model to be destroyed: admin/controller
Destroying model
Waiting for model to be removed, 2 machine(s), 5 application(s), 3 filesystems(s)...........
Waiting for model to be removed, 2 machine(s), 4 application(s), 3 filesystems(s)..................................
Waiting for model to be removed, 1 machine(s), 4 application(s), 2 filesystems(s).....................
Waiting for model to be removed, 1 machine(s), 3 application(s), 2 filesystems(s).........
Waiting for model to be removed, 1 machine(s), 2 application(s), 2 filesystems(s)...
Waiting for model to be removed, 1 machine(s), 1 application(s), 2 filesystems(s)....
Waiting for model to be removed, 1 application(s), 1 filesystems(s)...........
Waiting for model to be removed, 1 filesystems(s)...............................
................................................................................
........................

never finishes

juju status

$ juju status
Model Controller Cloud/Region Version SLA Timestamp Notes
controller anvil-controller maas/default 3.4.3 unsupported 12:16:51Z attempt 8 to destroy model failed (will retry): model not empty, found 1 machine, 1 filesystem (model not empty)

Machine State Address Inst id Base AZ Message
0 started 10.244.40.30 manual: ubuntu@22.04 Manually provisioned machine

$ juju storage
Unit Storage ID Type Pool Size Status Message
      pgdata/0 filesystem rootfs 124 GiB attached

$ juju show-storage pgdata/0
pgdata/0:
  kind: filesystem
  life: alive
  status:
    current: attached
    since: 17 Jun 2024 11:27:32Z
  persistent: false

$ mount|grep postg
/dev/sda1 on /var/snap/charmed-postgresql/common type ext4 (rw,relatime,discard,errors=remount-ro)
/var/lib/snapd/snaps/charmed-postgresql_96.snap on /snap/charmed-postgresql/96 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
nsfs on /run/snapd/ns/charmed-postgresql.mnt type nsfs (rw)