destroy-model does not finish, storage active

Bug #2069610 reported by Marian Gasparovic
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Invalid
Undecided
Unassigned

Bug Description

While testing maas-anvil I encountered this issue, it happens every single time

anvil deployed, juju status

Model Controller Cloud/Region Version SLA Timestamp
controller anvil-controller maas/default 3.4.3 unsupported 12:08:42Z

App Version Status Scale Charm Channel Rev Exposed Message
controller active 1 juju-controller 3.4/stable 101 no
haproxy active 3 haproxy latest/stable 75 no Unit is ready
maas-agent 35359 active 3 maas-agent latest/edge 9 no
maas-region 35359 active 3 maas-region latest/edge 9 no
postgresql 14.10 active 3 postgresql 14/stable 363 no

Unit Workload Agent Machine Public address Ports Message
controller/0* active idle 0 10.244.40.30
haproxy/0* active idle 0 10.244.40.30 80-81/tcp Unit is ready
haproxy/1 active idle 1 10.244.40.31 80-81/tcp Unit is ready
haproxy/2 active idle 2 10.244.40.32 80-81/tcp Unit is ready
maas-agent/0* active idle 0 10.244.40.30
maas-agent/1 active idle 1 10.244.40.31
maas-agent/2 active idle 2 10.244.40.32
maas-region/0* active idle 0 10.244.40.30 53,3128,5239-5247,5250-5274,5280-5284,5443,8000/tcp 53,67,69,123,323,5241-5247/udp
maas-region/1 active idle 1 10.244.40.31 53,3128,5239-5247,5250-5274,5280-5284,5443,8000/tcp 53,67,69,123,323,5241-5247/udp
maas-region/2 active idle 2 10.244.40.32 53,3128,5239-5247,5250-5274,5280-5284,5443,8000/tcp 53,67,69,123,323,5241-5247/udp
postgresql/0* active idle 0 10.244.40.30 5432/tcp Primary
postgresql/1 active idle 1 10.244.40.31 5432/tcp
postgresql/2 active idle 2 10.244.40.32 5432/tcp

Machine State Address Inst id Base AZ Message
0 started 10.244.40.30 manual: ubuntu@22.04 Manually provisioned machine
1 started 10.244.40.31 manual:10.244.40.31 ubuntu@22.04 Manually provisioned machine
2 started 10.244.40.32 manual:10.244.40.32 ubuntu@22.04 Manually provisioned machine

model-destroy

$ juju destroy-model admin/controller
WARNING This command will destroy the "admin/controller" model and all its resources. It cannot be stopped.

 - 2 machines will be destroyed
  - machine list: "0 (manual:)" "1 (manual:10.244.40.31)" "2 (manual:10.244.40.32)"
 - 5 applications will be removed
  - application list: "controller" "haproxy" "maas-agent" "maas-region" "postgresql"
 - 3 filesystems and 0 volume will be destroyed

To continue, enter the name of the model to be destroyed: admin/controller
Destroying model
Waiting for model to be removed, 2 machine(s), 5 application(s), 3 filesystems(s)...........
Waiting for model to be removed, 2 machine(s), 4 application(s), 3 filesystems(s)..................................
Waiting for model to be removed, 1 machine(s), 4 application(s), 2 filesystems(s).....................
Waiting for model to be removed, 1 machine(s), 3 application(s), 2 filesystems(s).........
Waiting for model to be removed, 1 machine(s), 2 application(s), 2 filesystems(s)...
Waiting for model to be removed, 1 machine(s), 1 application(s), 2 filesystems(s)....
Waiting for model to be removed, 1 application(s), 1 filesystems(s)...........
Waiting for model to be removed, 1 filesystems(s)...............................
................................................................................
........................

never finishes

juju status

$ juju status
Model Controller Cloud/Region Version SLA Timestamp Notes
controller anvil-controller maas/default 3.4.3 unsupported 12:16:51Z attempt 8 to destroy model failed (will retry): model not empty, found 1 machine, 1 filesystem (model not empty)

Machine State Address Inst id Base AZ Message
0 started 10.244.40.30 manual: ubuntu@22.04 Manually provisioned machine

$ juju storage
Unit Storage ID Type Pool Size Status Message
      pgdata/0 filesystem rootfs 124 GiB attached

$ juju show-storage pgdata/0
pgdata/0:
  kind: filesystem
  life: alive
  status:
    current: attached
    since: 17 Jun 2024 11:27:32Z
  persistent: false

$ mount|grep postg
/dev/sda1 on /var/snap/charmed-postgresql/common type ext4 (rw,relatime,discard,errors=remount-ro)
/var/lib/snapd/snaps/charmed-postgresql_96.snap on /snap/charmed-postgresql/96 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
nsfs on /run/snapd/ns/charmed-postgresql.mnt type nsfs (rw)

Revision history for this message
Bayani Carbone (bcarbone) wrote :

facing the same issue, here are the juju debug logs: https://pastebin.canonical.com/p/7STD54kdtX/

Revision history for this message
Joseph Phillips (manadart) wrote :

Part of the issue here is probably that you're destroying the controller model itself.

Is there a reason that "destroy-controller" is not being used?

Changed in juju:
status: New → Incomplete
Revision history for this message
Stamatis Katsaounis (skatsaounis) wrote :

Hi Joseph. I think that we do not have yet a complete cleanup story for maas-anvil. So we are trying to identify the proper way of deleting everything Juju related on the system in order to start again from scratch.

For your info, maas-anvil snap is communicating with Juju snap to do the following:

- create a manual cloud that includes all the machines that are participating in the maas-anvil cluster
- bootstrap juju controller on the first machine
- install postgres, pgbouncer, haproxy, maas-region, maas-agent machine charms at the same model as the controller

So what we want to achieve is to do a complete cleanup when tearing down maas-anvil. This is extremely helpful for quick redeployments. So during this cleanup, we tried to juju destroy the controller model, the only model that maas-anvil Juju manages.

Do you suggest instead of running "juju destroy-model admin/controller" to run "juju destroy-controller"?

If yes, based on https://juju.is/docs/juju/juju-destroy-controller, do we need to run this command "juju destroy-controller --destroy-all-models --destroy-storage"? Does it include any extra step from destroy-model _controller_model_name_?

Thank you,
Stamatis

Revision history for this message
Marian Gasparovic (marosg) wrote :

Joe, it is the only model available

$ juju models
Controller: anvil-controller

Model Cloud/Region Type Status Machines Cores Units Access Last connection
admin/controller* maas/default manual available 3 12 19 admin just now

Changed in juju:
status: Incomplete → New
Revision history for this message
Joseph Phillips (manadart) wrote :

Yes, destroy-controller is the way to go here.

Changed in juju:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.