destroy/add model actions are racy

[Peter Jose De Sousa]

Hello,

When destroying a model on 3.5.1 there is a risk that the model still does not cleanly remove from mongo db:
2024-06-13-13:34:37 root DEBUG Destroying model
2024-06-13-13:34:37 root DEBUG Waiting for model to be removed, 41 machine(s), 75 application(s)...
2024-06-13-13:34:40 root DEBUG .
2024-06-13-13:34:44 root DEBUG .
2024-06-13-13:34:46 root DEBUG .
2024-06-13-13:34:48 root DEBUG .
2024-06-13-13:34:52 root DEBUG .
2024-06-13-13:34:55 root DEBUG .
2024-06-13-13:34:57 root DEBUG .
2024-06-13-13:35:00 root DEBUG .
2024-06-13-13:35:03 root DEBUG .
2024-06-13-13:35:06 root DEBUG .
2024-06-13-13:35:09 root DEBUG .
2024-06-13-13:35:11 root DEBUG .
2024-06-13-13:35:14 root DEBUG .
2024-06-13-13:35:16 root DEBUG .
2024-06-13-13:35:19 root DEBUG .
2024-06-13-13:35:21 root DEBUG .
2024-06-13-13:35:23 root DEBUG .
2024-06-13-13:35:25 root DEBUG .
2024-06-13-13:35:28 root DEBUG Waiting for model to be removed, 40 machine(s), 75 application(s)...
2024-06-13-13:35:30 root DEBUG Waiting for model to be removed, 39 machine(s), 74 application(s)...
2024-06-13-13:35:33 root DEBUG Waiting for model to be removed, 38 machine(s), 74 application(s)...
2024-06-13-13:35:35 root DEBUG .
2024-06-13-13:35:37 root DEBUG .
2024-06-13-13:35:39 root DEBUG Waiting for model to be removed, 36 machine(s), 74 application(s)...
2024-06-13-13:35:42 root DEBUG Waiting for model to be removed, 35 machine(s), 74 application(s)...
2024-06-13-13:35:44 root DEBUG Waiting for model to be removed, 33 machine(s), 74 application(s)...
2024-06-13-13:35:46 root DEBUG Waiting for model to be removed, 31 machine(s), 74 application(s)...
2024-06-13-13:35:49 root DEBUG Waiting for model to be removed, 29 machine(s), 74 application(s)...
2024-06-13-13:35:51 root DEBUG Waiting for model to be removed, 26 machine(s), 74 application(s)...
2024-06-13-13:35:53 root DEBUG Waiting for model to be removed, 25 machine(s), 74 application(s)...
2024-06-13-13:35:55 root DEBUG Waiting for model to be removed, 22 machine(s), 74 application(s)...
2024-06-13-13:35:57 root DEBUG Waiting for model to be removed, 21 machine(s), 74 application(s)...
2024-06-13-13:35:59 root DEBUG Waiting for model to be removed, 17 machine(s), 74 application(s)...
2024-06-13-13:36:02 root DEBUG .
2024-06-13-13:36:04 root DEBUG Waiting for model to be removed, 14 machine(s), 74 application(s)...
2024-06-13-13:36:06 root DEBUG Waiting for model to be removed, 12 machine(s), 74 application(s)...
2024-06-13-13:36:08 root DEBUG Waiting for model to be removed, 10 machine(s), 74 application(s)...
2024-06-13-13:36:10 root DEBUG Waiting for model to be removed, 9 machine(s), 74 application(s)...
2024-06-13-13:36:12 root DEBUG Waiting for model to be removed, 5 machine(s), 74 application(s)...
2024-06-13-13:36:14 root DEBUG Waiting for model to be removed, 4 machine(s), 74 application(s)...
2024-06-13-13:36:16 root DEBUG Waiting for model to be removed, 2 machine(s), 72 application(s)...
2024-06-13-13:36:18 root DEBUG Waiting for model to be removed, 71 application(s)...
2024-06-13-13:36:20 root DEBUG .
2024-06-13-13:36:22 root DEBUG Waiting for model to be removed, 65 application(s)...
2024-06-13-13:36:24 root DEBUG Waiting for model to be removed, 49 application(s)...
2024-06-13-13:36:26 root DEBUG Waiting for model to be removed, 34 application(s)...
2024-06-13-13:36:28 root DEBUG Waiting for model to be removed, 17 application(s)...
2024-06-13-13:36:30 root DEBUG Waiting for model to be removed, 7 application(s)...
2024-06-13-13:36:32 root DEBUG Waiting for model to be removed...
2024-06-13-13:36:34 root DEBUG Model destroyed.
2024-06-13-13:36:36 root DEBUG fce --debug build --layer openstack --skip-validate
2024-06-13-13:36:36 root DEBUG FCE version: 2.21.2+git.23.g5f48a3ce.dirty
2024-06-13-13:36:36 root DEBUG [localhost]: juju export-bundle -m foundations-maas:openstack
2024-06-13-13:36:36 root ERROR [localhost] Command failed: juju export-bundle -m foundations-maas:openstack
2024-06-13-13:36:36 root ERROR 1[localhost] STDOUT follows:
b''
2024-06-13-13:36:36 root ERROR 2[localhost] STDERR follows:

The model was confirmed destroyed by juju -

Immediately a new model was added with the same name (Openstack):
2024-06-13-13:36:36 fce.build INFO Started building layer: openstack
2024-06-13-13:36:39 fce.openstack INFO Starting step: openstack:create_model
2024-06-13-13:36:39 root DEBUG [localhost]: juju status -m foundations-maas:openstack --format=yaml
2024-06-13-13:36:40 root ERROR [localhost] Command failed: juju status -m foundations-maas:openstack --format=yaml
2024-06-13-13:36:40 root ERROR 1[localhost] STDOUT follows:
{}

2024-06-13-13:36:40 root ERROR 2[localhost] STDERR follows:
ERROR model "openstack" has been removed from the controller, run 'juju models' and switch to one of them.

2024-06-13-13:36:40 root DEBUG [localhost]: juju add-model -c foundations-maas openstack maas_cloud
2024-06-13-13:36:41 fce.openstack INFO Finished step: openstack:create_model. Time elapsed (hh:mm:ss.ms): 0:00:01.540080

It succeeds the second time, and the model deployment continues...

The deployment was left to continue and ran to completion, unforunately when I returned a few days later - there were two models, and I could no longer query the older (Correct) model:
 ob60➜ ~ juju models --uuid
Controller: foundations-maas

Model UUID Cloud/Region Type Status Machines Cores Units Access Last connection
controller 02301cf4-e9a6-4b8c-88f1-e87b57c6fbc1 maas_cloud maas available 1 2 1 admin just now
microk8s f4192ed9-978e-4f28-8388-3901a0e0db2c maas_cloud maas available 1 4 1 admin 39 minutes ago
openstack* 4b9bf130-5cc6-4c82-8c71-fa1dfb635f07 maas_cloud maas available 41 37 104 admin 2024-06-13
openstack* 9687a3e2-6b60-40a1-8a08-a66bf8caa6d9 maas_cloud - 0 - - - 2024-06-13
 ob60➜ ~ juju status -m openstack
jujuERROR model cache: model "9687a3e2-6b60-40a1-8a08-a66bf8caa6d9" did not appear in cache timeout

To reproduce:

Add a model (dup-test):

$ juju add-model dup-test

In another pane: while true; do sleep 0.1 && juju add-model dup-test; done <- important - run this before destroy model

$ juju destroy-model dup-test

Expected outout:

ERROR failed to create new model: model "dup-test" for admin already exists (already exists)
ERROR failed to create new model: model "dup-test" for admin already exists (already exists)
ERROR failed to create new model: model "dup-test" for admin already exists (already exists)
ERROR failed to create new model: model "dup-test" for admin already exists (already exists)
ERROR failed to create new model: model "dup-test" for admin already exists (already exists)
ERROR failed to create new model: model "dup-test" for admin already exists (already exists)
ERROR failed to create new model: model "dup-test" for admin already exists (already exists)
Added 'dup-test' model with credential 'maas_cloud_credentials' for user 'admin'
Added 'dup-test' model with credential 'maas_cloud_credentials' for user 'admin' <- Created twice
ERROR failed to create new model: model "dup-test" for admin already exists (already exists)
ERROR failed to create new model: model "dup-test" for admin already exists (already exists)
ERROR failed to create new model: model "dup-test" for admin already exists (already exists)
ERROR failed to create new model: model "dup-test" for admin already exists (already exists)
ERROR failed to create new model: model "dup-test" for admin already exists (already exists)

[Workaround]

Deploy the model step by step to avoid race condition

Thanks,
Peter

[Harry Pidcock (hpidcock)]

https://github.com/juju/juju/pull/17575

[Peter Jose De Sousa (pjds)]

Thank you @hpidcock