juju destroy-model --force spins forever on k8s model with persistent storage

[Gareth Woolridge]

Attempting to destroy-model --force on a k8s model with persistent storage spins forever:

juju destroy-model discourse --force
WARNING! This command will destroy the "discourse" model.
This includes all containers, applications, data and other resources.

Continue [y/N]? y
Destroying model
Waiting for model to be removed, 1 application(s), 1 volume(s), 1 filesystems(s)..........................................................................................................................................................^Cctrl+c detected, aborting...

However checking juju debug-log -m controller:

controller-0: 14:33:24 WARNING juju.apiserver.common failed destroying model 5c03e992-ce74-48a9-89f0-1cbce2ee550b: failed to destroy model: model contains persistent storage

So list storages, and try to delete manually:

juju list-storage
Unit Storage id Type Pool Size Status Message
discourse-charm/0 discourse-storage/0 filesystem kubernetes 48MiB attached Successfully provisioned volume pvc-60ba8b8b-b388-427d-833a-80b7b0374e47

juju remove-storage discourse-storage/0
ERROR Juju command "remove-storage" not supported on kubernetes models

At this point I asked in #juju and hml suggested --destroy-storage:

juju destroy-model discourse --destroy-storage
WARNING! This command will destroy the "discourse" model.
This includes all containers, applications, data and other resources.

Continue [y/N]? y
Destroying model
Waiting for model to be removed, 1 application(s), 1 volume(s), 1 filesystems(s).....
Waiting for model to be removed, 1 application(s).......................
Waiting for model to be removed......
Model destroyed.

So this works. I can't quite recall why I was using --force, likely as iterating on a new operator charm and needing to kick broken units/models harder on destroy before I added storage to the charm.

Anyway, this feel bug worthy on feedback from --force from an end user perspective if using with k8s and storage, as it spins forever with no feedback of an issue unless you know to look in the controller logs.

[Haw Loeung (hloeung)]

Even on non-k8s, this fails to destroy a model with storage:

| jenkins@juju-59050e-is-mojo-ci-0:~$ juju storage
| Unit Storage id Volume id Provider Id Size State Message
| 0 e7a329f5-c9ba-48bb-8521-a7c2bd680e0a 100GiB detached

[Drew Freiberger (afreiberger)]

Agreeing with Haw, it's not just k8s, but juju-storage usage in general.

It appears that the inclusion of --force in model destruction avoids the path of the code resulting in the following operator notification:

ubuntu@drewn3ss-bastion:~$ juju destroy-model default -y
Destroying modelERROR cannot destroy model "default"

The model has persistent storage remaining:
 3 volumes

To destroy the storage, run the destroy-model
command again with the "--destroy-storage" option.

To release the storage from Juju's management
without destroying it, use the "--release-storage"
option instead. The storage can then be imported
into another Juju model.

This is an openstack model showing the same timeout on destroy-model when --force is used:

ubuntu@drewn3ss-bastion:~$ juju destroy-model default --force --no-wait -y
Destroying model
Waiting for model to be removed, 25 machine(s), 27 application(s), 3 volume(s)..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ERROR timeout after 30m0s timeout

[Ian Booth (wallyworld)]

To try and cover the points raised so far...

Juju is conservative when there's the possibility of deleting people's data, that's why destroying a model with storage requires an explicit decision whether to destroy the volumes or just release them and allow them to be reattached later or otherwise accessed by the user to get any data.

Juju also always attempts to cleanly destroy a model - this means that units have to fully run any relation departed/broken hooks etc, and the storage provioner worker has to be able to cleanly detach any volumes and report back that it has done so. If any of this fails, the model will not get cleaned up.

To allow for cases where a model cannot be destroyed cleanly (eg someone removed an openstack instance or cinder volume outside of juju), the --force option can be used. This will ignore cloud errors and continue with the model destruction. In juju 2.8, some fixes were added to address issues with --force and storage. Previously, --force didn't always work well with storage.

There's ongoing work to improve feedback from model destroy operations, but the current expectation is that --force at least should do what it says on the tin.

I'll mark as incomplete pending any further feedback that there's still an issue with Juju 2.8.0

[Trent Lloyd (lathiat)]

There is still an issue here.

Tested on juju 2.8.3 with the openstack provider

= Scenario 1: non-force destroy =
Test:
juju destroy-model k8s

Result:
You get a surfaced ERROR:
> ERROR cannot destroy model "k8s"
>
> The model has persistent storage remaining:
> 3 volumes
>
> To destroy the storage, run the destroy-model
> command again with the "--destroy-storage" option.

= Scenario 2: destroy with force =

Test:
juju destroy-model k8s --force

Result:
The client just sits there with the message "Waiting for model to be removed, 10 machine(s), 10 application(s), 3 volume(s)..." with no error and no progress made on destroying the model.

But from "juju debug-log -m controller" we see the error:
machine-0: 06:46:26 WARNING juju.apiserver.common failed destroying model a41433cc-ae36-49cf-899e-45bbf5b22786: failed to destroy model: model contains persistent storage

*Expected Results*

The client should surface the failure to destroy error the same as it does without --force. Then the user knows they can re-run with both --force and --destroy storage per Scenario 3 below

= Scenario 3: destroy with force and destroy-storage =

Test:
juju destroy-model k8s --force --destroy-storage

Result:
The model successfully destroys

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.