kayobe overcloud service destroy fails on compute nodes if any VMs are running, after controllers are already gone

[Martin Ananda Boeker]

Environment: kayobe 14.1, openstack 2023.1

service destroy should stop all VMs before starting to destroy all containers, or destroy them regardless of running VMs. Failing half way through after controllers are already destroyed means the `openstack` command can't be used anymore, so you have to ssh to the compute nodes, docker exec into the nova containers, and kill the qemu processes.

[Will Szumski (willjs)]

Good catch and thanks for the workaround. We should investigate moving the check to start of the process to avoid this situation.

[Martin Ananda Boeker (mboeker)]

Why even bother checking for qemu processes? If the user decides that they want to do a `service destroy -and-they-really-mean-it` then shouldn't it be a given that all VMs are going away? Or are they somehow preserved even after all nova and all volumes are gone?

[Will Szumski (willjs)]

> Why even bother checking for qemu processes? If the user decides that they want to do a `service destroy -and-they-really-mean-it` then shouldn't it be a given that all VMs are going away? Or are they somehow preserved even after all nova and all volumes are gone?

It would be useful to have the safety check if you were destroying services on a particular compute node in a production environment, but maybe we could add a flag to destroy all instances (more of a kolla-ansible feature)

[Martin Ananda Boeker (mboeker)]

Valid, especially if only destroying one node. However, that check should
definitely happen before the controllers and everything else are destroyed,
especially if no limit is set, and then force delete everything. Having
nothing left of OpenStack except nova containers is probably not useful in
any situation.

Mit freundlichen Grüßen

Martin Ananda Boeker
EDV-Consult IT GmbH

On Tue, Feb 20, 2024, 18:16 Will Szumski <email address hidden> wrote:

> > Why even bother checking for qemu processes? If the user decides that
> they want to do a `service destroy -and-they-really-mean-it` then
> shouldn't it be a given that all VMs are going away? Or are they somehow
> preserved even after all nova and all volumes are gone?
>
> It would be useful to have the safety check if you were destroying
> services on a particular compute node in a production environment, but
> maybe we could add a flag to destroy all instances (more of a kolla-
> ansible feature)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2050091
>
> Title:
> kayobe overcloud service destroy fails on compute nodes if any VMs are
> running, after controllers are already gone
>
> Status in kayobe:
> Triaged
>
> Bug description:
> Environment: kayobe 14.1, openstack 2023.1
>
> service destroy should stop all VMs before starting to destroy all
> containers, or destroy them regardless of running VMs. Failing half
> way through after controllers are already destroyed means the
> `openstack` command can't be used anymore, so you have to ssh to the
> compute nodes, docker exec into the nova containers, and kill the qemu
> processes.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/kayobe/+bug/2050091/+subscriptions
>
>