task_state stuck in "powering_on" when starting a server and the nova-compute host service is down

[Eric Miller]

Description
===========
After stopping the nova-compute service on a node, powering on VMs that were in a shutoff state, but assigned to the compute node where the compute service is no longer running, are powered on the host where the nova-compute service is stopped, causing the VM task state to be set to a bad state of "powering-on" forever.

Steps to reproduce
==================
Example:
Disable the nova-compute service on host compute001
VM001 was previously running on this host, but is now shut down
Power on VM001
nova-scheduler schedules VM001 on compute001

Expected result
===============
The scheduler filter ComputeFilter should see that compute001 is not "operational and enabled", as described here:
https://docs.openstack.org/ocata/config-reference/compute/schedulers.html#computefilter

and "not" schedule this VM on host compute001.

Actual result
=============
compute001 is chosen to power on VM001, leaving it in a bad task state of "powering-on" forever.

Environment
===========
stable/rocky using Kolla-Ansible 7.0.0.0rc3devXX and Kolla 7.0.0.0rc3devXX.
CentOS 7.5 with latest updates
Kernel: Linux 4.18.14-1.el7.elrepo.x86_64
Hypervisor: KVM
Storage: Ceph
Networking: DVR

[Matt Riedemann (mriedem)]

When you say "Disable the nova-compute service on host compute001" do you mean using the disable API for the compute service, or simply stopping the nova-compute process on the host? If the latter, are you waiting for it's status to e 'down' in the nova service-list output?

And when you say "Power on VM001" what specifically are you doing? Issuing "nova start VM001"? Because starting a stopped VM does nothing with the scheduler.

I think the issue is that the API changes the task state on the server and then RPC casts to the compute service, which is down, and then the task_state is stuck. Since it's an RPC cast, we don't know if the compute is down or not when we make the cast, and the API isn't checking that every operation performed on a VM has the underlying compute service running (except delete, and any move operations).

You can reset the state via the nova reset-state CLI:

https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-reset-state

[Chris Friesen (cbf123)]

Even if the API did check if the compute service was running, it wouldn't fix the problem since the compute service could die (or get taken down) right after sending the RPC cast.

[Eric Miller (erickmiller)]

@Matt - Since we are using Kolla-Ansible to deploy a container-based OpenStack, we stop the docker container that is running the nova-compute process(es), so it is effectively the same as stopping the nova-compute process (gracefully) on a non-containerized OpenStack.

I actually tried both waiting and not waiting for the status to appear as "down" in the nova service-list output. The same result occurred.

"openstack start VM001" was used to power on the VM (equivalent of nova start VM001). I thought that powering on triggered a scheduler event, but I guess I was wrong.

Regarding the nova reset-state, there are a few different states, including "task_state", which is where the "powering-on" state is located. The nova reset-state command resets the "vm_state" field in the instances table in the nova database (unless I'm mistaken - please correct if I'm wrong). Is there a command to reset the "task_state"? I had to do it manually in the database to recover.

Btw, it appears that the "openstack start VM001" command retries quite a few times since I started the nova_compute docker container a few minutes after stopping the container and the VM powered on and the task_state was set appropriately.

Let me re-state the issue.

We were looking for a way to put a host in maintenance mode where 1) no additional VMs could start on the host, 2) we could live migrate powered on VMs off this host, and 3) migrate powered off VMs off this host. So essentially what vSphere's maintenance mode function does.

We thought we could accomplish by:
#1 by stopping the nova compute service, but that doesn't work for VMs already assigned to that host.
#2 scripting, or using the "Migrate Host" option in Horizon after disabling the compute service on the host.
#3 ?? - seems impossible with the current API - the compute node assigned to a VM apparently can't be changed, as far as I have found, with the API/CLI/etc. This seems to be the "node" column in the instances table in the nova database, and can be changed manually, which works, but it seems like there should be a better way with the API/CLI/etc.

I thought that the "openstack host set" command would have everything we needed, and saw that it has --enable/--disable and --enable-maintanance/--disable-maintenance, but the latter seems to be only for Xen, and not KVM? The --enable/--disable doesn't seem to work either. All parameters end with a 400 error. So, my guess is that all of these parameters are for Xen and not KVM?

To re-iterate: #1 likely requires using the "openstack host set --disable <host>" command to disable scheduling of "new" VMs, #2 is possible, and #3 requires manual database changes. Any suggested alternatives?

Shelving doesn't appear to be the right approach, and unshelving powers-on a VM automatically after unshelving (at least from the Horizon interface), so that's no good.

Thanks!

Eric