Status does not update to "Shutoff" when instance shuts down itself
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Medium
|
Matt Riedemann | ||
Queens |
Fix Committed
|
Medium
|
Matt Riedemann |
Bug Description
For instances that shut down themselves, the status remains Active, even though the power state updates to Shutdown. If however the shutdown signal is sent via openstack it does change the status to "SHUTOFF".
After creating an instance, the states are:
+------
| Field | Value |
+------
| OS-EXT-
| OS-EXT-
| OS-EXT-STS:vm_state | active |
| status | ACTIVE |
+------
Using openstack server stop results in:
+------
| Field | Value |
+------
| OS-EXT-
| OS-EXT-
| OS-EXT-STS:vm_state | stopped |
| status | SHUTOFF |
+------
However logging into the instance and using the poweroff command results in:
+------
| Field | Value |
+------
| OS-EXT-
| OS-EXT-
| OS-EXT-STS:vm_state | active |
| status | ACTIVE |
+------
This results in being unable to use the openstack server start command on it fails and returns:
# openstack server start test_shutdown
Cannot 'start' instance 8881bebb-
despite the virtual machine being powered off. You can work around this by running openstack server stop, and then openstack server start.
This is also an issue for external applications that check the status (and how I noticed it to begin with).
This on devstack with commit id: 5d2add74534719c
Not sure if the following is useful, but
- Using hypervisor libvirt+qemu/kvm:
# virsh version
Compiled against library: libvirt 3.2.0
Using library: libvirt 3.2.0
Using API: QEMU 3.2.0
Running hypervisor: QEMU 2.9.0
- Using ephemeral storage with ex4
- Neutron with OpenVSwitch
tags: | added: compute |
Changed in nova: | |
status: | Incomplete → Triaged |
importance: | Undecided → Medium |
There is a periodic task in nova-compute controlled by the config option [DEFAULT] sync_power_ state_interval which defaults to 600 seconds (10 minutes). The periodic task will update the power_state in the database to reflect the power state returned by the virt driver. That's the power state update that you see.
Now, looking at that same periodic task, it's also supposed to be calling the compute "stop" API on the instance if the power state from the virt driver is found to be "shutdown". So once that occurs, you should see the instance with a vm_state of "stopped" and a status of SHUTOFF. Do you ever see that after the [DEFAULT] sync_power_ state_interval of 10 minutes?
There should be a DEBUG log message in nova-compute like this when it finds a powered off instance:
"Instance shutdown by itself. Calling the stop API"
and
"Stopping instance; current vm_state: ..."
Do you see those log messages or do you find any ERROR log messages in nova-compute around them?