Nova takes a long time to notice if a VM is shutdown by guest admin

[Daniel Berrange]

In a running guest, if the guest admin does 'shutdown -h now', the VM will completely shutdown. It will take Nova up to 10 minutes to notice this and update its state, during which time it continues to report the VM as running. The only way to deal with this is to also run 'nova delete' on the host.

The core problem is that the nova.compute.manager._sync_states() method which is used to reconcille state. This is set as a periodic task, but is only set to run once every 10 minutes. We need to react to terminating instances much faster than this.

[Daniel Berrange (berrange)]

Originally the state was synchronized every 60 seconds, but this was changed to 600 seconds in this change

commit 1c8ad4553b4b8d404f941c5297e3f6e42c9f7e6a
Author: Jay Pipes <email address hidden>
Date: Sun Feb 12 13:34:14 2012 -0500

    Completes fix for LP #928910 - libvirt performance

Based on the data in that bug, I wouldn't be happy reducing the sync interval back to 60 seconds.

Thus I think we need to explore some way for the virt drivers to notify nova that an instance has changed state. Libvirt has an event notification mechanism which can efficiently get this data. We just need to figure out how to feed it back to the manager

[Daniel Berrange (berrange)]

See also

http://lists.openstack.org/pipermail/openstack-dev/2013-January/004501.html

[Vish Ishaya (vishvananda)]

Fixed by: https://review.openstack.org/#/c/21802/