Nova takes a long time to notice if a VM is shutdown by guest admin

Bug #1099761 reported by Daniel Berrange
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Daniel Berrange

Bug Description

In a running guest, if the guest admin does 'shutdown -h now', the VM will completely shutdown. It will take Nova up to 10 minutes to notice this and update its state, during which time it continues to report the VM as running. The only way to deal with this is to also run 'nova delete' on the host.

The core problem is that the nova.compute.manager._sync_states() method which is used to reconcille state. This is set as a periodic task, but is only set to run once every 10 minutes. We need to react to terminating instances much faster than this.

Revision history for this message
Daniel Berrange (berrange) wrote :

Originally the state was synchronized every 60 seconds, but this was changed to 600 seconds in this change

commit 1c8ad4553b4b8d404f941c5297e3f6e42c9f7e6a
Author: Jay Pipes <email address hidden>
Date: Sun Feb 12 13:34:14 2012 -0500

    Completes fix for LP #928910 - libvirt performance

Based on the data in that bug, I wouldn't be happy reducing the sync interval back to 60 seconds.

Thus I think we need to explore some way for the virt drivers to notify nova that an instance has changed state. Libvirt has an event notification mechanism which can efficiently get this data. We just need to figure out how to feed it back to the manager

Revision history for this message
Daniel Berrange (berrange) wrote :
Revision history for this message
Vish Ishaya (vishvananda) wrote :
Changed in nova:
status: New → Fix Committed
importance: Undecided → Medium
assignee: nobody → Daniel Berrange (berrange)
milestone: none → grizzly-3
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-3 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers