I ran into a situation where an instance starting up and getting through the whole spawning process, was actually left in BUILD and never recovered:
2013-10-18 14:28:27.781 2566 DEBUG nova.virt.libvirt.driver [req-50162f97-7093-4373-a26c-8462f7077b61 57975150431487 50706474460443] [instance: f07617b9-9fd0-40ce-9459-71dba09566ee] Instance is running spawn /usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py:2072
2013-10-18 14:28:27.787 2566 INFO nova.virt.libvirt.driver [-] [instance: f07617b9-9fd0-40ce-9459-71dba09566ee] Instance spawned successfully.
2013-10-18 14:28:27.794 2566 DEBUG nova.compute.manager [req-50162f97-7093-4373-a26c-8462f7077b61 57975150431487 50706474460443] [instance: f07617b9-9fd0-40ce-9459-71dba09566ee] Checking state _get_power_state /usr/lib/python2.7/dist-packages/nova/compute/manager.py:801
2013-10-18 14:28:27.826 2566 WARNING nova.openstack.common.db.sqlalchemy.session [req-50162f97-7093-4373-a26c-8462f7077b61 57975150431487 50706474460443] Got mysql server has gone away: (2013, 'Lost connection to MySQL server during query')
It looks like this query was never retried and the result is inconsistent. The instance itself is running, getting all the traffic, disk files are periodically checked by nova-compute, but the state is not changing. Restarting nova-compute doesn't correct the status either:
2013-10-18 15:13:51.635 21590 DEBUG nova.compute.manager [req-d438f27e-13a8-4a59-b238-66b2227e4415 None None] [instance: f07617b9-9fd0-40ce-9459-71dba09566ee] Checking state _get_power_state /usr/lib/python2.7/dist-packages/nova/compute/manager.py:801
2013-10-18 15:13:51.642 21590 DEBUG nova.compute.manager [req-d438f27e-13a8-4a59-b238-66b2227e4415 None None] [instance: f07617b9-9fd0-40ce-9459-71dba09566ee] Current state is 1, state in DB is 1. _init_instance /usr/lib/python2.7/dist-packages/nova/compute/manager.py:702
Mysql connection disappeared only for a moment - the next query reconnected / worked without any issues.
Check out the blueprint here: https:/