Build failures are not reflected in server status

[Troy Toman]

A server create failed because of a lack of available memory on the host. But, the server stayed in a BUILD status when it should have been moved to an ERROR status. The following exception was noted during the attempt to change the status:

rt/xenapi/vm_utils.py:493
2011-09-15 21:24:40,889 DEBUG nova.virt.xenapi.vm_utils [-] image_size_bytes=1458287104, allowed_size_bytes=42949672960 from (pid=8316) _check_vdi_size /usr/lib/pymodules/python2.6/nova/virt/xenapi/vm_utils.py:510
2011-09-15 21:24:41,083 ERROR nova.virt.xenapi.vmops [-] instance instance-00010799: not enough free memory
(nova.virt.xenapi.vmops): TRACE: None
(nova.virt.xenapi.vmops): TRACE:
2011-09-15 21:24:41,084 ERROR nova.compute.manager [-] Instance '10799' failed to spawn. Is virtualization enabled in the BIOS? Details: 'module' object has no attribute 'instance_set_state'
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 424, in _run_instance
(nova.compute.manager): TRACE: network_info, block_device_info)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi_conn.py", line 190, in spawn
(nova.compute.manager): TRACE: self._vmops.spawn(context, instance, network_info)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vmops.py", line 149, in spawn
(nova.compute.manager): TRACE: vm_ref = self._create_vm(context, instance, vdis, network_info)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/xenapi/vmops.py", line 176, in _create_vm
(nova.compute.manager): TRACE: power_state.SHUTDOWN)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/db/api.py", line 573, in instance_set_state
(nova.compute.manager): TRACE: return IMPL.instance_set_state(context, instance_id, state, description)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/utils.py", line 452, in __getattr__
(nova.compute.manager): TRACE: return getattr(backend, key)
(nova.compute.manager): TRACE: AttributeError: 'module' object has no attribute 'instance_set_state'
(nova.compute.manager): TRACE:
2011-09-15 21:24:41,205 INFO nova.virt.xenapi [-] Task [Async.host.call_plugin] OpaqueRef:3fbf7df1-3cfa-9078-846b-78568381fd6f status: success <value>{&quot;message&quot;: &quot;&quot;, &quot;returncode&quot;: &quot;0&quot;}</value>

[Aaron Lee (aaron-lee)]

It looks like this has uncovered two problems. The first is that instance_set_state is not defined on the implementation of that interface. The second, and IMHO scarier issue is that an exception in run_instance left the instance in the building status...

[Aaron Lee (aaron-lee)]

Looking for a review: https://review.openstack.org/#change,676

[Thierry Carrez (ttx)]

Do not set to fixcommitted until it's merged in trunk.

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/676
Committed: http://github.com/openstack/nova/commit/7dba1d9aa989760b190f1cf3bad2ed22bb2e2fc5
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 7dba1d9aa989760b190f1cf3bad2ed22bb2e2fc5
Author: Aaron Lee <email address hidden>
Date: Mon Sep 26 18:22:03 2011 -0500

    Raise InsufficientFreeMemory

    Kind of fixes bug 851374 & bug 858679

    Raises InsufficientFreeMemory if an instance can't
    start because of that. This will cause the normal
    instance failure recovery to catch this problem,
    set the state, and log the error. This also
    removes instance_set_state from db/api.py as that
    was causing these exceptions in the first place.

    Change-Id: I199aa6900890531b175e28c3b93d8dfb88e135d0

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/953
Committed: http://github.com/openstack/nova/commit/410a8df435fb424105872dc0a39f68eed902fbf2
Submitter: Jenkins
Branch: stable/diablo

 status fixcommitted
 done

commit 410a8df435fb424105872dc0a39f68eed902fbf2
Author: Aaron Lee <email address hidden>
Date: Mon Sep 26 18:22:03 2011 -0500

    Raise InsufficientFreeMemory

    Kind of fixes bug 851374 & bug 858679

    Raises InsufficientFreeMemory if an instance can't
    start because of that. This will cause the normal
    instance failure recovery to catch this problem,
    set the state, and log the error. This also
    removes instance_set_state from db/api.py as that
    was causing these exceptions in the first place.

    (cherry picked from commit 7dba1d9aa989760b190f1cf3bad2ed22bb2e2fc5)

    Change-Id: I93d4e3e2c264f520c0cd37c778a0db42eeb8345d

[Martin Pitt (pitti)]

Hello Troy, or anyone else affected,

Accepted nova into oneiric-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!