Rebuilding a node in ERROR state should set status to REBUILD

[Clint Byrum]

I recently had a few nova-driven ironic nodes fail to deploy, and resurrected them by issuing another nova rebuild.

This worked quite nicely, but the Status stayed as ERROR, when I would have expected it to change back to REBUILD

[Rakesh H S (rh-s)]

Was able to reproduce this issue.

when nova uses ironic driver, rebuilding a node does not set the status to REBUILD.
(However, when nova uses libvirt driver, rebuilding a node sets the status to REBUILD.)

[Rakesh H S (rh-s)]

sorry, This issue occurs for libvirt driver also. It is not specific to ironic driver.

In an setup wherein nova uses libvirt driver, the instance turned ERROR state for whatever reason.
I tried to 'rebuild' the instance and here is the result.

stack@ubuntu:/home/rakesh/devstack$ nova rebuild cc35c64d-fba7-416b-8934-92aff76ab52f 90625fc0-cce1-419d-b6c2-ab441c7d5e87
...
stack@ubuntu:/home/rakesh/devstack$ nova list
+--------------------------------------+------+--------+------------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------------+-------------+------------------+
| cc35c64d-fba7-416b-8934-92aff76ab52f | test | ERROR | rebuild_spawning | Running | private=10.0.0.2 |
+--------------------------------------+------+--------+------------------+-------------+------------------+

You can see that the 'Status' did not turn to 'REBUILD'
Hence this issue is not specific to nova using 'ironic driver'.

[Rakesh H S (rh-s)]

figured out the root cause of this issue:

Fix needs to be done in nova:

In nova, generally STATUS field is determined by the 'vm_state' and 'task_state'.
And presently for an instance with vm_state= ERROR, will always report status as ERROR. (since it does not take into account the present task_state of instance)

i.e in,
nova/nova/api/openstack/common.py

   vm_states.ERROR: {
        'default': 'ERROR',
    },
As you can see above the task_state of instance is not taken into account if vm_state is in ERROR.

FIX:
need to change it to:
   vm_states.ERROR: {
        'default': 'ERROR',
        task_states.REBUILDING: 'REBUILD',
        task_states.REBUILD_BLOCK_DEVICE_MAPPING: 'REBUILD',
        task_states.REBUILD_SPAWNING: 'REBUILD',
    },

will wait for nova guys to let me know if they are ok with this change.

[Sean Dague (sdague)]

This seems sane to me, it's mostly a display item.

[Matt Riedemann (mriedem)]

The compute API definitely allows you to rebuild and instance in error state (and active and stopped for that matter):

http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n2208

    @check_instance_state(vm_state=[vm_states.ACTIVE, vm_states.STOPPED,
                                    vm_states.ERROR])

So it seems right to update the status mapping as noted above.

[Rakesh H S (rh-s)]

Thanks for replying guys.. will proceed with the changes and will push an patch soon.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/123628

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/123628
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=93d25c38b8a1c7554ec124c5cf71a3e0d74e5bb3
Submitter: Jenkins
Branch: master

commit 93d25c38b8a1c7554ec124c5cf71a3e0d74e5bb3
Author: Rakesh H S <email address hidden>
Date: Wed Sep 24 10:23:19 2014 +0530

    consistently set status as REBUILD when rebuilding

    Consider an instance is in SHUTOFF status(stopped). When a rebuild
    is issued on this instance, the status does not display as REBUILD.

    Similarly, when a rebuild is issued on an instance and the operation
    fails, the instance turns to ERROR state. Now if we issue a rebuild
    again, the status does not display as REBUILD.

    This patch fixes the above mentioned issues by displaying the correct
    status i.e REBUILD.

    Change-Id: Id11c9d31683a049c61fd751dc0ddabaf22c96cb9
    Closes-Bug: #1362733