ERROR in log when rebooting too soon after a reboot

[David Kranz]

With grizzly 1:2013.1+git201303180832~precise-0ubuntu1

Doing

nova reboot --hard test;nova reboot --hard test

causes the following ERROR in the log. The second reboot is returning 422. I presume the second call is expected to fail though 422 is not one of the codes mentioned in the spec. If so, there should not be an ERROR in the log. Interestingly, if you do the same thing with soft reboot, there is no error and the response to the second call is 409.

Mar 19 10:58:09 se11-1eth0 2013-03-19 10:58:09.154 ERROR nova.api.openstack.compute.servers [req-3ca09d16-7195-43cc-b0c8-4e52ac334920 5dda90b838634950ae3bb5e43b9225e0 8f6f26e2d384488989abad72c7f2612f] [instance: 10e0dfa0-5c7e-4526-b2a1-fa9b21ba531a] Error in reboot unexpected task state: expecting [None, 'rebooting'] but the actual state is rebooting_hard#0122013-03-19 10:58:09.154 32202 TRACE nova.api.openstack.compute.servers [instance: 10e0dfa0-5c7e-4526-b2a1-fa9b21ba531a] Traceback (most recent call last):#0122013-03-19 10:58:09.154 32202 TRACE nova.api.openstack.compute.servers [instance: 10e0dfa0-5c7e-4526-b2a1-fa9b21ba531a] File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 1072, in _action_reboot#0122013-03-19 10:58:09.154 32202 TRACE nova.api.openstack.compute.servers [instance: 10e0dfa0-5c7e-4526-b2a1-fa9b21ba531a] self.compute_api.reboot(context, instance, reboot_type)#0122013-03-19 10:58:09.154 32202 TRACE nova.api.openstack.compute.servers [instance: 10e0dfa0-5c7e-4526-b2a1-fa9b21ba531a] File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 163, in wrapped#0122013-03-19 10:58:09.154 32202 TRACE nova.api.openstack.compute.servers [instance: 10e0dfa0-5c7e-4526-b2a1-fa9b21ba531a] return func(self, context, target, *args, **kwargs)#0122013-03-19 10:58:09.154 32202 TRACE nova.api.openstack.compute.servers [instance: 10e0dfa0-5c7e-4526-b2a1-fa9b21ba531a] File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 153, in inner#0122013-03-19 10:58:09.154 32202 TRACE nova.api.openstack.compute.servers [instance: 10e0dfa0-5c7e-4526-b2a1-fa9b21ba531a] return function(self, context, instance, *args, **kwargs)#0122013-03-19 10:58:09.154 32202 TRACE nova.api.openstack.compute.servers [instance: 10e0dfa0-5c7e-4526-b2a1-fa9b21ba531a] File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 134, in inner#0122013-03-19 10:58:09.154 32202 TRACE nova.api.openstack.compute.servers [instance: 10e0dfa0-5c7e-4526-b2a1-fa9b21ba531a] return f(self, context, ins
Mar 19 10:58:09 se11-1eth0 2013-03-19 10:58:09.156 INFO nova.api.openstack.wsgi [req-3ca09d16-7195-43cc-b0c8-4e52ac334920 5dda90b838634950ae3bb5e43b9225e0 8f6f26e2d384488989abad72c7f2612f] HTTP exception thrown: Unable to process the contained instructions
Mar 19 10:58:09 se11-1eth0 2013-03-19 10:58:09.157 INFO nova.osapi_compute.wsgi.server [req-3ca09d16-7195-43cc-b0c8-4e52ac334920 5dda90b83863

[Russell Bryant (russellb)]

Can you clarify exactly which version or revision of nova this package relates to? The line numbers don't match up to the most recent code at least.

[David Kranz (david-kranz)]

The report began with

With grizzly 1:2013.1+git201303180832~precise-0ubuntu1

Can you be more specific about what you want clarified?

[Chuck Short (zulcss)]

I am not able to reproduce this which hypervisor are you using? If you are using libvirt which version?

[David Kranz (david-kranz)]

The original report was from a real cluster using the grizzly test ppa but I just reproduced it with a fresh install of vanilla devstack from trunk. It uses libvirt 0.9.8-2ubuntu17.7.

stack@xg08:~/devstack$ nova boot --image cirros-0.3.1-x86_64-uec --flavor 42 test
stack@xg08:~/devstack$ nova reboot --hard test;nova reboot --hard test
ERROR: Unable to process the contained instructions (HTTP 422) (Request-ID: req-6b8b6146-ada4-460f-b61a-9fc37eae72c5)

 42 is the "nano" flavor set up by devstack/tempest

[Andrew Laski (alaski)]

Change I516a19894ab3d183057c774e84c4faa7053a6463 addressed the 422 issue, and now a 500 is returned. There's still the issue of the traceback and 500 rather than 409 though.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/28603

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/28603
Committed: http://github.com/openstack/nova/commit/662a793056d4544c9c9d73fc21b99d77404537cc
Submitter: Jenkins
Branch: master

commit 662a793056d4544c9c9d73fc21b99d77404537cc
Author: Andrew Laski <email address hidden>
Date: Wed May 8 15:59:55 2013 -0400

    Raise InstanceInvalidState for double hard reboot

    A hard reboot is not allowed if the instance task_state is
    REBOOTING_HARD so check for that state and raise InstanceInvalidState.
    This returns the correct status of 409 to the caller.

    Previously the hard reboot would fail the the expected_task_state check
    on db.instance_update_and_get_original which would log a traceback and
    return a 500 to the caller.

    Bug 1157237

    Change-Id: Ifb6111790352da5c404289b218c50d8104384301

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/29264

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/29264
Committed: http://github.com/openstack/nova/commit/784b6e4d1a5d18274dd67958d8ece1efdf46e8d8
Submitter: Jenkins
Branch: stable/grizzly

commit 784b6e4d1a5d18274dd67958d8ece1efdf46e8d8
Author: Andrew Laski <email address hidden>
Date: Wed May 8 15:59:55 2013 -0400

    Raise InstanceInvalidState for double hard reboot

    A hard reboot is not allowed if the instance task_state is
    REBOOTING_HARD so check for that state and raise InstanceInvalidState.
    This returns the correct status of 409 to the caller.

    Previously the hard reboot would fail the the expected_task_state check
    on db.instance_update_and_get_original which would log a traceback and
    return a 500 to the caller.

    Bug 1157237

    Change-Id: Ifb6111790352da5c404289b218c50d8104384301
    (cherry picked from commit 662a793056d4544c9c9d73fc21b99d77404537cc)

[egon (egon-p)]

This commit seems to undo the intentions of commit a69251804e62f871f93ffa420978f9b61a36df9c, which was to allow a double hard reboot in the case of a failed first hard reboot.

After this commit, it is once again possible to have VMs terminally in rebooting_hard, especially if a user tries to reboot/hard_reboot while there is some problem with the hypervisor. When the HV comes back, the VM is stuck in rebooting_hard, and it requires an admin to reset the state to active, and then reboot the VM.

[Andrew Laski (alaski)]

@egon, please note that it was not possible to issue a double hard reboot before this bug fix was put in place, it just failed in an unexpected way. Now it fails in an expected way.