Exceptions during soft reboot should result in a hard reboot

[Phil Day]

The current code for soft_reboot in libvirt will only fall back to a hard reboot if the soft reboot returns a value of False. However there are some cases where an exception in soft reboot can be cleared by a hard reboot - for example if the image file is missing soft reboot will raise an exception, whereas a hard reboot (because it rebuilds the domain) will re-collect the image from Glance.

The code in libvirt/reboot should wrap the soft reboot attempt in a try block, and attempt a hard reboot is the soft reboot raises an libvirtError exception.

Steps to reproduce in Devstack:

Create an instance
 $ nova boot --image aebde4c5-10d3-4d76-90ab-28be2c891047 --flavor 1 phil
 $ nova list
 $ nova list
 +--------------------------------------+------+--------+------------+-------------+------------------+
 | ID | Name | Status | Task State | Power State | Networks |
 +--------------------------------------+------+--------+------------+-------------+------------------+
 | 2b9451fe-3d23-4e4a-b12c-9633f0467156 | phil | ACTIVE | None | Running | private=10.0.0.3 |
 +--------------------------------------+------+--------+------------+-------------+------------------+

Find and remove the base image backing file
  $ cd /opt/stack/data/nova/instances
  $ file 2b9451fe-3d23-4e4a-b12c-9633f0467156/disk
     2b9451fe-3d23-4e4a-b12c-9633f0467156/disk: QEMU QCOW Image (v2), has backing file (path
        /opt/stack/data/nova/instances/_base/552f54f49b488dcb1c787bd87b), 1073741824 bytes
(Note file name may be truncated)
   $ rm /opt/stack/data/nova/instances/_base/552f54f49b488dcb1c787bd87b2294272048874c

Reboot the instance
 $ nova reboot 2b9451fe-3d23-4e4a-b12c-9633f0467156
 $ nova list
 +--------------------------------------+------+--------+------------+-------------+------------------+
 | ID | Name | Status | Task State | Power State | Networks |
 +--------------------------------------+------+--------+------------+-------------+------------------+
 | 2b9451fe-3d23-4e4a-b12c-9633f0467156 | phil | REBOOT | rebooting | Running | private=10.0.0.3 |
 +--------------------------------------+------+--------+------------+-------------+------------------+

 $ nova list
 +--------------------------------------+------+--------+------------+-------------+------------------+
 | ID | Name | Status | Task State | Power State | Networks |
 +--------------------------------------+------+--------+------------+-------------+------------------+
 | 2b9451fe-3d23-4e4a-b12c-9633f0467156 | phil | ERROR | None | Running | private=10.0.0.3 |
 +--------------------------------------+------+--------+------------+-------------+------------------+

Hard reboot the instance
 $ nova reboot --hard 2b9451fe-3d23-4e4a-b12c-9633f0467156
 $ nova list
 +--------------------------------------+------+--------+------------+-------------+------------------+
 | ID | Name | Status | Task State | Power State | Networks |
 +--------------------------------------+------+--------+------------+-------------+------------------+
 | 2b9451fe-3d23-4e4a-b12c-9633f0467156 | phil | ACTIVE | None | Running | private=10.0.0.3 |
 +--------------------------------------+------+--------+------------+-------------+------------------+

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/38085

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/38085
Committed: http://github.com/openstack/nova/commit/9925c9f1b353240ebb1eccde50574055ce271313
Submitter: Jenkins
Branch: master

commit 9925c9f1b353240ebb1eccde50574055ce271313
Author: Matt Riedemann <email address hidden>
Date: Sun Jul 21 12:15:14 2013 -0700

    Performs hard reboot if libvirt soft reboot raises libvirtError

    Currently the libvirt driver will only attempt a hard reboot if a soft
    reboot fails and returns False. If a libvirt exception is raised,
    however, the hard reboot is not attempted. This patch wraps the soft
    reboot in a try block to continue with the hard reboot attempt if a
    libvirtError is raised.

    Fixes bug 1202974

    Change-Id: Ifaff75b33b5123cdd3293e8f263ccb4f72ae726a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/48415

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/48415
Committed: http://github.com/openstack/nova/commit/5e679a3911ccaf12f8f18256c22f0822a5dee378
Submitter: Jenkins
Branch: stable/grizzly

commit 5e679a3911ccaf12f8f18256c22f0822a5dee378
Author: Matt Riedemann <email address hidden>
Date: Sun Jul 21 12:15:14 2013 -0700

    Performs hard reboot if libvirt soft reboot raises libvirtError

    Currently the libvirt driver will only attempt a hard reboot if a soft
    reboot fails and returns False. If a libvirt exception is raised,
    however, the hard reboot is not attempted. This patch wraps the soft
    reboot in a try block to continue with the hard reboot attempt if a
    libvirtError is raised.

    Fixes bug 1202974

    (cherry picked from commit 9925c9f1b353240ebb1eccde50574055ce271313)

    Conflicts:
     nova/tests/test_libvirt.py

    Change-Id: Ifaff75b33b5123cdd3293e8f263ccb4f72ae726a