Soft-reboot fails after volume-update

[melanie witt]

During a volume-update, we don't save the instance XML with the new volume attachment, so soft-reboot fails with:

DEBUG nova.virt.libvirt.driver [req-149741f9-ad36-4787-ac6c-1ed7a273ab6a 75fbc7a6db34480091d2a53e2e20b695 62e53e5e804e49a9890928a5a4846f60 - - -] [instance: 5f06f6f8-f8b8-4b46-afa4-cefff24d7167] Instance soft reboot failed: Cannot access storage file '/dev/disk/by-path/ip-192.168.11.121:3260-iscsi-iqn.2010-10.org.openstack:volume-d558d988-b17c-4940-8c05-9222ad750955-lun-0' (as uid:107, gid:107): No such file or directory reboot /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:2395
WARNING nova.virt.libvirt.driver [req-149741f9-ad36-4787-ac6c-1ed7a273ab6a 75fbc7a6db34480091d2a53e2e20b695 62e53e5e804e49a9890928a5a4846f60 - - -] [instance: 5f06f6f8-f8b8-4b46-afa4-cefff24d7167] Failed to soft reboot instance. Trying hard reboot.

as it can't access the old volume attachment defined in the XML. It falls back on a hard reboot which is like pulling the power cord and can result in file system inconsistencies.

We should save the updated XML to the persistent configuration after updating the volume.

Steps to reproduce:

1. Boot an instance (nova boot --image <image> --flavor <flavor> test0)
2. Create two volumes (cinder create --name A 1, cinder create --name B 1)
3. Attach the first volume to the instance (nova volume-attach test0 <uuid of volume A>)
4. Swap to the second volume (nova volume-update test0 <uuid of volume A> <uuid of volume B>)
5. Try to reboot the instance (nova reboot test0)

Actual result:

Instance fails to soft-reboot and hard reboots instead.

Expected result:

Instance soft-reboots successfully.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/498983

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/498983
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5b008c6540f948d28dc50a0ef84095ebd96d198d
Submitter: Zuul
Branch: master

commit 5b008c6540f948d28dc50a0ef84095ebd96d198d
Author: melanie witt <email address hidden>
Date: Tue Aug 29 22:53:44 2017 +0000

    Save updated libvirt domain XML after swapping volume

    When a user calls the volume-update API, we swap_volume in the libvirt
    driver from the old volume attachment to the new volume attachment.
    Currently, we're saving the domain XML with the old configuration prior
    to updating the volume and upon a soft-reboot request, it results in an
    error:

      Instance soft reboot failed: Cannot access storage file <old path>

    and falls back to a hard reboot, which is like pulling the power cord,
    possibly resulting in file system inconsistencies.

    This changes to saving the new, updated domain XML after the volume
    swap.

    Closes-Bug: #1713857

    Change-Id: I166cde5ad8b00699e4ec02609f0d7b69236d855d

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/523910

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/523911

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/523910
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3dbcf9d5be0f631f4d963117a242006df3649b8d
Submitter: Zuul
Branch: stable/pike

commit 3dbcf9d5be0f631f4d963117a242006df3649b8d
Author: melanie witt <email address hidden>
Date: Tue Aug 29 22:53:44 2017 +0000

    Save updated libvirt domain XML after swapping volume

    When a user calls the volume-update API, we swap_volume in the libvirt
    driver from the old volume attachment to the new volume attachment.
    Currently, we're saving the domain XML with the old configuration prior
    to updating the volume and upon a soft-reboot request, it results in an
    error:

      Instance soft reboot failed: Cannot access storage file <old path>

    and falls back to a hard reboot, which is like pulling the power cord,
    possibly resulting in file system inconsistencies.

    This changes to saving the new, updated domain XML after the volume
    swap.

    Closes-Bug: #1713857

    Change-Id: I166cde5ad8b00699e4ec02609f0d7b69236d855d
    (cherry picked from commit 5b008c6540f948d28dc50a0ef84095ebd96d198d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/523911
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=aa0e4ebd2c02c042bbe2c0d58e690e3e43b04a17
Submitter: Zuul
Branch: stable/ocata

commit aa0e4ebd2c02c042bbe2c0d58e690e3e43b04a17
Author: melanie witt <email address hidden>
Date: Tue Aug 29 22:53:44 2017 +0000

    Save updated libvirt domain XML after swapping volume

    When a user calls the volume-update API, we swap_volume in the libvirt
    driver from the old volume attachment to the new volume attachment.
    Currently, we're saving the domain XML with the old configuration prior
    to updating the volume and upon a soft-reboot request, it results in an
    error:

      Instance soft reboot failed: Cannot access storage file <old path>

    and falls back to a hard reboot, which is like pulling the power cord,
    possibly resulting in file system inconsistencies.

    This changes to saving the new, updated domain XML after the volume
    swap.

    Closes-Bug: #1713857

    NOTE(melwitt): The assert in test_swap_volume_block_old_libvirt had to
    be modified to account for the additional XMLDesc call from the bug
    fix. The test_swap_volume_block_old_libvirt doesn't exist in Pike, so
    that's why we didn't hit this in the master patch.

    Change-Id: I166cde5ad8b00699e4ec02609f0d7b69236d855d
    (cherry picked from commit 5b008c6540f948d28dc50a0ef84095ebd96d198d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.1.1 release.