nova resize fails with 'Could not open backing file' error

[Anna Babich]

It was reproduced on ISO #512 (env with 3 controllers, 2 computes):
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "512"
  build_id: "512"
  fuel-nailgun_sha: "19fb6afdafcc17f87922e10e4cc90689c087d49c"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "af8b5f7b3d1e231d0b04de5e79dd60b5b35d5ad7"
  fuel-ostf_sha: "5fe41945c2a49f26c849df1fd46329f6db1ab6b0"
  fuel-mirror_sha: "1e93fe1794b988677ff0942788bd48b61a89d307"
  fuelmenu_sha: "234cb4cbb30fbd2df00f388c28f31606d9cae15f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "94507c5e4dad6d8cfbd8f5d41aa8389d5335990a"

When trying to resize and instance from micro to tiny flavor, an instance is migrated from node-5 to node-4 but falls to Error state with the traceback:
<179>Feb 10 14:56:17 node-4 nova-compute: 2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [req-08684722-453f-41ef-ae2f-a59a67d1ad36 e10fbbe0cd3
a4f169eae2a49e1e76528 9c5b2bd5f39e45ceaae2b7a0fca853e3 - - -] [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] Setting instance vm_state to ERROR
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] Traceback (most recent call last):
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] File "/usr/lib/python2.7/dist-packages/nova/
compute/manager.py", line 3952, in finish_resize
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] disk_info, image)
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] File "/usr/lib/python2.7/dist-packages/nova/
compute/manager.py", line 3918, in _finish_resize
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] old_instance_type)
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] File "/usr/lib/python2.7/dist-packages/oslo_
utils/excutils.py", line 195, in __exit__
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] six.reraise(self.type_, self.value, self.t
b)
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] File "/usr/lib/python2.7/dist-packages/nova/
compute/manager.py", line 3913, in _finish_resize
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] block_device_info, power_on)
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] File "/usr/lib/python2.7/dist-packages/nova/
virt/libvirt/driver.py", line 6936, in finish_migration
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] self._disk_resize(image, size)
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] File "/usr/lib/python2.7/dist-packages/nova/
virt/libvirt/driver.py", line 6915, in _disk_resize
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] disk.extend(image, size)
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] File "/usr/lib/python2.7/dist-packages/nova/
virt/disk/api.py", line 190, in extend
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] utils.execute('qemu-img', 'resize', image.
path, size)
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] File "/usr/lib/python2.7/dist-packages/nova/
utils.py", line 390, in execute
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] return processutils.execute(*cmd, **kwargs
)
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] File "/usr/lib/python2.7/dist-packages/oslo_
concurrency/processutils.py", line 275, in execute
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] cmd=sanitized_cmd)
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] ProcessExecutionError: Unexpected error while running command.
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] Command: qemu-img resize /var/lib/nova/instances/3a5fa796-e393-4823-89e0-4b5ab3fc2b60/disk 1073741824
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] Exit code: 1
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] Stdout: u''
2016-02-10 14:56:17.095 27043 ERROR nova.compute.manager [instance: 3a5fa796-e393-4823-89e0-4b5ab3fc2b60] Stderr: u"qemu-img: Could not open '/var/lib/nova/instances/3a5fa796-e393-4823-89e0-4b5ab3fc2b60/disk': Could not open backing file: Could not open '/var/lib/nova/instances/_base/58823af74e4ffb4b317d81587539f620a23cff11': No such file or directory\n"

[Roman Podoliaka (rpodolyaka)]

We'll recheck this to make sure it's reproduced.

Anyway, even if it is, there's a workaround of copying the backing file manually (yes, this is ugly :P ). And this must only be reproduced when an instance is moved from one node to another during resize (i.e. only when there is not enough resources to place the new VM on the same node).

I suggest we move this to 8.0-MU1 and 9.0.

[stgleb (gstepanov)]

In my opinion the root of problem that code flow that handles copying instance image from one host to another
doesn't actually copy backing file, that is why it cannot be opened on destination host. I'm going to propose patch that fixes that problem.

[Roman Podoliaka (rpodolyaka)]

The fix is present in Mitaka and downstream 9.0 branch:

rpodolyaka@rpodolyaka-pc:~/src/nova$ git branch -a --contains db7fd539f261ea53f6c005478049424b9dae1543 | grep mirantis
  remotes/mirantis/9.0/mitaka
  remotes/mirantis/master
  remotes/mirantis/stable/mitaka
rpodolyaka@rpodolyaka-pc:~/src/nova$ git show db7fd539f261ea53f6c005478049424b9dae1543
commit db7fd539f261ea53f6c005478049424b9dae1543
Author: Matthew Booth <email address hidden>
Date: Fri Mar 4 18:34:21 2016 +0000

    libvirt: Fix resize of instance with deleted glance image

    finish_migration() in the libvirt driver was attempting to resize an
    image before checking that its backing file was present. This patch
    re-orders these 2 operations. In doing so, we also have to resolve an
    overloading of the 'disk_info' variable.

    Closes-Bug: #1546778

    Change-Id: I03e08fae97416ebe5cdedcf238a06d1b90203c5d

[Roman Podoliaka (rpodolyaka)]

It was also merged to stable/liberty:

https://review.openstack.org/#/q/I03e08fae97416ebe5cdedcf238a06d1b90203c5d,n,z

MOS Maintenance can get it automatically with the code sync.

[Rodion Tikunov (rtikunov)]

Fix now presented in 8.0 after code sync with stable/liberty

[Rodion Tikunov (rtikunov)]

Can not reproduce bug in MOS 8.0 with MU-1