backing store missing for ephemeral disk on cold migration

[Chris Friesen]

I'm on stable/mitaka, but the master code looks similar.

I have compute nodes configured to use qcow2 and libvirt. The flavor has an ephemeral disk and a swap disk, with root disk size of "0" for auto-size. I boot an instance with this flavor.

When I try to cold-migrate the instance, I get an error:
2016-07-21 23:33:48.561 46340 ERROR nova.compute.manager [instance: 4e52bfd8-0c71-48dc-89fb-6f6b31dc06bb] libvirtError: Cannot access backing file '/etc/nova/instances/_base/ephemeral_1_0706d66' of storage file '/etc/nova/instances/4e52bfd8-0c71-48dc-89fb-6f6b31dc06bb/disk.eph0' (as uid:0, gid:0): No such file or directory

The problem seems to be that in nova.virt.libvirt.driver.LibvirtDriver.finish_migration() we call self._create_image(...block_device_info=None...)

Down in _create_image() we handle the case of a "disk.local" ephemeral device, but that doesn't help because the device is actually named "disk.eph0". It looks like we then try to loop over any ephemerals in block_device_info, but that's None so we don't handle any of those (which is too bad since it looks like they would be named correctly).

The end result is that we have a qcow2 "disk.eph0" image, but with potentially no backing store in <instances_path>/_base. When we tell libvirt to start the instance, this results in the above error.

[Chris Friesen (cbf123)]

The proposed fix is at https://review.openstack.org/#/c/346263/

[Thiago Martins (martinx)]

Please, include this fix on Mitaka.

[Thiago Martins (martinx)]

I'm trying to include the patch:

https://review.openstack.org/changes/346263/revisions/d7b831e38cbc029b7975d1059ed212762a5f6a5e/patch?zip

...on top of Ubuntu Xenial packages but, it fails here:

---
apt source nova
cd nova-13.1.1

nova-13.1.1$ patch -p1 < ../d7b831e3.diff
patching file nova/tests/unit/virt/libvirt/fake_imagebackend.py
Hunk #1 FAILED at 138.
1 out of 1 hunk FAILED -- saving rejects to file nova/tests/unit/virt/libvirt/fake_imagebackend.py.rej
patching file nova/tests/unit/virt/libvirt/test_driver.py
Hunk #1 succeeded at 17 (offset -2 lines).
Hunk #2 succeeded at 9549 (offset -975 lines).
Hunk #3 succeeded at 9574 with fuzz 2 (offset -980 lines).
Hunk #4 succeeded at 9674 (offset -969 lines).
Hunk #5 succeeded at 14565 (offset -1092 lines).
Hunk #6 succeeded at 14579 (offset -1092 lines).
Hunk #7 succeeded at 14647 with fuzz 2 (offset -1095 lines).
patching file nova/virt/libvirt/driver.py
Hunk #1 succeeded at 3112 (offset 152 lines).
Hunk #2 succeeded at 3238 (offset 185 lines).
Hunk #3 succeeded at 7397 with fuzz 1 (offset 191 lines).
---

 We need a proper fix for this on Ubuntu! :-P

 I'll try to rebuild the package with this patch but, don't know yet if tests will pass... If not, is there any way to disable the tests during package creation?

Cheers!
Thiago

[Chris Friesen (cbf123)]

Thiago: It looks like the patch will need reworking for Mitaka-on-Xenial to make it apply cleanly.

[Feodor Tersin (ftersin)]

@Thiago, see tests on PS4. I guess such implementation may work in Mitaka (i mostly mean usage of @mock.patch.object(nova.virt.libvirt.imagebackend.Backend, 'image') instead ImageBackendFixture). So you might rewrite tests to use it, but i have no idea if the result will be accepted as the backport to Mitaka.

I know nothing about package rules. I thought that a package must exactly contain a tagged version of an OS project source code, but recently i found an exception of this rule...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/346263
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d7b831e38cbc029b7975d1059ed212762a5f6a5e
Submitter: Jenkins
Branch: master

commit d7b831e38cbc029b7975d1059ed212762a5f6a5e
Author: Chris Friesen <email address hidden>
Date: Fri Jul 22 16:33:43 2016 -0600

    Fix cold migration with qcow2 ephemeral disks

    If we have qcow2 ephemeral disks we need to ensure that the
    backing file gets created on a cold migration.

    This requires passing in the block_device_info when calling
    _create_image() so that we can loop over the ephemeral disks.

    Closes-Bug: #1605720
    Co-Authored-By: Feodor Tersin <email address hidden>
    Change-Id: Ie278bb10e1675ba1d903aaa3c0249be0d1cf147b

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/398812

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/398812
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=842f1b6651f6104f8cb0e5ef32b80e6ffb10b067
Submitter: Jenkins
Branch: stable/newton

commit 842f1b6651f6104f8cb0e5ef32b80e6ffb10b067
Author: Chris Friesen <email address hidden>
Date: Fri Jul 22 16:33:43 2016 -0600

    Fix cold migration with qcow2 ephemeral disks

    If we have qcow2 ephemeral disks we need to ensure that the
    backing file gets created on a cold migration.

    This requires passing in the block_device_info when calling
    _create_image() so that we can loop over the ephemeral disks.

    Closes-Bug: #1605720
    Co-Authored-By: Feodor Tersin <email address hidden>
    Change-Id: Ie278bb10e1675ba1d903aaa3c0249be0d1cf147b
    (cherry picked from commit d7b831e38cbc029b7975d1059ed212762a5f6a5e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.4 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/372721
Reason: This review is > 4 weeks without comment, and is not mergable in it's current state. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.