Live migration fails when the instance has a config_drive_format=iso9660

[Hiroyuki Eguchi]

This error has occurred under the following situations.

・ Live migration of volume backed instance which has a config-drive
・ Block migration of instance which has a config-drive

we have to create a config-drive on destination host in pre-live-migration.

[Matt Riedemann (mriedem)]

What level of code? What virt driver? What errors in the nova compute log?

[Boris Deschenes (boris-michel-deschenes)]

this is real

on the source node:

2013-11-27 17:05:13.337 6143 ERROR nova.virt.libvirt.driver [-] [instance: f3505fac-d7f7-482f-a418-62c5732c84db] Live Migration failure: cannot open file '/var/lib/nova/instances/f3505fac-d7f7-482f-a418-62c5732c84db/disk.config': No such file or directory

on the destination node:

2013-11-27 17:05:13.632 WARNING nova.compute.manager [req-f30471ae-7b26-42a9-882b-c03b5d498b27 589dc2275a4b43f8906b4b5ed8540348 5a76694904ed4f97b9495a29f1bba9f5] [instance: f3505fac-d7f7-482f-a418-62c5732c84db] Detaching volume from unknown instance
2013-11-27 17:05:13.639 ERROR nova.compute.manager [req-f30471ae-7b26-42a9-882b-c03b5d498b27 589dc2275a4b43f8906b4b5ed8540348 5a76694904ed4f97b9495a29f1bba9f5] [instance: f3505fac-d7f7-482f-a418-62c5732c84db] Failed to detach volume f10391a9-7104-4223-a623-626bd4414060 from vda
2013-11-27 17:05:13.639 15756 TRACE nova.compute.manager [instance: f3505fac-d7f7-482f-a418-62c5732c84db] Traceback (most recent call last):
2013-11-27 17:05:13.639 15756 TRACE nova.compute.manager [instance: f3505fac-d7f7-482f-a418-62c5732c84db] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2925, in _detach_volume
2013-11-27 17:05:13.639 15756 TRACE nova.compute.manager [instance: f3505fac-d7f7-482f-a418-62c5732c84db] mp)
2013-11-27 17:05:13.639 15756 TRACE nova.compute.manager [instance: f3505fac-d7f7-482f-a418-62c5732c84db] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1014, in detach_volume
2013-11-27 17:05:13.639 15756 TRACE nova.compute.manager [instance: f3505fac-d7f7-482f-a418-62c5732c84db] virt_dom = self._lookup_by_name(instance_name)
2013-11-27 17:05:13.639 15756 TRACE nova.compute.manager [instance: f3505fac-d7f7-482f-a418-62c5732c84db] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2406, in _lookup_by_name
2013-11-27 17:05:13.639 15756 TRACE nova.compute.manager [instance: f3505fac-d7f7-482f-a418-62c5732c84db] raise exception.InstanceNotFound(instance_id=instance_name)
2013-11-27 17:05:13.639 15756 TRACE nova.compute.manager [instance: f3505fac-d7f7-482f-a418-62c5732c84db] InstanceNotFound: Instance instance-0000001f could not be found.
2013-11-27 17:05:13.639 15756 TRACE nova.compute.manager [instance: f3505fac-d7f7-482f-a418-62c5732c84db]

[Michael Still (mikal)]

We see users with this problem with grizzly and libvirt.

[Michael Still (mikal)]

Ok. I have re-created this problem in devstack on trunk. What I am seeing is:

 - I can boot an instance with config drive. The config drive has configuration information in it.
 - I block migrate that instance to another hypervisor node. The config drive is now an invalid filesystem.

Can I confirm this is the error that you are seeing?

[Michael Still (mikal)]

Ok, it looks to me that the problem is that libvirt doesn't migrate cdrom devices, even if they're virtual. If I change the disk type of the disk.config to "disk", then everything works fine.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/74848

[Tracy Jones (tjones-i)]

this bug could be pushed to icehouse-rc-potential if not merged by 2/24 12pm UTC

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/74848
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ee4a26bb57b08c06e6d738d2e23a1677a1c7fcf2
Submitter: Jenkins
Branch: master

commit ee4a26bb57b08c06e6d738d2e23a1677a1c7fcf2
Author: Michael Still <email address hidden>
Date: Thu Mar 20 08:36:38 2014 +1100

    Refuse to block migrate instances with config drive

    libvirt does not block migrate read only devices such as config
    drives. It decides that the config drive is read only because it
    is attached as a cdrom. So, we just throw our hands up in the
    air and bravely refuse to block migrate those instances.

    Partial-Bug: 1246201

    Change-Id: I24f4f83c1ddf1cf3e37db4663ab82e61f0a2d00f
    DocImpact: you should read all that stuff above.

[Dan Smith (danms)]

This made it in

[Michael Still (mikal)]

We should note that what got committed is a partial fix, and will need to be revisited.

[Dmitry Borodaenko (angdraug)]

Given that this is a partial fix, should this bug now be reopened?

[Michael Still (mikal)]

Reopening because previous fix was partial.

[Loganathan Parthipan (parthipan)]

Would it be OK to make this configurable? In installations where config drives are vfat, the administrator can make the decision to migrate if they wish.

I've sent in a review for this:

https://review.openstack.org/98999

[Antony Messerli (antonym)]

I've been testing block migrations with vfat and config drives and they seem to work without any issues. Any progress on getting this made configurable?

[Mehdi Abaakouk (sileht)]

I have proposed a change related to this issue: https://review.openstack.org/#/c/112014/

[Matt Symonds (msymonds)]

Any updates on being able to migrate vfat config_drives being merged in?

[Sean Severson (sseverson)]

This had been working been working fine after Mehdi's fix, but that fix has just been reverted, breaking the block migration once again in Juno.

[Oleksii Aleksieiev (alexzzman)]

Mehdi's fix is missing code for virt/configdrive.py to actually creates and put data into configdrive.

Looks like its assumes configdrive.ConfigDriveBuilder will create the drive properly but it does not.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/130505

[Luo Gangyi (luogangyi)]

Oops, after a whole year, the problem still exists...

Can we just detach the config drive-->live migrate-->attach config drive again ?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/133919

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/98782
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/98999
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/130505
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Tony Breeds (o-tony)]

I'm going to fix this in libvirt.

The best we can do now in terms of a workaround in nova is to either (disable it unconditionally) or add a config option for those that know it works in their environment.

[Davanum Srinivas (DIMS) (dims-v)]

Fix proposed in libvirt

[Pádraig Brady (p-draigbrady)]

libvirt fix proposed at https://www.redhat.com/archives/libvir-list/2015-February/msg00499.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/159087

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mehdi Abaakouk (<email address hidden>) on branch: master
Review: https://review.openstack.org/159087
Reason: Merged in previous review

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/162220

[OpenStack Infra (hudson-openstack)]

Change abandoned by Davanum Srinivas (dims) (<email address hidden>) on branch: master
Review: https://review.openstack.org/162220

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/162233

[OpenStack Infra (hudson-openstack)]

Change abandoned by Davanum Srinivas (dims) (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/162233

[Davanum Srinivas (DIMS) (dims-v)]

Tony,

Can you please open a libvirt bug as requested by Kashyap in the libvirt thread
https://www.mail-archive.com/libvir-list%40redhat.com/msg109485.html

thanks,
dims

[Tony Breeds (o-tony)]

Done: https://bugzilla.redhat.com/show_bug.cgi?id=1203032

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/133919
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4e665112f275f17a90c6f96daa805af652c66fa0
Submitter: Jenkins
Branch: master

commit 4e665112f275f17a90c6f96daa805af652c66fa0
Author: Luo Gangyi <email address hidden>
Date: Wed Nov 12 01:42:37 2014 -0800

    libvirt: partial fix for live-migration with config drive

    In current version of nova, live-migration with config drive on
    local disk is forbidden due to the bug of libvirt of copying readonly
    disk. However, if we use vfat as the format of config drive,
    the function of live-migration works well.

    In this patch, we re-open the function of live-migration to admin users.
    Notice you should add 'config_drive_format=vfat' in nova.conf explicitly.

    This patch doesn't solve the problem fundamentally which need further
    efforts, but offers a simple and feasible workaround to user.

    DocImpact

    Co-Authored-By: Davanum Srinivas <email address hidden>
    Change-Id: I7429e12766da7f7f8d484b3a9df6247e832816b0
    Partial-Bug: #1246201

[Pavel Boldin (pboldin)]

@Tony, if you don't have enough time I can take action to implement this. Let's work together.

[Pavel Boldin (pboldin)]

Both `libvirt' (NBD) and `qemu' (block-migration) only migrate non-readonly devices.

So, both `libvirt' and `qemu' need to be modified in order to implement block-migration.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/168492

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/168646

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jordan Tardif (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/168492

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/168646
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7c4be65733e48dcf44bc306ab964fafb71b37774
Submitter: Jenkins
Branch: stable/juno

commit 7c4be65733e48dcf44bc306ab964fafb71b37774
Author: Luo Gangyi <email address hidden>
Date: Wed Nov 12 01:42:37 2014 -0800

    libvirt: partial fix for live-migration with config drive

    In current version of nova, live-migration with config drive on
    local disk is forbidden due to the bug of libvirt of copying readonly
    disk. However, if we use vfat as the format of config drive,
    the function of live-migration works well.

    In this patch, we re-open the function of live-migration to admin users.
    Notice you should add 'config_drive_format=vfat' in nova.conf explicitly.

    This patch doesn't solve the problem fundamentally which need further
    efforts, but offers a simple and feasible workaround to user.

    DocImpact

    Conflicts:
            nova/tests/unit/virt/libvirt/test_driver.py

    NOTE(mriedem): The conflicts are due to:

    1. The tests being moved in Kilo.
    2. Not having commit 152fb73a3 in stable/juno, this is why we
       remove the volume entry from the expected return data.
    3. Not having commit 96195d51f in stalbe/juno, this is why we
       aren't using objects in the test and have to explicitly
       set name and kernel_id on self.test_instance.

    Co-Authored-By: Davanum Srinivas <email address hidden>
    Change-Id: I7429e12766da7f7f8d484b3a9df6247e832816b0
    Partial-Bug: #1246201
    (cherry picked from commit 4e665112f275f17a90c6f96daa805af652c66fa0)

[Pavel Boldin (pboldin)]

Seems connected: https://bugs.launchpad.net/nova/+bug/1398999

I have started on implementation of this thing for libvirt and libvirt-python:
https://github.com/paboldin/libvirt
https://github.com/paboldin/libvirt-python

[OpenStack Infra (hudson-openstack)]

Change abandoned by Joe Gordon (<email address hidden>) on branch: master
Review: https://review.openstack.org/123073
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Matt Symonds (msymonds)]

Hi,

Is https://review.openstack.org/#/c/168646/ likely to be merged into IceHouse?

[Matt Riedemann (mriedem)]

@Matt, no one has proposed it as a backport to stable/icehouse, but you could. However, we're getting pretty close to closing out icehouse (icehouse-eol) so you'd need to act quickly if you want to get a backport proposed.

[Matt Symonds (msymonds)]

I'm currently applying this patch manually. How would I go about proposing it to be back-ported?

[Matt Riedemann (mriedem)]

There are some docs here:

https://wiki.openstack.org/wiki/StableBranch#Proposing_Fixes

You want to cherry pick the git hash of the merged change on stable/juno to stable/icehouse using "git cherry-pick -x 7c4be65733e48dcf44bc306ab964fafb71b37774", otherwise there is a 'cherry pick' button in the gerrit UI for the stable/juno backport, you could just use that to propose to stable/icehouse assuming it merges cleanly.

If it doesn't backport cleanly you have to do it manually and resolve conflicts and make sure the tests pass.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/188444

[Matt Symonds (msymonds)]

@Matt: Thanks for pointing me in the right direction.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/188444
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8876294fe67048fc8cea0c3e0f5094e30c9f2c36
Submitter: Jenkins
Branch: stable/icehouse

commit 8876294fe67048fc8cea0c3e0f5094e30c9f2c36
Author: Matt Symonds <email address hidden>
Date: Wed Nov 12 01:42:37 2014 -0800

    libvirt: partial fix for live-migration with config drive

    In current version of nova, live-migration with config drive on
    local disk is forbidden due to the bug of libvirt of copying readonly
    disk. However, if we use vfat as the format of config drive,
    the function of live-migration works well.

    In this patch, we re-open the function of live-migration to admin users.
    Notice you should add 'config_drive_format=vfat' in nova.conf explicitly.

    This patch doesn't solve the problem fundamentally which need further
    efforts, but offers a simple and feasible workaround to user.

    DocImpact

    Conflicts:
            nova/virt/libvirt/driver.py

    NOTE(symonds): The conflicts are due to:
            not using is_shared_instance_path or is_shared_block_storage in stable/icehouse.

    Conflicts:
            nova/tests/unit/virt/libvirt/test_driver.py

    NOTE(symonds): The conflicts are due to:
            libvirt_driver.LibvirtDriver().pre_live_migration() returning None on success
                instead of Spice and VNC information.
            not using is_shared_instance_path or is_shared_block_storage in stable/icehouse.

    Co-Authored-By: Davanum Srinivas <email address hidden>, Matt Riedemann <email address hidden>
    Change-Id: I7429e12766da7f7f8d484b3a9df6247e832816b0
    Partial-Bug: #1246201
    (cherry picked from commit 7c4be65733e48dcf44bc306ab964fafb71b37774)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/123073
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=adecf780d3ed4315e4ce305cb1821d493650494b
Submitter: Jenkins
Branch: master

commit adecf780d3ed4315e4ce305cb1821d493650494b
Author: Michael Still <email address hidden>
Date: Tue Nov 25 15:42:47 2014 +0300

    Handle config drives being stored on rbd

    rbd is the only example of a currently supported image storage
    backend where it makes sense to put the config drive in the
    configured storage backend instead of local hypervisor disk. I
    don't think this makes sense for LVM, where we would be creating
    a LV for a tens of megabytes file, which seems like overkill to
    me. The other storage backends use local disk for their data
    already.

    This use case was covered by the now reverted changes:
      228d0221763b12f11ecbacde4db38b1151f96e31
      0b01e846d40f3b343da9ebe1dae89cca8bc2ac66
      ecce888c469c62374a3cc43e3cede11d8aa1e799

    Support this special case by moving the image to rbd once it has
    been created in the local instance directory on the hypervisor.

    I've tested this change in devstack and it works.

    Related-bug: #1369627
    Related-bug: #1361840
    Related-bug: #1246201

    Co-Authored-By: Mehdi Abaakouk <email address hidden>
    Co-Authored-By: Dan Smith <email address hidden>

    Change-Id: Ia3ca5a18c79d62b71b9c042a612d12dd074b245e

[Pavel Boldin (pboldin)]

I have finished implementation of the selective block device migration in the `libvirt' [1] and want to implement the Nova part as well.

Can I reassign this bug on me?

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1203032#c11

[Markus Zoeller (markus_z) (mzoeller)]

@Pavel in response to #53:

It's over 3 month since user lvmxh assigned the bug to himself, I think it's fair to set yourself as new assignee and provide a patch set in the near future.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/kilo
Review: https://review.openstack.org/214773

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rosario Di Somma (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/214773

[Tyler Bishop (tyler-bishop)]

Why is this being abandon?

[Rick Harris (rconradharris)]

Pavel: Still working on the corresponding Nova fix to take advantage of the Libvirt change?

[Pavel Boldin (pboldin)]

Yes. But at the moment I'm very busy with others tasks so this one is on-hold.

There are published reviews from other contributors: https://review.openstack.org/227278/

I have not yet prepared any decent of mine partly because there is this blueprint that should be implemented first in my opinion: https://blueprints.launchpad.net/nova/+spec/split-different-live-migration-types

[Tony Breeds (o-tony)]

@taylor: only the kilo backport was abandoned and that's because it wasn't valid.

The review that Pavel points at is the "correct" solution for master (and may be baclportable to liberty) but requires a very modern libvirt.

BTW Pavel, Thanks again for doing that work.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/234659

[Matthew Thode (prometheanfire)]

what is the current status, has the fix been merged to libvirt, if so, can we allow migration again for iso9660 config drive instances in mitaka?

[Eli Qiao (taget-9)]

Hi Matthew,
please check https://review.openstack.org/#/c/234659/20 which is pushed by Pawel Koniszewski (pawel-koniszewski) , most of codes are done, one +2 from core and still some comments on the test cases.

Eli.

[OpenStack Infra (hudson-openstack)]

Change abandoned by John Garbutt (<email address hidden>) on branch: master
Review: https://review.openstack.org/234659
Reason: I have marked the associated blueprint as approved, lets track this as purely a bug fix now?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/234659
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=bc4b9980fa74e15e98aed5633c61480c751767b5
Submitter: Jenkins
Branch: master

commit bc4b9980fa74e15e98aed5633c61480c751767b5
Author: Pawel Koniszewski <email address hidden>
Date: Wed Jul 6 11:04:16 2016 +0200

    [libvirt] Live migration fails when config_drive_format=iso9660

    This patch enables block live migration with read-only config drive.
    Basically it copies config drive to destination host prior to migration
    start, so that QEMU will not try to write to read-only device and
    therefore it will not raise an exception.

    Co-Authored-By: Bartosz Fic <email address hidden>
    Co-Authored-By: Michael Still <email address hidden>

    Change-Id: I9ddaef658e3d9a77e4982275d7df9e56b0a5459f
    Closes-Bug: #1246201

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/365420

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/365420
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=48d30c16d339c8d1218c3b85c9da7a69f7e99bb8
Submitter: Jenkins
Branch: stable/mitaka

commit 48d30c16d339c8d1218c3b85c9da7a69f7e99bb8
Author: Pawel Koniszewski <email address hidden>
Date: Wed Jul 6 11:04:16 2016 +0200

    [libvirt] Live migration fails when config_drive_format=iso9660

    This patch enables block live migration with read-only config drive.
    Basically it copies config drive to destination host prior to migration
    start, so that QEMU will not try to write to read-only device and
    therefore it will not raise an exception.

    Co-Authored-By: Bartosz Fic <email address hidden>
    Co-Authored-By: Michael Still <email address hidden>

    Change-Id: I9ddaef658e3d9a77e4982275d7df9e56b0a5459f
    Closes-Bug: #1246201
    (cherry picked from commit bc4b9980fa74e15e98aed5633c61480c751767b5)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 13.1.2 release.

[Nobuto Murata (nobuto)]

config_drive_format is now deprecated for removal.
https://opendev.org/openstack/nova/commit/c19ecc34ea7fc19e5c4278eb4fda8b5cf7e9d638

But I've seen a corner case where config_drive_format=vfat is still required:
https://bugs.launchpad.net/nova/+bug/1939869