Live block migration fails for instances whose glance images have been deleted

[Loganathan Parthipan]

Once the glance image from which an instance was spawned is deleted it's not possible to block migrate this instance.

To recreate:

1. Boot an instance off a public image or snapshot
2. Delete the image from glance
3. Live block migrate this instance. It will fail at pre-live-migration stage as the image could not be downloaded.

[Joe Gordon (jogo)]

This makes sense as we pull down the base images from glance. We could fix this by pulling the image from the other machine but this sounds like more of a feature request then a bug.

[Loganathan Parthipan (parthipan)]

It should be possible, in theory, to rebase the disk so that the backing file is removed in the destination on the exception ImageNotFound from glance client.

source node:
disk (qcow2) -> _base/backing_file

destination node:
disk (qcow2) - But no backing file

This would ensure that this particular instance would keep running and migrate again.

Not having a backing file in _base is acceptable in this case since there isn't going to be another instance off this rootdisk again.

[Loganathan Parthipan (parthipan)]

A possible design to elaborate comment #2.

1. source---> call pre_live_migration on destination
2. glance fetch returns ImageNotFound on destination
3. Handle exception and create disk, but keep it single layer qcow2. (ie. without a backing file)
4. return call back to source

5. Initiate live migration

Now in step 5, an incremental block copy would not work since the destination disk file would look like the source overlay file. We need to tell libvirt/kvm to do a deep copy. I don't know if this is possible with the current libvirt API. However, since qemu-img lets you rebase to any other backing file or flatten, I believe the mechanism exists.

[Mehdi Abaakouk (sileht)]

@parthipan, unfortunatly this solution won't work for instance with AMI image that have depends of AKI and ARI image, libvirt/kvm copy only disk passed as disk to kvm, not the files passed to -kernel, or -initrd

I will try to submit something that use the same logic than the migrate_power_off code

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/90321

[George Shuklin (george-shuklin)]

I'd like to note that non-live migration fails too due this bug (not wishlist!), but in different way - instance failed to start after migration, but it can be workarounded by reset-state and stop/start.

[Warren Wang (w-emailme)]

Wouldn't this be easier to resolve if a new image status was available in Glance? Something like an archive mode where new images may not be launched from the image, but the image may still be used as a reference for existing instances to do things like migrate, snapshot, etc. Private does not work, and obviously neither does delete.

[melanie witt (melwitt)]

Based on the comments in the proposed patch and the fact that this bug affects 11 people, I'm unmarking this as a High bug to aid in tracking.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/90321
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Davanum Srinivas (DIMS) (dims-v)]

Looks like Melanie picked this back up

[Davanum Srinivas (DIMS) (dims-v)]

Fix Proposed again.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/90321
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3345ca029fd2527eec8de365a37779fd37809398
Submitter: Jenkins
Branch: master

commit 3345ca029fd2527eec8de365a37779fd37809398
Author: Mehdi Abaakouk <email address hidden>
Date: Fri Apr 25 10:39:48 2014 +0200

    libvirt: Fix migration when image doesn't exist

    When nova is used without shared_storage and glance doesn't have the
    images required by the instance anymore, the live block migration code
    can't prepare the destination host properly.

    This patch catches the case when images are not found in glance, and
    copies the missing images from the source host, like the
    migrate_disk_and_power_off code already does.

    The KVM disk deep copy method is not used because it won't work for AMI
    image that depends of AKI/ARI image. kernel and initrd are not considered
    by the kvm disk migration because they are readonly file (by the kvm pov)
    used only to boot the VM.

    Co-Authored-By: Sahid Orentino Ferdjaoui <email address hidden>
    Closes-bug: #1270825

    Change-Id: If81f8b1bbe3e738579ffe2d8f36807afb77560d8

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/189923

[OpenStack Infra (hudson-openstack)]

Change abandoned by Artom Lifshitz (<email address hidden>) on branch: stable/juno
Review: https://review.openstack.org/189923