live migration (block-migrate) may failed if instance image is deleted in glance

Bug #1829000 reported by Alexandre arents on 2019-05-14
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Alexandre arents
Queens
Medium
Lee Yarwood
Rocky
Medium
Lee Yarwood
Stein
Medium
Lee Yarwood

Bug Description

Description
===========
When we run live block migration on instance with a deleted glance image,
it may failed with following logs:

-- nova-compute-log: --
2019-05-10 11:06:27.417 248758 ERROR nova.virt.libvirt.driver [req-b28b9aca-9135-4258-93a6-a802e6192c60 f7929cd1d8994661b88aff12977c8b9e 54f4d231201b4944a5fa4587a09bda28 - - -] [instance: 84601bd4-a6ee-4e00-a5bc-f7c80def7ec5] Migration operation has aborted
2019-05-10 11:06:27.566 248758 ERROR nova.virt.libvirt.driver [req-b28b9aca-9135-4258-93a6-a802e6192c60 f7929cd1d8994661b88aff12977c8b9e 54f4d231201b4944a5fa4587a09bda28 - - -] [instance: 84601bd4-a6ee-4e00-a5bc-f7c80def7ec5] Live Migration failure: internal error: info migration reply was missing return status

-- on target host /var/log/libvirt/qemu/instance-xxx.log: --
/build/qemu-nffE1h/qemu-2.x/nbd.c:nbd_trip():L1310: From: 2416967680, Len: 65536, Size: 2361393152, Offset: 0
/build/qemu-nffE1h/qemu-2.x/nbd.c:nbd_trip():L1311: requested operation past EOF--bad client?
/build/qemu-nffE1h/qemu-2.x/nbd.c:nbd_trip():L1310: From: 3624927232, Len: 589824, Size: 2361393152, Offset: 0

Seems that pre_live_migration task do not setup correctly target instance disk:
-because glance image is not existant, it fallbacks to remote host copy method.
-in this context, image.cache() is called without instance disk size parameter.
-consequence is instance disk is not resized to the correct size and remain with the size of backing file, so the disk is too small, making failed libvirt live migration.

Steps to reproduce
==================
* Spawn qcow2 instance with glance image size << of flavor disk instance size
* generate few user data in instance.
* delete glance image.
* run live block migration.

Environment
===========
Issue observed in Newton, still present in master.

Fix proposed to branch: master
Review: https://review.opendev.org/659054

Changed in nova:
assignee: nobody → Alexandre arents (aarents)
status: New → In Progress
Matt Riedemann (mriedem) on 2019-05-22
tags: added: libvirt live-migration
Matt Riedemann (mriedem) on 2019-05-22
Changed in nova:
importance: Undecided → Medium

Reviewed: https://review.opendev.org/659054
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c1782bacd8461bdd8c833792864e61228fa451f1
Submitter: Zuul
Branch: master

commit c1782bacd8461bdd8c833792864e61228fa451f1
Author: Alexandre Arents <email address hidden>
Date: Tue May 14 11:37:12 2019 +0000

    Fix live-migration when glance image deleted

    When block live-migration is run on instance with
    a deleted glance image, image.cache() is called
    without specyfing instance disk size parameter,
    preventing the resize of disk on the target host.

    Change-Id: Id0f05bb1275cc816d98b662820e02eae25dc57a3
    Closes-Bug: #1829000

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/662152
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b45f47c7577f7b9694ba0a6060312f1a0ec06abd
Submitter: Zuul
Branch: stable/stein

commit b45f47c7577f7b9694ba0a6060312f1a0ec06abd
Author: Alexandre Arents <email address hidden>
Date: Tue May 14 11:37:12 2019 +0000

    Fix live-migration when glance image deleted

    When block live-migration is run on instance with
    a deleted glance image, image.cache() is called
    without specyfing instance disk size parameter,
    preventing the resize of disk on the target host.

    Change-Id: Id0f05bb1275cc816d98b662820e02eae25dc57a3
    Closes-Bug: #1829000
    (cherry picked from commit c1782bacd8461bdd8c833792864e61228fa451f1)

This issue was fixed in the openstack/nova 19.0.1 release.

Reviewed: https://review.opendev.org/662153
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e06a66fe9f9cb05c2d05cae336e2676fb5e5f3d2
Submitter: Zuul
Branch: stable/rocky

commit e06a66fe9f9cb05c2d05cae336e2676fb5e5f3d2
Author: Alexandre Arents <email address hidden>
Date: Tue May 14 11:37:12 2019 +0000

    Fix live-migration when glance image deleted

    When block live-migration is run on instance with
    a deleted glance image, image.cache() is called
    without specyfing instance disk size parameter,
    preventing the resize of disk on the target host.

    Change-Id: Id0f05bb1275cc816d98b662820e02eae25dc57a3
    Closes-Bug: #1829000
    (cherry picked from commit c1782bacd8461bdd8c833792864e61228fa451f1)
    (cherry picked from commit b45f47c7577f7b9694ba0a6060312f1a0ec06abd)

This issue was fixed in the openstack/nova 18.2.1 release.

Reviewed: https://review.opendev.org/662154
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=55deddf65de824ca6287537ec6947b73021557ee
Submitter: Zuul
Branch: stable/queens

commit 55deddf65de824ca6287537ec6947b73021557ee
Author: Alexandre Arents <email address hidden>
Date: Tue May 14 11:37:12 2019 +0000

    Fix live-migration when glance image deleted

    When block live-migration is run on instance with
    a deleted glance image, image.cache() is called
    without specyfing instance disk size parameter,
    preventing the resize of disk on the target host.

    Change-Id: Id0f05bb1275cc816d98b662820e02eae25dc57a3
    Closes-Bug: #1829000
    (cherry picked from commit c1782bacd8461bdd8c833792864e61228fa451f1)
    (cherry picked from commit b45f47c7577f7b9694ba0a6060312f1a0ec06abd)
    (cherry picked from commit e06a66fe9f9cb05c2d05cae336e2676fb5e5f3d2)

This issue was fixed in the openstack/nova 17.0.11 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers