Live migration fails with an attached non-bootable Cinder volume (Pike)

[Christoph Fiehe]

I have setup a fresh OpenStack HA Pike environment on Ubuntu 16.04.3. Live migration has been enabled and works so far. As storage backend I am using Ceph v12.2.0 (Luminous). When you attach a secondary volume to a vm and try to live migrate the vm to another host, it fails with the following exception:

2017-09-07 08:30:46.621 3246 ERROR nova.virt.libvirt.driver [req-832fc6c9-4d8e-46f8-94be-95cb5c7a4114 dddfba8e02f746799a6408a523e6cd25 ed2d2efd86dd40e7a45491d8502318d3 - default default] [instance: 414e8bc1-0b85-4e7f-8897-74b416b9caf8] Live Migration failure: unsupported configuration: Target device drive address 0:0:0 does not match source 0:0:1: libvirtError: unsupported configuration: Target device drive address 0:0:0 does not match source 0:0:1
2017-09-07 08:30:47.293 3246 ERROR nova.virt.libvirt.driver [req-832fc6c9-4d8e-46f8-94be-95cb5c7a4114 dddfba8e02f746799a6408a523e6cd25 ed2d2efd86dd40e7a45491d8502318d3 - default default] [instance: 414e8bc1-0b85-4e7f-8897-74b416b9caf8] Migration operation has aborted

When you (not live) migrate the corresponding vm with the attached volume, migration succeeds. When you launch the vm from a bootable volume, migration and live migration succeeds in both cases. Only live migration with an additional attached volume fails. Because of Ceph RDB volumes, migration does not require "Block migration".

Compute node:
$ pip list | grep -E 'nova|cinder'
nova (16.0.0)
python-cinderclient (3.1.0)
python-novaclient (9.1.0)

Controller node:
$ pip list | grep -E 'nova|cinder'
cinder (11.0.0)
nova (16.0.0)
python-cinderclient (3.1.0)
python-novaclient (9.1.0)

$ ceph --version
ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous (rc)

Is this a normal behaviour?

[Christoph Fiehe (fiehe)]

This is the corresponding libvirt configuration file of the vm.

#############################
instance-00000030.xml
#############################
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit instance-00000030
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>instance-00000030</name>
  <uuid>58538546-09f7-4efb-abe1-4eaf008fe756</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="16.0.0"/>
      <nova:name>test-vm02</nova:name>
      <nova:creationTime>2017-09-07 06:59:17</nova:creationTime>
      <nova:flavor name="m1.small">
        <nova:memory>2048</nova:memory>
        <nova:disk>20</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>1</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="dddfba8e02f746799a6408a523e6cd25">admin</nova:user>
        <nova:project uuid="ed2d2efd86dd40e7a45491d8502318d3">demo</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="b2b7cfc7-ce74-421d-98a9-79768f36e5e1"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>1024</shares>
  </cputune>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>16.0.0</entry>
      <entry name='serial'>74bf283c-f6a8-c600-0293-894b59a50724</entry>
      <entry name='uuid'>58538546-09f7-4efb-abe1-4eaf008fe756</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-xenial'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>SandyBridge</model>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none' discard='unmap'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='e5479084-e43e-4a1e-959a-b9989f02e632'/>
      </auth>
      <source protocol='rbd' name='vms/58538546-09f7-4efb-abe1-4eaf008fe756_disk'>
        <host name='10.30.200.141' port='6789'/>
        <host name='10.30.200.142' port='6789'/>
        <host name='10.30.200.143' port='6789'/>
      </source>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writ...

[Mike Lowe (jomlowe)]

I am also having this problem with ocata

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/518022

[Jay Pipes (jaypipes)]

Doing some investigation into this bug, I've found that the code that creates the XML snippet for disk devices hasn't changed in 2+ years.

The code is now in nova/virt/libvirt/migration.py now, but the code used to be in libvirt/driver.py. Sahid's refactoring patches to break driver.py into separate modules moved a lot of this code around but did not functionally change the code.

This particular section was moved in the following patch:

https://github.com/openstack/nova/commit/23191308ff1bb151e441e398d69e46fb845f4e65

I remember reviewing the patch in Gerrit. :)

https://review.openstack.org/#/c/299490/

What is strange to me is that this bug is only recently being reported. I can't imagine that this has been buggy for 2+ years and nobody has previously noticed.

[Mike Lowe (jomlowe)]

I had attributed the change to RHEL 7 derived distros moving from libvirt 2.x to 3.x. My problems coincided with the upgrade from CentOS 7.3 to 7.4. This may not be the case as the original report was with Ubuntu 16.04.

[Mike Lowe (jomlowe)]

Writing out the xml before and after the update the update function changes the address.

It turns this:

disk device="disk" type="network">
 <driver cache="writeback" name="qemu" type="raw"/>
 <auth username="cinder">
  <secret type="ceph" uuid="1a790a26-dd49-4825-8d16-3dd627cf05a9"/>
 </auth>
 <source name="cinder-volumes/volume-de400476-b68a-45a2-b04f-739313f42bef" protocol="rbd">
  <host name="172.16.128.101" port="6789"/>
  <host name="172.16.128.121" port="6789"/>
  <host name="172.16.128.130" port="6789"/>
 </source>
 <target bus="scsi" dev="sdb"/>
 <serial>de400476-b68a-45a2-b04f-739313f42bef</serial>
 <address bus="0" controller="0" target="0" type="drive" unit="1"/>
</disk>

Into this:

<disk device="disk" type="network">
 <driver cache="writeback" name="qemu" type="raw"/>
 <auth username="cinder">
  <secret type="ceph" uuid="1a790a26-dd49-4825-8d16-3dd627cf05a9"/>
 </auth>
 <source name="cinder-volumes/volume-de400476-b68a-45a2-b04f-739313f42bef" protocol="rbd">
  <host name="172.16.128.101" port="6789"/>
  <host name="172.16.128.121" port="6789"/>
  <host name="172.16.128.130" port="6789"/>
 </source>
 <target bus="scsi" dev="sdb"/>
 <serial>de400476-b68a-45a2-b04f-739313f42bef</serial>
 <address controller="0" type="drive"/>
</disk>

[melanie witt (melwitt)]

Thanks Mike for providing detail of the change in the XML after the update function.

I dug around in the code and based on the XML excerpt you show, this appears to be a regression caused by this patch [1] (which was also backported to ocata) which made a change to the libvirt driver _get_volume_config function. It added code to set the 'address' XML element when 'bus' == 'iscsi' when previously the 'address' element was left unmodified. The patch sets the address.controller element to 0 and sets the address.unit element only if it's been set in disk_info. And it's only set in disk_info in the attach_volume function in the driver, so that's why we're not seeing it set to 1 as expected during a live migration (attach_volume is not called). So we see it getting set as 0 in the original error message in comment 1:

"Live Migration failure: unsupported configuration: Target device drive address 0:0:0 does not match source 0:0:1: libvirtError: unsupported configuration: Target device drive address 0:0:0 does not match source 0:0:1"

So it seems to be the right thing to do to avoid modifying the address element for live migration only (as the proposed patch is doing).

[1] https://review.openstack.org/#/c/459741

[Jay Pipes (jaypipes)]

Good sleuthing, Melanie, I concur with your conclusion.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/518022
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b196857f04e41dde294eaacc2c1a991807ecc829
Submitter: Zuul
Branch: master

commit b196857f04e41dde294eaacc2c1a991807ecc829
Author: Mike Lowe <email address hidden>
Date: Mon Nov 6 11:06:46 2017 -0500

    live-mig: keep disk device address same

    During live migration disk devices are updated with the latest
    block device mapping information for volumes. Previously this
    relied on libvirt to assign addresses in order after the already
    assigned devices like the root disk had been accounted for. In
    the latest libvirt the unassigned devices are allocated first which
    makes the root disk address double allocated causing the migration to
    fail. A running instance should never have the hardware addresses
    of its disks changed mid flight. While disk address changes during
    live migration produce fatal errors for the operator it would likely
    cause errors inside the instance and unexpected behavior if the device
    addresses change during cold migrationt review. With this disk addresses are no
    longer updated with block device mapping information while every
    other element of the disk definition for a volume is updated.

    Closes-Bug: 1715569

    Change-Id: I17af9848f4c0edcbcb101b30e45ca4afa93dcdbb

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/541642

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/541642
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d91e881890933363550ffb9e114c6df4c32a4090
Submitter: Zuul
Branch: stable/pike

commit d91e881890933363550ffb9e114c6df4c32a4090
Author: Mike Lowe <email address hidden>
Date: Mon Nov 6 11:06:46 2017 -0500

    live-mig: keep disk device address same

    During live migration disk devices are updated with the latest
    block device mapping information for volumes. Previously this
    relied on libvirt to assign addresses in order after the already
    assigned devices like the root disk had been accounted for. In
    the latest libvirt the unassigned devices are allocated first which
    makes the root disk address double allocated causing the migration to
    fail. A running instance should never have the hardware addresses
    of its disks changed mid flight. While disk address changes during
    live migration produce fatal errors for the operator it would likely
    cause errors inside the instance and unexpected behavior if the device
    addresses change during cold migrationt review. With this disk addresses are no
    longer updated with block device mapping information while every
    other element of the disk definition for a volume is updated.

    Closes-Bug: 1715569

    Change-Id: I17af9848f4c0edcbcb101b30e45ca4afa93dcdbb
    (cherry picked from commit b196857f04e41dde294eaacc2c1a991807ecc829)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/541904
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6b753f0b5ea734432d2f45fb11884ea1b110ed77
Submitter: Zuul
Branch: stable/ocata

commit 6b753f0b5ea734432d2f45fb11884ea1b110ed77
Author: Mike Lowe <email address hidden>
Date: Mon Nov 6 11:06:46 2017 -0500

    live-mig: keep disk device address same

    During live migration disk devices are updated with the latest
    block device mapping information for volumes. Previously this
    relied on libvirt to assign addresses in order after the already
    assigned devices like the root disk had been accounted for. In
    the latest libvirt the unassigned devices are allocated first which
    makes the root disk address double allocated causing the migration to
    fail. A running instance should never have the hardware addresses
    of its disks changed mid flight. While disk address changes during
    live migration produce fatal errors for the operator it would likely
    cause errors inside the instance and unexpected behavior if the device
    addresses change during cold migrationt review. With this disk addresses are no
    longer updated with block device mapping information while every
    other element of the disk definition for a volume is updated.

    Closes-Bug: 1715569

    Change-Id: I17af9848f4c0edcbcb101b30e45ca4afa93dcdbb
    (cherry picked from commit b196857f04e41dde294eaacc2c1a991807ecc829)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.1.1 release.