s390x: failed to live migrate VM

[jichenjc]

see following logs when doing live migration on s390s platform with KVM

openstack server migrate --live kvm02 --block-migration d28caa4a-215b-44c8-bed0-e0e7faca07e5

ogs:

2019-10-10 12:03:25.710 19003 ERROR nova.virt.libvirt.driver [req-83d11ac0-3414-489e-8ad2-bfd0078e059f 44cdcb0bbe9e40fc91c043533d4dcbac 4067c50d412549c29b2deb58ec400ea1 - default default] CPU doesn't have compatibility.

XML error: Missing CPU model name

Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult: libvirtError: XML error: Missing CPU model name
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server [req-83d11ac0-3414-489e-8ad2-bfd0078e059f 44cdcb0bbe9e40fc91c043533d4dcbac 4067c50d412549c29b2deb58ec400ea1 - default default] Exception during message handling: MigrationPreCheckError: Migration pre-check error: CPU doesn't have compatibility.

XML error: Missing CPU model name

Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server return self.do_dispatch(endpoint, method, ctxt, args)
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in do_dispatch
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 79, in wrapped
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server function_name, call_dict, binary, tb)
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in exit
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server self.force_reraise()
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server six.reraise(self.type, self.value, self.tb)
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1418, in decorated_function
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 215, in decorated_function
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in exit
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server self.force_reraise()
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server six.reraise(self.type, self.value, self.tb)
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 203, in decorated_function
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6262, in check_can_live_migrate_destination
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server block_migration, disk_over_commit)
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7205, in check_can_live_migrate_destination
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server self._compare_cpu(None, source_cpu_info, instance)
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7452, in _compare_cpu
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server reason=m % {'ret': e, 'u': u})
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server MigrationPreCheckError: Migration pre-check error: CPU doesn't have compatibility.
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server XML error: Missing CPU model name
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
2019-10-10 12:03:25.748 19003 ERROR oslo_messaging.rpc.server

[jichenjc (jichenjc)]

nova compute will call check_can_live_migrate_destination when doing live migration
in turn it will call
self.driver.check_can_live_migrate_destination(ctxt,
            instance, src_compute_info, dst_compute_info,
            block_migration, disk_over_commit)

however, print at virt layer shows the CPU model is NONE

# Compare CPU
        LOG.info("---%s---%s---%s---", instance.vcpu_model, instance.vcpu_model.model, src_compute_info['cpu_info'])
        if not instance.vcpu_model or not instance.vcpu_model.model:
            source_cpu_info = src_compute_info['cpu_info']
            self._compare_cpu(None, source_cpu_info, instance)
        else:
            self._compare_cpu(instance.vcpu_model, None, instance)

INFO nova.virt.libvirt.driver [req-4e3ec3e2-7785-45b3-ba94-19e062a08d9b 44cdcb0bbe9e40fc91c043533d4dcbac 4067c50d412549c29b2deb58ec400ea1 - default default] ---VirtCPUModel(arch=None,features=[],match='exact',mode=None,model=None,topology=VirtCPUTopology,vendor=None)---None---{"vendor": null, "model": null, "arch": "s390x", "features": [], "topology": {"cores": 16, "cells": 1, "threads": 1, "sockets": 1}}---

[jichenjc (jichenjc)]

this is because if instance cpu model is none, it will check host info, but for host info
saw this log indicate that host calculation is not support at host side

2019-11-27 11:54:49.636 21154 INFO nova.virt.libvirt.driver [req-268522df-c953-4b92-8770-7f3528984e06 - - - - -] URI qemu:///system does not support full set of host capabilities: this function is not supported by the connection driver: cannot compute baseline CPU of s390x architecture

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/696228

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696228
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=011cce6adb30c50737b45ec02e161fd71ab5b3e3
Submitter: Zuul
Branch: master

commit 011cce6adb30c50737b45ec02e161fd71ab5b3e3
Author: jichenjc <email address hidden>
Date: Wed Dec 4 03:34:23 2019 +0000

    libvirt: avoid cpu check at s390x arch

    nova compute will call check_can_live_migrate_destination when doing
    live migration and it will compare cpu model, however, following info
    indicated that cpu compare is not supported at s390x arch.

    URI qemu:///system does not support full set of host capabilities: this
    function is not supported by the connection driver:
    cannot compute baseline CPU of s390x architecture

    https://www.libvirt.org/news.html has the info
    v5.9.0 has Improvements part indicated the compare was added at 5.9

    so the workaround is to avoid the check and let the migration proceed.

    Change-Id: I253f4f305ecf8b5331212be87caef41f2ebb747e
    Closes-Bug: 1854126

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/701525

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/701526

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/701525
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7d7a3ba70a36f05b07ee471490f1b587663348f8
Submitter: Zuul
Branch: stable/train

commit 7d7a3ba70a36f05b07ee471490f1b587663348f8
Author: jichenjc <email address hidden>
Date: Wed Dec 4 03:34:23 2019 +0000

    libvirt: avoid cpu check at s390x arch

    nova compute will call check_can_live_migrate_destination when doing
    live migration and it will compare cpu model, however, following info
    indicated that cpu compare is not supported at s390x arch.

    URI qemu:///system does not support full set of host capabilities: this
    function is not supported by the connection driver:
    cannot compute baseline CPU of s390x architecture

    https://www.libvirt.org/news.html has the info
    v5.9.0 has Improvements part indicated the compare was added at 5.9

    so the workaround is to avoid the check and let the migration proceed.

    Change-Id: I253f4f305ecf8b5331212be87caef41f2ebb747e
    Closes-Bug: 1854126
    (cherry picked from commit 011cce6adb30c50737b45ec02e161fd71ab5b3e3)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/701526
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=21642202ffe834a8086d428dd500bf09413bc002
Submitter: Zuul
Branch: stable/stein

commit 21642202ffe834a8086d428dd500bf09413bc002
Author: jichenjc <email address hidden>
Date: Wed Dec 4 03:34:23 2019 +0000

    libvirt: avoid cpu check at s390x arch

    nova compute will call check_can_live_migrate_destination when doing
    live migration and it will compare cpu model, however, following info
    indicated that cpu compare is not supported at s390x arch.

    URI qemu:///system does not support full set of host capabilities: this
    function is not supported by the connection driver:
    cannot compute baseline CPU of s390x architecture

    https://www.libvirt.org/news.html has the info
    v5.9.0 has Improvements part indicated the compare was added at 5.9

    so the workaround is to avoid the check and let the migration proceed.

    Change-Id: I253f4f305ecf8b5331212be87caef41f2ebb747e
    Closes-Bug: 1854126
    (cherry picked from commit 011cce6adb30c50737b45ec02e161fd71ab5b3e3)
    (cherry picked from commit 7d7a3ba70a36f05b07ee471490f1b587663348f8)