libvirt: _compare_cpu doesn't consider NotSupportedError

Bug #1434429 reported by Markus Zoeller (markus_z)
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
Markus Zoeller (markus_z)
Kilo
Fix Released
Undecided
Unassigned

Bug Description

Issue
=====

The libvirt driver method "_compare_cpu" doesn't consider that the
underlying libvirt function could throw a NotSupportedError (like
baselineCPU call in "host.py" module [1])

Steps to reproduce
==================

* Create setup with at least 2 compute nodes
* Create cinder volume with bootable image
* Launch instance from that volume
* Start live migration of instance to another host

Expected behavior
=================

If the target host has the same CPU architecture like the source host,
the live migration should be triggered.

Actual behavior
===============

The live migration gets aborted and rolled back because all libvirt
errors gets treated equally.

Logs & Env.
===========

section "libvirt" in "/etc/nova/nova.conf" in both nodes:

    [libvirt]
    live_migration_flag = VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE, VIR_MIGRATE_TUNNELLED
    disk_cachemodes = block=none
    vif_driver = nova.virt.libvirt.vif.LibvirtGenericVIFDriver
    inject_partition = -2
    live_migration_uri = qemu+tcp://stack@%s/system
    use_usb_tablet = False
    cpu_mode = none
    virt_type = kvm

Nova version
------------

/opt/stack/nova$ git log --oneline -n5
90ee915 Merge "Add api microvesion unit test case for wsgi.action"
7885b74 Merge "Remove db layer hard-code permission checks for flavor-manager"
416f310 Merge "Remove db layer hard-code permission checks for migrations_get*"
ecb306b Merge "Remove db layer hard-code permission checks for migration_create/update"
6efc8ad Merge "libvirt: don't allow to resize down the default ephemeral disk"

References
==========

[1] baselineCPU call to libvirt catches NotSupportedError;
    https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L753

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/166130

Changed in nova:
assignee: nobody → Markus Zoeller (mzoeller)
status: New → In Progress
Changed in nova:
importance: Undecided → Low
Revision history for this message
Sean Dague (sdague) wrote :

Under what circumstances will 'Not Supported' be thrown?

Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

The libvirt for system z (s390 architecture) for example doesn't implement that yet. It's waiting for a qemu feature which is still under discussion [1]. Though, the live migration to another s390 host is possible (to some degree) without comparing CPUs.

[1] https://lists.gnu.org/archive/html/qemu-devel/2015-03/msg05996.html

Matt Riedemann (mriedem)
tags: added: kilo-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/166130
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=da9301fbc7877529bc2b099ffeb622e02ef54f0e
Submitter: Jenkins
Branch: master

commit da9301fbc7877529bc2b099ffeb622e02ef54f0e
Author: Markus Zoeller <email address hidden>
Date: Thu Mar 19 18:36:09 2015 +0100

    libvirt: handle NotSupportedError in compareCPU

    In case of a NotSupportedError during the CPU comparison, the live
    migration was aborted and rolled back because all libvirt errors were
    treated equally.
    If the target host has the same CPU architecture like the source host,
    the live migration gets triggered now, despite no CPU comparison was
    done.

    Closes-Bug: 1434429

    Change-Id: I9276f0a7ce9a9b58943e1ed9b9c7d8855482d9cd

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/184240

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/184758

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/184758
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b3d4c1b6efdf7eb7a7293c069f453e05ffa9f338
Submitter: Jenkins
Branch: master

commit b3d4c1b6efdf7eb7a7293c069f453e05ffa9f338
Author: Nikola Dipanov <email address hidden>
Date: Thu May 21 11:39:05 2015 +0100

    libvirt: Fix cpu_compare tests and a wrong method when logging

    Commit da9301fb tries to special-case a not-supported exception but the
    logging bits call a non existant method and also mocking in the test
    added was off.

    The reason that this passes gate might seems to be that comparing 0 with
    a MagicMock object in the libvirt.driver._compare_cpu() method always
    fails - which seems to be arbitrary behavior according to:

    https://docs.python.org/2/library/stdtypes.html#comparisons

    Change-Id: I2732410512fc521e42d4d63ce59c676f2033bf22
    Related-bug: #1434429

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/kilo)

Reviewed: https://review.openstack.org/184240
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d18058e8d7c6b3e66ca496a8c112c477bcb73851
Submitter: Jenkins
Branch: stable/kilo

commit d18058e8d7c6b3e66ca496a8c112c477bcb73851
Author: Markus Zoeller <email address hidden>
Date: Thu Mar 19 18:36:09 2015 +0100

    libvirt: handle NotSupportedError in compareCPU

    In case of a NotSupportedError during the CPU comparison, the live
    migration was aborted and rolled back because all libvirt errors were
    treated equally.
    If the target host has the same CPU architecture like the source host,
    the live migration gets triggered now, despite no CPU comparison was
    done.

    Closes-Bug: 1434429

    Conflicts:
        nova/tests/unit/virt/libvirt/test_driver.py

    Change-Id: I9276f0a7ce9a9b58943e1ed9b9c7d8855482d9cd
    (cherry picked from commit da9301fbc7877529bc2b099ffeb622e02ef54f0e)

tags: added: in-stable-kilo
Thierry Carrez (ttx)
Changed in nova:
milestone: none → liberty-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: liberty-1 → 12.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.