NUMA tuning broken in select libvirt versions

[Stephen Finucane]

#1438226 reported that CPU pinning was broken in select versions of libvirt. Further investigation has highlighted issues with NUMA tuning in general on these versions. On some versions of libvirt, the same error messages seen when configuring CPU pinning are seen when configuring NUMA tuning (e.g. with use of the 'hw:numa-nodes' flavor key). This would suggest that the entire NUMA tuning feature is broken on these versions, rather than just CPU pinning. The results from testing, mostly duplicated from the aforementioned bug report, are given below.

This is somewhat related to #1422775 ("nova libvirt driver assumes qemu support for NUMA pinning").

---

# Testing Configuration

Testing was conducted in a container which provided a single-node, Fedora 21-based (3.17.8-300.fc21.x86_64) OpenStack instance (built with devstack). The yum-provided libvirt and its dependencies were removed and libvirt and libvirt-python were built and installed from source.

# Results

The results are as follows (currently incomplete):

    versions status
    -------- ------
    1.2.9 ok
    1.2.9.1 ok
    1.2.9.2 fail
    1.2.9.3 ok
    1.2.10 ok
    1.2.11 ok
    1.2.12 ok

v1.2.9.2 is broken by this (backported) patch:

    https://www.redhat.com/archives/libvir-list/2014-November/msg00275.html

This can be seen as commit

    e226772 (qemu: fix domain startup failing with 'strict' mode in numatune)

# Error logs

v1.2.9.2 produces the following exception:

    Traceback (most recent call last):
      File "/opt/stack/nova/nova/compute/manager.py", line 2301, in _build_resources
        yield resources
      File "/opt/stack/nova/nova/compute/manager.py", line 2171, in _build_and_run_instance
        flavor=flavor)
      File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2357, in spawn
        block_device_info=block_device_info)
      File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4376, in _create_domain_and_network
        power_on=power_on)
      File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4307, in _create_domain
        LOG.error(err)
      File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 82, in __exit__
        six.reraise(self.type_, self.value, self.tb)
      File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4297, in _create_domain
        domain.createWithFlags(launch_flags)
      File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
        result = proxy_call(self._autowrap, f, *args, **kwargs)
      File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
        rv = execute(f, *args, **kwargs)
      File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
        six.reraise(c, e, tb)
      File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
        rv = meth(*args, **kwargs)
      File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1029, in createWithFlags
        if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
    libvirtError: Failed to create controller cpu for group: No such file or directory

[Tristan Cacqueray (tristan-cacqueray)]

You currently have this open as a security bug, indicating you believe it represents an exploitable vulnerability in the software. Can you elaborate on the circumstances under which this bug might be exploited by a malicious actor, and the risks it implies?

[Przemyslaw Czesnowicz (pczesno)]

As far as i can tell this bug has no security implications.
 I must have mis clicked something.
Sorry about that.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/178188

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/188427

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/178188
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7ca56106def7950aceecacf40b2ae8de7c846cb2
Submitter: Jenkins
Branch: master

commit 7ca56106def7950aceecacf40b2ae8de7c846cb2
Author: Stephen Finucane <email address hidden>
Date: Thu Apr 23 14:01:10 2015 +0100

    libvirt: Disable NUMA for broken libvirt

    Ensure versions of libvirt with broken NUMA tuning support are not used
    for said feature.

    Change-Id: I6de388f1ca98c1ae16f2968f59881e3b0dba5f8d
    Closes-Bug: #1449028
    Related-Bug: #1438226

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/188427
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9efd47018dffc4c0f1f8f4d1b667b7b6744b01f5
Submitter: Jenkins
Branch: master

commit 9efd47018dffc4c0f1f8f4d1b667b7b6744b01f5
Author: Matt Riedemann <email address hidden>
Date: Thu Jun 4 07:21:49 2015 -0700

    libvirt: log when BAD_LIBVIRT_NUMA_VERSIONS detected

    This addresses a comment from the review on change
    I6de388f1ca98c1ae16f2968f59881e3b0dba5f8d where we should log something
    when a bad version of libvirt is detected for NUMA support.

    Since the method is called from periodic tasks and when new instances
    are created, we only want to log the warning once.

    Related-Bug: #1449028

    Change-Id: I6d3526ac892532a7b15e994391b1e86a619aa9c3