nova-compute failed to start because get_host_capablilities failed

[QiangGuan]

When starting nova-compute service on node with lxc hypervisor, it failed with the following error:

 2013-12-11 03:04:41.439 ERROR nova.openstack.common.threadgroup [-] can only parse strings
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 117, in wait
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup x.wait()
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 49, in wait
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/service.py", line 448, in run_service
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup service.start()
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/service.py", line 164, in start
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup self.manager.pre_start_hook()
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 822, in pre_start_hook
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup self.update_available_resource(nova.context.get_admin_context())
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 4971, in update_available_resource
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup nodenames = set(self.driver.get_available_nodes())
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/virt/driver.py", line 980, in get_available_nodes
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup stats = self.get_host_stats(refresh=refresh)
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4569, in get_host_stats
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup return self.host_state.get_host_stats(refresh=refresh)
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 429, in host_state
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup self._host_state = HostState(self)
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4960, in __init__
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup self.update_status()
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4999, in update_status
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup self.driver.get_instance_capabilities()
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3702, in get_instance_capabilities
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup caps = self.get_host_capabilities()
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2739, in get_host_capabilities
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup self._caps.host.cpu.parse_str(features)
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/virt/libvirt/config.py", line 61, in parse_str
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup self.parse_dom(etree.fromstring(xmlstr))
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "lxml.etree.pyx", line 2993, in lxml.etree.fromstring (src/lxml/lxml.etree.c:62980)
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup File "parser.pxi", line 1614, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:92786)
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup ValueError: can only parse strings
2013-12-11 03:04:41.439 TRACE nova.openstack.common.threadgroup

This is because no cpu model info can be gotten from libvirt getCapabilities interface for lxc hypervisor.

[Matt Riedemann (mriedem)]

Patch is here: https://review.openstack.org/#/c/61310/

[wangpan (hzwangpan)]

https://review.openstack.org/#/c/43984/
the bug is caused by this commit, and Xu Chen has already find this bug.

[wangpan (hzwangpan)]

I read code in libvirt python library, It will return -1 in this case.
Function Details:

#define VIR_PY_INT_FAIL (libvirt_intWrap(-1))

static PyObject *
libvirt_virConnectBaselineCPU(PyObject *self ATTRIBUTE_UNUSED,
                              PyObject *args) {
    PyObject *pyobj_conn;
    PyObject *list;
    virConnectPtr conn;
    unsigned int flags;
    const char **xmlcpus = NULL;
    int ncpus = 0;
    char *base_cpu;
    PyObject *pybase_cpu;

    if (!PyArg_ParseTuple(args, (char *)"OOi:virConnectBaselineCPU",
                          &pyobj_conn, &list, &flags))
        return NULL;
    conn = (virConnectPtr) PyvirConnect_Get(pyobj_conn);

    if (PyList_Check(list)) {
        size_t i;

        ncpus = PyList_Size(list);
        if (VIR_ALLOC_N_QUIET(xmlcpus, ncpus) < 0)
            return VIR_PY_INT_FAIL;

        for (i = 0; i < ncpus; i++) {
            xmlcpus[i] = PyString_AsString(PyList_GetItem(list, i));
            if (xmlcpus[i] == NULL) {
                VIR_FREE(xmlcpus);
                return VIR_PY_INT_FAIL;
            }
        }
    }

    LIBVIRT_BEGIN_ALLOW_THREADS;
    base_cpu = virConnectBaselineCPU(conn, xmlcpus, ncpus, flags);
    LIBVIRT_END_ALLOW_THREADS;
   ...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/61310
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5228d01bcde531cc2c141f38a150dbb2daef1c07
Submitter: Jenkins
Branch: master

commit 5228d01bcde531cc2c141f38a150dbb2daef1c07
Author: Guan Qiang <email address hidden>
Date: Wed Dec 11 03:21:34 2013 +0000

    lxc: Fix a bug of baselineCPU parse failure

    When starting nova-compute on compute node with lxc hypervisor,
    a valueError exception will be raised because a libvirt bug of
    the incorrect return value of baselineCPU() function, in function
    get_host_capabilities, which introduced by commit
    3b8c26321bce07b8ad588099c1fdcef858e748a8.

    For libvirt interface baselineCPU() in python library, -1 will be
    returned if basecpu info is None from version 1.1.2, and the return
    value of this interface is corrected to None from version 1.2.0.

    So a checking is added to make sure the return value of baselineCPU()
    is not -1, and also a test case is added.

    Closes-bug: #1259796

    Co-authored-by: Wangpan <email address hidden>

    Change-Id: I5e4cb4a8d09b660edc4f6c6a4ad4bfe58a4dd5cb

[Bhuvan Arumugam (bhuvan)]

For the record, the above fix break unit tests. Filed https://bugs.launchpad.net/nova/+bug/1266534 to fix it.