AttributeError: virConnect instance has no attribute 'registerCloseCallback'

[Sahid Orentino]

During Jenkins tests I got this error two times with a different patchset

ft1.8205: nova.tests.virt.libvirt.test_libvirt.LibvirtNonblockingTestCase.test_connection_to_primitive_StringException: Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{WARNING [nova.virt.libvirt.driver] URI test:///default does not support events: internal error: could not initialize domain event timer}}}

Traceback (most recent call last):
  File "nova/tests/virt/libvirt/test_libvirt.py", line 7570, in test_connection_to_primitive
    jsonutils.to_primitive(connection._conn, convert_instances=True)
  File "nova/virt/libvirt/driver.py", line 678, in _get_connection
    wrapped_conn = self._get_new_connection()
  File "nova/virt/libvirt/driver.py", line 664, in _get_new_connection
    wrapped_conn.registerCloseCallback(
  File "/home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/eventlet/tpool.py", line 172, in __getattr__
    f = getattr(self._obj,attr_name)
AttributeError: virConnect instance has no attribute 'registerCloseCallback'

[Sean Dague (sdague)]

This is a 100% failure at this point in the gate, categorized by - http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiQXR0cmlidXRlRXJyb3I6IHZpckNvbm5lY3QgaW5zdGFuY2UgaGFzIG5vIGF0dHJpYnV0ZSAncmVnaXN0ZXJDbG9zZUNhbGxiYWNrJ1wiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiIxNzI4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzg5MDg0ODQwODAyfQ==

Until this gets fixed, all nova changes will fail.

Based on the timing of when it started showing up, this was probably triggered by a change in the base image that nodepool created, possibly a libvirt upgrade? Unfortunately for unit tests we don't dump the package list, which would make that easier to debug.

[Ken'ichi Ohmichi (oomichi)]

How about adding the existence check for wrapped_conn.registerCloseCallback in _get_new_connection() ?

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/65267

[Sahid Orentino (sahid-ferdjaoui)]

Sean, how we can know the used version of the python binding of libvirt?

[Sahid Orentino (sahid-ferdjaoui)]

libvirt can be used as fakelibvirt:
    https://github.com/openstack/nova/blob/master/nova/tests/virt/libvirt/test_libvirt.py#L81

if it is the case, there is no method registerCloseCallback defined :
https://github.com/openstack/nova/blob/master/nova/tests/virt/libvirt/test_fakelibvirt.py

[Sahid Orentino (sahid-ferdjaoui)]

Can you take a look? https://review.openstack.org/#/c/65272/

[Ken'ichi Ohmichi (oomichi)]

libvirt 1.1.4 contains registerCloseCallback in python/libvirt-override-virConnect.py:

http://libvirt.org/git/?p=libvirt.git;a=blob;f=python/libvirt-override-virConnect.py;h=4ba3d308757da1eed8f96c9827e71b956a58b20f;hb=a4a7c7e9c4df0c9dbf304466c420b98ccf271d16

but I cannot find the python files in libvirt 1.2.0, anyone know it?

[Chmouel Boudjnah (chmouel)]

There is another attempt by Sahid to correct it here: https://review.openstack.org/#/c/65272

[Sean Dague (sdague)]

Sahid's approach is incorrect.

The crux of the issue seems to be that

MIN_LIBVIRT_CLOSE_CALLBACK_VERSION = (1, 0, 1)

is *wrong*

because Ubuntu's libvirt 1.1.1 doesn't have that interface, and any time we trip over a cloud archive upgrade it doesn't work.

Our logic for detecting libvirt features is entirely based on version number, which is apparently insufficient. This particular nova bug has actually kept us off using the ubuntu cloud archive, which means we're not testing any other libvirt >= 1.x bits as well, which means there are a lot of other places we could be breaking that we don't know about yet.

I think that Kenichi's approach of actually checking for the attr is the right approach, and remove testing for this feature by version at all. Honestly moving to a model where we are not testing by version but instead by feature seems more sane.

[Ken'ichi Ohmichi (oomichi)]

https://review.openstack.org/65267 is updated based on Sean's comment.

[Jeremy Stanley (fungi)]

The trigger for this was an inadvertent upgrade to libvirt 1.11 on all the slaves. When we determined that http://git.openstack.org/cgit/openstack-infra/config/commit/?id=bdcc115 was broken and reverted it with http://git.openstack.org/cgit/openstack-infra/config/commit/?id=f7b9581 (back in late September), we neglected to remove the /etc/apt/sources.list.d/cloudarchive.list file it left behind. This went unnoticed because unattended-upgrade job does not upgrade from unofficial repositories unless explicitly whitelisted, so it wasn't until early this morning when we approved http://git.openstack.org/cgit/openstack-infra/config/commit/?id=0385b96 that apt-get ended up pulling libvirt-dev and its dependencies from Ubuntu Cloud Archive rather than main that this came to light.

The ensuing mayhem was resolved by merging http://git.openstack.org/cgit/openstack-infra/config/commit/?id=7282ca4 and then using salt.run to execute the following on all precise.* slaves:

    rm -f /etc/apt/sources.list.d/cloudarchive.list
    apt-get update
    apt-get install -y --force-yes \
        libvirt-bin=0.9.8-2ubuntu17.16 \
        libvirt-dev=0.9.8-2ubuntu17.16 \
        libvirt0=0.9.8-2ubuntu17.16 \
        libxenstore3.0=4.1.5-0ubuntu0.12.04.2 \
        libxen-dev=4.1.5-0ubuntu0.12.04.2

Since then, subsequent unit tests are back to using old libvirt in CI and passing normally.

[Jeremy Stanley (fungi)]

It's also worth mentioning that this is probably related to bug 1228977 (maybe even a duplicate), which was the reason we reverted it in the first place.

[Bhuvan Arumugam (bhuvan)]

Sahid, you may find the libvirt version from python api, as follows:

>>> import libvirt
>>> libvirt.getVersion()
10002

[Davanum Srinivas (DIMS) (dims-v)]

ubuntu cloud archive now has libvirt at version 1.1.1 (specifically 1.1.1-0ubuntu8~cloud2). Looking at:
http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/precise-updates/havana/main/binary-i386/Packages

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/65267
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=85068cc9f68aa704895ffe5ac185bb9cf7d05e2d
Submitter: Jenkins
Branch: master

commit 85068cc9f68aa704895ffe5ac185bb9cf7d05e2d
Author: Ken'ichi Ohmichi <email address hidden>
Date: Wed Jan 8 06:43:42 2014 +0900

    Fix bug with not implemented virConnect.registerCloseCallback

    The unit test test_connection_to_primitive fails many times due to
    "virConnect instance has no attribute 'registerCloseCallback'".

    registerCloseCallback has been added since original(community's)
    python-libvirt v1.0.0, and v1.2.0 also contains the method.
    However ubuntu cloud's v1.0.1 does not contain the method, and current
    version check of python-libvirt does not work.
    This patch tries to operate the method and catch TypeError exception if
    the method does not exist instead of the version check.

    Co-Authored-By: Sahid Orentino Ferdjaoui <email address hidden>
    Closes-Bug: #1266711
    Change-Id: I4ec9ff9a684639ae5b146f400c90115c83afcda7

[Matt Riedemann (mriedem)]

I'm looking to backport this to stable/havana but it requires (in order):

https://review.openstack.org/#/c/57000/
https://review.openstack.org/#/c/55866/
https://review.openstack.org/#/c/54595/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/112422

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/112422
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6484fc6f0f4c8a67192804592150ef0185f6a7cc
Submitter: Jenkins
Branch: stable/havana

commit 6484fc6f0f4c8a67192804592150ef0185f6a7cc
Author: Ken'ichi Ohmichi <email address hidden>
Date: Wed Jan 8 06:43:42 2014 +0900

    Fix bug with not implemented virConnect.registerCloseCallback

    The unit test test_connection_to_primitive fails many times due to
    "virConnect instance has no attribute 'registerCloseCallback'".

    registerCloseCallback has been added since original(community's)
    python-libvirt v1.0.0, and v1.2.0 also contains the method.
    However ubuntu cloud's v1.0.1 does not contain the method, and current
    version check of python-libvirt does not work.
    This patch tries to operate the method and catch TypeError exception if
    the method does not exist instead of the version check.

    Conflicts:
            nova/tests/virt/libvirt/test_libvirt.py

    NOTE(mriedem): The conflict is due to set_host_enabled
    calls which aren't in stable/havana.

    Co-Authored-By: Sahid Orentino Ferdjaoui <email address hidden>
    Closes-Bug: #1266711
    Change-Id: I4ec9ff9a684639ae5b146f400c90115c83afcda7
    (cherry picked from commit 85068cc9f68aa704895ffe5ac185bb9cf7d05e2d)