AttributeError: virConnect instance has no attribute 'registerCloseCallback'

Bug #1266711 reported by Sahid Orentino on 2014-01-07
24
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Critical
Sahid Orentino
Havana
Undecided
Unassigned
OpenStack Core Infrastructure
Fix Released
Critical
Jeremy Stanley

Bug Description

During Jenkins tests I got this error two times with a different patchset

ft1.8205: nova.tests.virt.libvirt.test_libvirt.LibvirtNonblockingTestCase.test_connection_to_primitive_StringException: Empty attachments:
  stderr
  stdout

pythonlogging:'': {{{WARNING [nova.virt.libvirt.driver] URI test:///default does not support events: internal error: could not initialize domain event timer}}}

Traceback (most recent call last):
  File "nova/tests/virt/libvirt/test_libvirt.py", line 7570, in test_connection_to_primitive
    jsonutils.to_primitive(connection._conn, convert_instances=True)
  File "nova/virt/libvirt/driver.py", line 678, in _get_connection
    wrapped_conn = self._get_new_connection()
  File "nova/virt/libvirt/driver.py", line 664, in _get_new_connection
    wrapped_conn.registerCloseCallback(
  File "/home/jenkins/workspace/gate-nova-python27/.tox/py27/local/lib/python2.7/site-packages/eventlet/tpool.py", line 172, in __getattr__
    f = getattr(self._obj,attr_name)
AttributeError: virConnect instance has no attribute 'registerCloseCallback'

Sean Dague (sdague) wrote :

This is a 100% failure at this point in the gate, categorized by - http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiQXR0cmlidXRlRXJyb3I6IHZpckNvbm5lY3QgaW5zdGFuY2UgaGFzIG5vIGF0dHJpYnV0ZSAncmVnaXN0ZXJDbG9zZUNhbGxiYWNrJ1wiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiIxNzI4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzg5MDg0ODQwODAyfQ==

Until this gets fixed, all nova changes will fail.

Based on the timing of when it started showing up, this was probably triggered by a change in the base image that nodepool created, possibly a libvirt upgrade? Unfortunately for unit tests we don't dump the package list, which would make that easier to debug.

Changed in nova:
status: New → Confirmed
importance: Undecided → Critical
Ken'ichi Ohmichi (oomichi) wrote :

How about adding the existence check for wrapped_conn.registerCloseCallback in _get_new_connection() ?

Sean, how we can know the used version of the python binding of libvirt?

Ken'ichi Ohmichi (oomichi) wrote :

libvirt 1.1.4 contains registerCloseCallback in python/libvirt-override-virConnect.py:

http://libvirt.org/git/?p=libvirt.git;a=blob;f=python/libvirt-override-virConnect.py;h=4ba3d308757da1eed8f96c9827e71b956a58b20f;hb=a4a7c7e9c4df0c9dbf304466c420b98ccf271d16

but I cannot find the python files in libvirt 1.2.0, anyone know it?

Chmouel Boudjnah (chmouel) wrote :

There is another attempt by Sahid to correct it here: https://review.openstack.org/#/c/65272

Changed in nova:
milestone: none → icehouse-2
Sean Dague (sdague) wrote :

Sahid's approach is incorrect.

The crux of the issue seems to be that

MIN_LIBVIRT_CLOSE_CALLBACK_VERSION = (1, 0, 1)

is *wrong*

because Ubuntu's libvirt 1.1.1 doesn't have that interface, and any time we trip over a cloud archive upgrade it doesn't work.

Our logic for detecting libvirt features is entirely based on version number, which is apparently insufficient. This particular nova bug has actually kept us off using the ubuntu cloud archive, which means we're not testing any other libvirt >= 1.x bits as well, which means there are a lot of other places we could be breaking that we don't know about yet.

I think that Kenichi's approach of actually checking for the attr is the right approach, and remove testing for this feature by version at all. Honestly moving to a model where we are not testing by version but instead by feature seems more sane.

Jeremy Stanley (fungi) on 2014-01-07
Changed in openstack-ci:
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → Jeremy Stanley (fungi)
milestone: none → icehouse
Ken'ichi Ohmichi (oomichi) wrote :

https://review.openstack.org/65267 is updated based on Sean's comment.

Jeremy Stanley (fungi) wrote :

The trigger for this was an inadvertent upgrade to libvirt 1.11 on all the slaves. When we determined that http://git.openstack.org/cgit/openstack-infra/config/commit/?id=bdcc115 was broken and reverted it with http://git.openstack.org/cgit/openstack-infra/config/commit/?id=f7b9581 (back in late September), we neglected to remove the /etc/apt/sources.list.d/cloudarchive.list file it left behind. This went unnoticed because unattended-upgrade job does not upgrade from unofficial repositories unless explicitly whitelisted, so it wasn't until early this morning when we approved http://git.openstack.org/cgit/openstack-infra/config/commit/?id=0385b96 that apt-get ended up pulling libvirt-dev and its dependencies from Ubuntu Cloud Archive rather than main that this came to light.

The ensuing mayhem was resolved by merging http://git.openstack.org/cgit/openstack-infra/config/commit/?id=7282ca4 and then using salt.run to execute the following on all precise.* slaves:

    rm -f /etc/apt/sources.list.d/cloudarchive.list
    apt-get update
    apt-get install -y --force-yes \
        libvirt-bin=0.9.8-2ubuntu17.16 \
        libvirt-dev=0.9.8-2ubuntu17.16 \
        libvirt0=0.9.8-2ubuntu17.16 \
        libxenstore3.0=4.1.5-0ubuntu0.12.04.2 \
        libxen-dev=4.1.5-0ubuntu0.12.04.2

Since then, subsequent unit tests are back to using old libvirt in CI and passing normally.

Changed in openstack-ci:
status: In Progress → Fix Released
Jeremy Stanley (fungi) wrote :

It's also worth mentioning that this is probably related to bug 1228977 (maybe even a duplicate), which was the reason we reverted it in the first place.

Changed in nova:
assignee: nobody → sahid (sahid-ferdjaoui)
Bhuvan Arumugam (bhuvan) wrote :

Sahid, you may find the libvirt version from python api, as follows:

>>> import libvirt
>>> libvirt.getVersion()
10002

Changed in nova:
status: Confirmed → In Progress

ubuntu cloud archive now has libvirt at version 1.1.1 (specifically 1.1.1-0ubuntu8~cloud2). Looking at:
http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/precise-updates/havana/main/binary-i386/Packages

Thierry Carrez (ttx) on 2014-01-22
Changed in nova:
milestone: icehouse-2 → icehouse-3

Reviewed: https://review.openstack.org/65267
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=85068cc9f68aa704895ffe5ac185bb9cf7d05e2d
Submitter: Jenkins
Branch: master

commit 85068cc9f68aa704895ffe5ac185bb9cf7d05e2d
Author: Ken'ichi Ohmichi <email address hidden>
Date: Wed Jan 8 06:43:42 2014 +0900

    Fix bug with not implemented virConnect.registerCloseCallback

    The unit test test_connection_to_primitive fails many times due to
    "virConnect instance has no attribute 'registerCloseCallback'".

    registerCloseCallback has been added since original(community's)
    python-libvirt v1.0.0, and v1.2.0 also contains the method.
    However ubuntu cloud's v1.0.1 does not contain the method, and current
    version check of python-libvirt does not work.
    This patch tries to operate the method and catch TypeError exception if
    the method does not exist instead of the version check.

    Co-Authored-By: Sahid Orentino Ferdjaoui <email address hidden>
    Closes-Bug: #1266711
    Change-Id: I4ec9ff9a684639ae5b146f400c90115c83afcda7

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2014-03-05
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-04-17
Changed in nova:
milestone: icehouse-3 → 2014.1
Matt Riedemann (mriedem) wrote :

Reviewed: https://review.openstack.org/112422
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6484fc6f0f4c8a67192804592150ef0185f6a7cc
Submitter: Jenkins
Branch: stable/havana

commit 6484fc6f0f4c8a67192804592150ef0185f6a7cc
Author: Ken'ichi Ohmichi <email address hidden>
Date: Wed Jan 8 06:43:42 2014 +0900

    Fix bug with not implemented virConnect.registerCloseCallback

    The unit test test_connection_to_primitive fails many times due to
    "virConnect instance has no attribute 'registerCloseCallback'".

    registerCloseCallback has been added since original(community's)
    python-libvirt v1.0.0, and v1.2.0 also contains the method.
    However ubuntu cloud's v1.0.1 does not contain the method, and current
    version check of python-libvirt does not work.
    This patch tries to operate the method and catch TypeError exception if
    the method does not exist instead of the version check.

    Conflicts:
            nova/tests/virt/libvirt/test_libvirt.py

    NOTE(mriedem): The conflict is due to set_host_enabled
    calls which aren't in stable/havana.

    Co-Authored-By: Sahid Orentino Ferdjaoui <email address hidden>
    Closes-Bug: #1266711
    Change-Id: I4ec9ff9a684639ae5b146f400c90115c83afcda7
    (cherry picked from commit 85068cc9f68aa704895ffe5ac185bb9cf7d05e2d)

tags: added: in-stable-havana
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers