Libvirt native thread used for "forbidden" purposes

Bug #1290487 reported by John Warren
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
John Warren

Bug Description

In the nova.virt.libvirt.driver.LibvirtDriver. _get_new_connection method two different libvirt event handlers are registered, one for lifecycle events (_event_lifecycle_callback) and one for connection events (_close_callback). These callbacks are called by a native thread that is continually calling libvirt.virEventRunDefaultImpl() in the _native_thread method; the latter method's Docstring contains the following note:

        This is a native thread which runs the default
        libvirt event loop implementation. This processes
        any incoming async events from libvirtd and queues
        them for later dispatch. This thread is only
        permitted to use libvirt python APIs, and the
        driver.queue_event method. In particular any use
        of logging is forbidden, since it will confuse
        eventlet's greenthread integration

while this rule is adhered to by the _event_lifecycle_callback method, it is violated by _close_callback (the other callback) because it calls the _set_host_enabled method which, among other things, writes to the log.

The _close_callback method needs to be altered so that it does not execute any logic that may interfere with eventlet's greenthread integration.

Tags: libvirt
Matt Riedemann (mriedem)
tags: added: libvirt
Changed in nova:
status: New → Confirmed
Revision history for this message
John Warren (jswarren) wrote :

Correction: the _close_callback is not being called by the native thread that is running _native_thread. It is being called by the native thread that is spawned by tpool.proxy_call which is used to instantiate the connection.

Changed in nova:
assignee: nobody → John Warren (jswarren)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/79617
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=dba898226ec26ab49cecd24e9d6c255cf6153cd1
Submitter: Jenkins
Branch: master

commit dba898226ec26ab49cecd24e9d6c255cf6153cd1
Author: John Warren <email address hidden>
Date: Tue Mar 11 14:40:31 2014 +0000

    Change libvirt close callback to use green thread

    The native thread that calls the close callback is being
    used to update a host's status, potentially causing threading
    issues, since it involves operations that should only be
    performed by green threads. This change fixes the issue
    by limiting the native thread's interaction to adding data
    to a queue and leaving the remaining work for
    a green thread.

    Closes-Bug: #1290487
    Closes-Bug: #1293641

    Change-Id: I59c14da15e4655dc4bf21d4de3d509472b146f7a

Changed in nova:
status: In Progress → Fix Committed
Changed in nova:
milestone: none → icehouse-rc1
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: icehouse-rc1 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.