libvirtd restart can sometimes cause multiple nova-compute connections

[Tom Hancock]

libvirt driver's get_connection() is not thread safe in the
presence of a libvirtd restart during concurrent incoming
requests.

With existing code, each request will in turn call get_connection,
find the connection is broken, try to create new one, block
for a while and yield to the next thread to do the same.
You get as many connections as there are incoming requests
and only the last one is used finally. If enough are incoming
these connections can exhaust the client pool configured
for libvirtd.

One fix is to hold a lock while creating the connection.
Note that has_min_version calls _conn which calls get_connection
so the lock may not be held over the call to has_min_version()

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/52401
Committed: http://github.com/openstack/nova/commit/b2e64e379835f57128e66f507438130eda716814
Submitter: Jenkins
Branch: master

commit b2e64e379835f57128e66f507438130eda716814
Author: Tom Hancock <email address hidden>
Date: Thu Oct 17 09:48:54 2013 +0000

    make libvirt driver get_connection thread-safe

    libvirt driver's get_connection is not thread safe in the
    presence of a libvirtd restart during concurrent incoming
    requests.

    With existing code each will in turn call get_connection,
    find the connection is broken, try to create new one, block
    for a while and yield to the next thread to do the same.
    You get as many connections as there are incoming requests
    and only the last one is used finally. If enough are incoming
    these connections can exhaust the client pool configured
    for libvirtd.
    One fix is to hold a lock while creating the connection.
    Note that has_min_version calls _conn which calls get_connection
    and thus the direct call to _has_min_version()

    Also added the exception text if it fails to register an event
    handler for lifecycle events.

    Change-Id: I090765802bfe443440f16722bc7c43b6280fe56a
    Fixes: bug #1240905

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/54595

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/54595
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=98ab49bbb29890730ce544b785f1babff3e694e1
Submitter: Jenkins
Branch: stable/havana

commit 98ab49bbb29890730ce544b785f1babff3e694e1
Author: Tom Hancock <email address hidden>
Date: Thu Oct 17 09:48:54 2013 +0000

    make libvirt driver get_connection thread-safe

    libvirt driver's get_connection is not thread safe in the
    presence of a libvirtd restart during concurrent incoming
    requests.

    With existing code each will in turn call get_connection,
    find the connection is broken, try to create new one, block
    for a while and yield to the next thread to do the same.
    You get as many connections as there are incoming requests
    and only the last one is used finally. If enough are incoming
    these connections can exhaust the client pool configured
    for libvirtd.
    One fix is to hold a lock while creating the connection.
    Note that has_min_version calls _conn which calls get_connection
    and thus the direct call to _has_min_version()

    Also added the exception text if it fails to register an event
    handler for lifecycle events.

    Change-Id: I090765802bfe443440f16722bc7c43b6280fe56a
    Fixes: bug #1240905
    (cherry picked from commit b2e64e379835f57128e66f507438130eda716814)