Libvirt's close callback causes deadlocks
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
High
|
John Warren |
Bug Description
Libvirt's close callback causes deadlocks
Unlike libvirt's lifecycle event callback which is triggered every time an event occurs, the close callback is only triggered when an attempt is made to use a connection that has been closed. In that case, the sequence of events is usually as follows:
LibvirtDriver.
LibvirtDriver.
LibvirtDriver.
libvirt.
LibvirtDriver.
_get_connection cannot release the lock because it is waiting for _close_callback to return and the latter cannot complete until it has acquired the lock.
Making the handling of the close callback asynchronous (like the lifecycle event handling) won't work, because by the time the lock is released, the connection object that was passed into the callback will no longer be equal to LibvirtDriver.
The best solution would appear to be to simply not register a close callback. The only case where it might provide some benefit is when a connection is closed after _get_connection has returned a reference to it. The benefit of disabling the instance a little earlier in such marginal cases is arguably outweighed by the complexity of a thread-safe implementation, especially when the difficulty of testing such an implementation (to ensure it is indeed thread safe) is taken into consideration.
Note that having _close_callback use _wrapped_
Changed in nova: | |
status: | Incomplete → Confirmed |
importance: | Undecided → High |
status: | Confirmed → In Progress |
assignee: | nobody → John Warren (jswarren) |
tags: | added: libvirt |
Changed in nova: | |
status: | Fix Committed → Fix Released |
Changed in nova: | |
milestone: | icehouse-rc1 → 2014.1 |
adding to rc1 for visibility