OpenStack Compute (Nova)

single xenapi session causes concurrency issues

Reported by Chris Behrens on 2011-10-20
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Johannes Erdfelt

Bug Description

Multiple XenAPI requests can happen at the same time... but there's an issue with that because we only use a single XenAPISession().

I think it's the cause of tracebacks like this periodically:

(nova.rpc): TRACE: File "/usr/local/lib/python2.6/dist-packages/XenAPI.py", line 229, in __call__
(nova.rpc): TRACE: return self.__send(self.__name, args)
(nova.rpc): TRACE: File "/usr/local/lib/python2.6/dist-packages/XenAPI.py", line 133, in xenapi_request
(nova.rpc): TRACE: result = _parse_result(getattr(self, methodname)(*full_params))
(nova.rpc): TRACE: File "/usr/local/lib/python2.6/dist-packages/XenAPI.py", line 203, in _parse_result
(nova.rpc): TRACE: raise Failure(result['ErrorDescription'])
(nova.rpc): TRACE: Failure: ['HANDLE_INVALID', 'VM', 'OpaqueRef:d1054be1-0ad6-532a-fa87-60ea50b863e2']

It has been also reported that httplib will sometimes return CannotSendRequest exceptions... when there appears to be multiple XenAPI requests at nearly the same time...

Chris Behrens (cbehrens) on 2011-10-20
Changed in nova:
importance: Undecided → High
milestone: none → essex-1
description: updated
Changed in nova:
assignee: nobody → Johannes Erdfelt (johannes.erdfelt)
status: New → In Progress

CannotSendRequest exceptions are a result of using an httplib connection in an invalid state. That's certainly a bug in the calling code, which appears to be XenAPI.

Turns out this problem only appears on Python 2.7. This is because Python 2.6 would always create a new HTTP connection in xmlrpclib, but in Python 2.7, it will reuse one HTTP connection.

Since a new connection was always created in Python 2.6, there would be shared state between threads and the problem wouldn't be reproduced.

This would only cause the CannotSendRequest exception, not the XenAPI HANDLE_INVALID failure.

Reviewed: https://review.openstack.org/1069
Committed: http://github.com/openstack/nova/commit/97cfccc2ce35b29fd98809b6c7ecdd86c56f76e3
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 97cfccc2ce35b29fd98809b6c7ecdd86c56f76e3
Author: Johannes Erdfelt <email address hidden>
Date: Mon Oct 24 21:55:12 2011 +0000

    Fix concurrency of XenAPI sessions

    Fixes bug 879044

    Nova currently does not serialize access to the XenAPI session which can
    result in multiple (green)threads trying to use the same HTTP connection.
    This will typically only affect Python 2.7 which has updated xmlrpclib to
    try to use one HTTP connection for multiple requests.

    Change-Id: I101d63b822c8bf8c28674a836e4b54aa8259e1a8

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2011-11-17
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-1 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers