xenapi network info injection is racey

Bug #1190026 reported by Chris Behrens on 2013-06-11
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Chris Behrens

Bug Description

Xenapi's vmops.py:inject_network_info() is racey. If 2 network updates are happening at the same time, this can be raised:

['MAP_DUPLICATE_KEY', 'VM', 'xenstore_data', 'OpaqueRef:<REF>', 'vm-data/networking/<MACADDR>']
  File "/opt/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 234, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/opt/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 2755, in add_fixed_ip_to_instance
    network_info = self._inject_network_info(context, instance=instance)
  File "/opt/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 2916, in _inject_network_info
    self._legacy_nw_info(network_info))
  File "/opt/nova/lib/python2.6/site-packages/nova/virt/xenapi/driver.py", line 290, in inject_network_info
    self._vmops.inject_network_info(instance, network_info)
  File "/opt/nova/lib/python2.6/site-packages/nova/virt/xenapi/vmops.py", line 1641, in inject_network_info
    jsonutils.dumps(xs_data))
  File "/opt/nova/lib/python2.6/site-packages/nova/virt/xenapi/vmops.py", line 1813, in _add_to_param_xenstore
    self._session.call_xenapi('VM.add_to_xenstore_data', vm_ref, key, val)
  File "/opt/nova/lib/python2.6/site-packages/nova/virt/xenapi/driver.py", line 723, in call_xenapi
    return session.xenapi_request(method, args)
  File "/opt/nova/lib/python2.6/site-packages/XenAPI.py", line 133, in xenapi_request
    result = _parse_result(getattr(self, methodname)(*full_params))
  File "/opt/nova/lib/python2.6/site-packages/XenAPI.py", line 203, in _parse_result
    raise Failure(result['ErrorDescription'])

There's a 'remove' and then a 'add' in _add_to_param_xenstore. This is racey.

Fix proposed to branch: master
Review: https://review.openstack.org/32627

Changed in nova:
assignee: nobody → Chris Behrens (cbehrens)
status: New → In Progress
Chris Behrens (cbehrens) on 2013-06-11
Changed in nova:
importance: Undecided → Medium

Reviewed: https://review.openstack.org/32627
Committed: http://github.com/openstack/nova/commit/10c46619479e41e85b14343bd32159efda32775b
Submitter: Jenkins
Branch: master

commit 10c46619479e41e85b14343bd32159efda32775b
Author: Chris Behrens <email address hidden>
Date: Tue Jun 11 19:00:02 2013 +0000

    Fix race conditions with xenstore

    xenapi code updates xenstore using _add_to_param_xenstore(), which has
    to ensure a key doesn't exist before it can add the same key. However,
    if you have 2 calls at the same time, this is racey.

    Fixes bug 1190026

    Change-Id: I545b7a9fab0bc4c3749bc75387ed3f6ff27a512f

Changed in nova:
status: In Progress → Fix Committed
tags: added: xenserver
Thierry Carrez (ttx) on 2013-07-17
Changed in nova:
milestone: none → havana-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2013-10-17
Changed in nova:
milestone: havana-2 → 2013.2
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers