VMware: concurrent access error when deleting snapshot

Bug #1310817 reported by Ryan Hsu on 2014-04-21
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Gary Kotton

Bug Description

The VMware Minesweeper CI is occasionally seeing an error when deleting snapshots. The error is:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
    incoming.message))
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)
  File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
    payload)
  File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/nova/nova/exception.py", line 71, in wrapped
    return f(self, context, *args, **kw)
  File "/opt/stack/nova/nova/compute/manager.py", line 280, in decorated_function
    pass
  File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/nova/nova/compute/manager.py", line 266, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/opt/stack/nova/nova/compute/manager.py", line 309, in decorated_function
    e, sys.exc_info())
  File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/nova/nova/compute/manager.py", line 296, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/opt/stack/nova/nova/compute/manager.py", line 2692, in backup_instance
    task_states.IMAGE_BACKUP)
  File "/opt/stack/nova/nova/compute/manager.py", line 2758, in _snapshot_instance
    update_task_state)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 645, in snapshot
    _vmops.snapshot(context, instance, name, update_task_state)
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 873, in snapshot
    self._delete_vm_snapshot(instance, vm_ref, snapshot)
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 780, in _delete_vm_snapshot
    self._session._wait_for_task(delete_snapshot_task)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 948, in _wait_for_task
    ret_val = done.wait()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
    return hubs.get_hub().switch()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
    return self.greenlet.switch()
VMwareDriverException: A general system error occurred: concurrent access

Full logs for an affected run can be found here: http://10.148.255.241/logs/85961/2

Gary Kotton (garyk) on 2014-04-23
tags: added: icehouse-backport-potential
Changed in nova:
importance: Undecided → High
assignee: nobody → Gary Kotton (garyk)
milestone: none → juno-1
Changed in nova:
status: New → In Progress
John Garbutt (johngarbutt) wrote :

We use milestones only at the very end of the milestone to track bugs that should block the release of the milestone. Please use tags not milestones to track bugs, try things like: my_bug_topic_milestone_name

Changed in nova:
milestone: juno-1 → none
John Garbutt (johngarbutt) wrote :

Gerrit seems to have failed to link the patch here, maybe its worth adding it?

Gary Kotton (garyk) on 2014-07-03
Changed in nova:
milestone: none → juno-2
Gary Kotton (garyk) wrote :
John Garbutt (johngarbutt) wrote :

We use milestones only at the very end of the milestone to track bugs that should block the release of the milestone. Please use tags not milestones to track bugs, try things like: my_bug_topic_milestone_name

Changed in nova:
milestone: juno-2 → none

Reviewed: https://review.openstack.org/89530
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f89d13b141eba66487b3d858cd075a47b2de6016
Submitter: Jenkins
Branch: master

commit f89d13b141eba66487b3d858cd075a47b2de6016
Author: Gary Kotton <email address hidden>
Date: Tue Apr 22 03:36:51 2014 -0700

    VMware: handle case when VM snapshot delete fails

    Minesweeper fails on occasion with concurrent access to files.

    If a snapshot deletion fails with the backend exception
    TaskInProgress then we will retry.

    A generic decorator has been added to do the retry operations.

    Closes-bug: #1310817

    Change-Id: I8ed7a24ccd34aeea49352ac98f34ec2960edbf97

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2014-09-05
Changed in nova:
milestone: none → juno-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2014-10-16
Changed in nova:
milestone: juno-3 → 2014.2
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers