OSError Invalid cross-device link during resize

Bug #939916 reported by Mark Washenberger
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Mark Washenberger

Bug Description

Recent changes (https://review.openstack.org/4368) to the xen plugins fixed an issue where shutil.move was resulting in a copy even if a much faster rename would suffice. However, those fixes meant that if /images isn't on the same device as /var/run/sr-mount/<uuid>, then the resize would fail completely. Since it is totally possible that a xen hypervisor has many sr-mounts, and the default one could change when the compute manager is reconfigured, we should expect the plugin code to work even if /images is on a different device than the given sr-mount.

Here's an example trace I saw when this happened

2012-02-23 16:51:59 WARNING nova.virt.xenapi_conn [-] Task [Async.host.call_plugin] OpaqueRef:f5b9678f-a87b-015e-8b6a-0b48c16ad9ec status: failure ['XENAPI_PLUGIN_EXCEPTION', 'move_vhds_into_sr', 'OSError', '[Errno 18] Invalid cross-device link']
2012-02-23 16:51:59 ERROR nova.compute.manager [-] ['XENAPI_PLUGIN_EXCEPTION', 'move_vhds_into_sr', 'OSError', '[Errno 18] Invalid cross-device link']. Setting instance vm_state to ERROR
2012-02-23 16:51:59 ERROR nova.rpc.common [-] Exception during message handling
(nova.rpc.common): TRACE: Traceback (most recent call last):
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/rpc/amqp.py", line 250, in _process_data
(nova.rpc.common): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/exception.py", line 112, in wrapped
(nova.rpc.common): TRACE: return f(*args, **kw)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 156, in decorated_function
(nova.rpc.common): TRACE: function(self, context, instance_uuid, *args, **kwargs)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 180, in decorated_function
(nova.rpc.common): TRACE: sys.exc_info())
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/contextlib.py", line 23, in __exit__
(nova.rpc.common): TRACE: self.gen.next()
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 174, in decorated_function
(nova.rpc.common): TRACE: return function(self, context, instance_uuid, *args, **kwargs)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 1379, in finish_resize
(nova.rpc.common): TRACE: self._set_instance_error_state(context, instance_ref.uuid)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/contextlib.py", line 23, in __exit__
(nova.rpc.common): TRACE: self.gen.next()
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 1374, in finish_resize
(nova.rpc.common): TRACE: disk_info)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 1346, in _finish_resize
(nova.rpc.common): TRACE: image_meta, resize_instance)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/virt/xenapi_conn.py", line 223, in finish_migration
(nova.rpc.common): TRACE: network_info, image_meta, resize_instance)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/virt/xenapi/vmops.py", line 161, in finish_migration
(nova.rpc.common): TRACE: vdi_uuid = self._move_disks(instance, disk_info)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/virt/xenapi/vmops.py", line 844, in _move_disks
(nova.rpc.common): TRACE: self._session.wait_for_task(task, instance['uuid'])
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/virt/xenapi_conn.py", line 647, in wait_for_task
(nova.rpc.common): TRACE: raise self.XenAPI.Failure(error_info)
(nova.rpc.common): TRACE: Failure: ['XENAPI_PLUGIN_EXCEPTION', 'move_vhds_into_sr', 'OSError', '[Errno 18] Invalid cross-device link']
(nova.rpc.common): TRACE:

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/4473

Changed in nova:
assignee: nobody → Mark Washenberger (markwash)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/4473
Committed: http://github.com/openstack/nova/commit/0d487d4f2350e42d6a63febd413b70e663053a1b
Submitter: Jenkins
Branch: master

commit 0d487d4f2350e42d6a63febd413b70e663053a1b
Author: Mark Washenberger <email address hidden>
Date: Thu Feb 23 18:39:03 2012 -0500

    Copy data when migration dst is on a different FS

    Fixes bug 939916

    Change-Id: I678e15a13f99b59b16bd446f566b2c48dcba6057

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → essex-4
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: essex-4 → 2012.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.