OSError Invalid cross-device link during resize

Bug #939916 reported by Mark Washenberger on 2012-02-23
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Mark Washenberger

Bug Description

Recent changes (https://review.openstack.org/4368) to the xen plugins fixed an issue where shutil.move was resulting in a copy even if a much faster rename would suffice. However, those fixes meant that if /images isn't on the same device as /var/run/sr-mount/<uuid>, then the resize would fail completely. Since it is totally possible that a xen hypervisor has many sr-mounts, and the default one could change when the compute manager is reconfigured, we should expect the plugin code to work even if /images is on a different device than the given sr-mount.

Here's an example trace I saw when this happened

2012-02-23 16:51:59 WARNING nova.virt.xenapi_conn [-] Task [Async.host.call_plugin] OpaqueRef:f5b9678f-a87b-015e-8b6a-0b48c16ad9ec status: failure ['XENAPI_PLUGIN_EXCEPTION', 'move_vhds_into_sr', 'OSError', '[Errno 18] Invalid cross-device link']
2012-02-23 16:51:59 ERROR nova.compute.manager [-] ['XENAPI_PLUGIN_EXCEPTION', 'move_vhds_into_sr', 'OSError', '[Errno 18] Invalid cross-device link']. Setting instance vm_state to ERROR
2012-02-23 16:51:59 ERROR nova.rpc.common [-] Exception during message handling
(nova.rpc.common): TRACE: Traceback (most recent call last):
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/rpc/amqp.py", line 250, in _process_data
(nova.rpc.common): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/exception.py", line 112, in wrapped
(nova.rpc.common): TRACE: return f(*args, **kw)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 156, in decorated_function
(nova.rpc.common): TRACE: function(self, context, instance_uuid, *args, **kwargs)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 180, in decorated_function
(nova.rpc.common): TRACE: sys.exc_info())
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/contextlib.py", line 23, in __exit__
(nova.rpc.common): TRACE: self.gen.next()
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 174, in decorated_function
(nova.rpc.common): TRACE: return function(self, context, instance_uuid, *args, **kwargs)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 1379, in finish_resize
(nova.rpc.common): TRACE: self._set_instance_error_state(context, instance_ref.uuid)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/contextlib.py", line 23, in __exit__
(nova.rpc.common): TRACE: self.gen.next()
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 1374, in finish_resize
(nova.rpc.common): TRACE: disk_info)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/compute/manager.py", line 1346, in _finish_resize
(nova.rpc.common): TRACE: image_meta, resize_instance)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/virt/xenapi_conn.py", line 223, in finish_migration
(nova.rpc.common): TRACE: network_info, image_meta, resize_instance)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/virt/xenapi/vmops.py", line 161, in finish_migration
(nova.rpc.common): TRACE: vdi_uuid = self._move_disks(instance, disk_info)
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/virt/xenapi/vmops.py", line 844, in _move_disks
(nova.rpc.common): TRACE: self._session.wait_for_task(task, instance['uuid'])
(nova.rpc.common): TRACE: File "/usr/lib/python2.6/dist-packages/nova/virt/xenapi_conn.py", line 647, in wait_for_task
(nova.rpc.common): TRACE: raise self.XenAPI.Failure(error_info)
(nova.rpc.common): TRACE: Failure: ['XENAPI_PLUGIN_EXCEPTION', 'move_vhds_into_sr', 'OSError', '[Errno 18] Invalid cross-device link']
(nova.rpc.common): TRACE:

Fix proposed to branch: master
Review: https://review.openstack.org/4473

Changed in nova:
assignee: nobody → Mark Washenberger (markwash)
status: New → In Progress

Reviewed: https://review.openstack.org/4473
Committed: http://github.com/openstack/nova/commit/0d487d4f2350e42d6a63febd413b70e663053a1b
Submitter: Jenkins
Branch: master

commit 0d487d4f2350e42d6a63febd413b70e663053a1b
Author: Mark Washenberger <email address hidden>
Date: Thu Feb 23 18:39:03 2012 -0500

    Copy data when migration dst is on a different FS

    Fixes bug 939916

    Change-Id: I678e15a13f99b59b16bd446f566b2c48dcba6057

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2012-02-29
Changed in nova:
milestone: none → essex-4
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-4 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers