XenAPI Resize Fails with Migration Error: Failed to transfer vhd to new host

[Vangelis Tasoulas]

Setup:
OpenStack Grizzly and XCP on Debian Wheezy from the repositories (Kronos).
Shared storage based on NFS as the default SR configured for live migration.

When pressing the "More->Migrate Instance" on Horizon web interface, a resize is initiated (Migrate here is misleading since it's basically resize but this is another thing) which always fails with the following errors:

2013-07-16 17:49:35.684 ERROR nova.compute.manager [req-7a04e85c-0663-4772-bb5b-006516bde71f 8ca2f0ccd55f4d9ba53847755f2e0b18 10c16ce5ffb54c56925152b7d331a8d2] [instance: 93cb5f1a-5f63-4ac0-b7f4-77ee8e976484] Migration error: Failed to transfer vhd to new host. Setting instance vm_state to ERROR
2013-07-16 17:49:41.862 ERROR nova.openstack.common.rpc.amqp [req-7a04e85c-0663-4772-bb5b-006516bde71f 8ca2f0ccd55f4d9ba53847755f2e0b18 10c16ce5ffb54c56925152b7d331a8d2] Exception during message handling
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in decorated_function
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp pass
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in decorated_function
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2373, in resize_instance
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp block_device_info)
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/driver.py", line 243, in migrate_disk_and_power_off
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp dest, instance_type)
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 905, in migrate_disk_and_power_off
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp context, instance, dest, vm_ref, sr_path)
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 850, in _migrate_disk_resizing_up
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp self._migrate_vhd(instance, vdi_uuid, dest, sr_path, seq_num)
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/xenapi/vmops.py", line 765, in _migrate_vhd
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp raise exception.MigrationError(reason=msg)
2013-07-16 17:49:41.862 5363 TRACE nova.openstack.common.rpc.amqp MigrationError: Migration error: Failed to transfer vhd to new host

Exactly the same happens when I try to resize an instance from the command line (either up or down)

~# nova resize <image-id> m1.small --poll
Instance resizing... 40% complete
Error resizing instance

I assume it might be because of the shared storage?
Anyway it's a bug which puts the VM in ERROR state so it should be handled properly.

[Bob Ball (bob-ball)]

Agreed the shared storage is the likely cause. The resize code (as opposed to the migrate) is specifically intended for local storage.
VM migrate (from the command line) should work in this scenario

[Vangelis Tasoulas (cyberang3l)]

Live migration from the command line works with some effort: https://answers.launchpad.net/nova/+question/232484

[John Garbutt (johngarbutt)]

So we don't really fully support the use of shared storage with xenserver today, most of the things that work are down to "luck".

live-migrate was added to work with pools, but the pool support is very "dodgy".

You could use migrate between pools, and it will work.

Really, there should be a better error message when you try migrate with pools.

[Bob Ball (bob-ball)]

No current plans to fix this; use live migration not resize