cold migration fails in VMware driver

Bug #1270332 reported by Sabari Murugesan on 2014-01-18
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Wishlist
Radoslav Gerganov

Bug Description

With two compute nodes (on different hosts) configured to two different clusters in the same vCenter Server i.e :-

nova migrate <server> fails to migrate a server with the following error :-

2014-01-17 16:00:21.336 ERROR nova.openstack.common.rpc.amqp [req-0c587eb7-3a23-4790-b23d-f4ad005b5fe7 admin admin] Exception during message handling
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp **args)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 90, in wrapped
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp payload)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp six.reraise(self.type_, self.value, self.tb)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 73, in wrapped
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 244, in decorated_function
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp pass
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp six.reraise(self.type_, self.value, self.tb)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 230, in decorated_function
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 295, in decorated_function
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 272, in decorated_function
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp six.reraise(self.type_, self.value, self.tb)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 259, in decorated_function
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3163, in resize_instance
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp block_device_info)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 406, in migrate_disk_and_power_off
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp dest, flavor)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 1182, in migrate_disk_and_power_off
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp self._session._wait_for_task(instance['uuid'], vm_clone_task)
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 857, in _wait_for_task
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp ret_val = done.wait()
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp return hubs.get_hub().switch()
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp return self.greenlet.switch()
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp NovaException: Unable to access the virtual machine configuration: Unable to access file [datastore1]
2014-01-17 16:00:21.336 TRACE nova.openstack.common.rpc.amqp
2014-01-17 16:00:21.497 DEBUG nova.openstack.common.rpc.amqp [-] Making synchronous call on conductor ... from (pid=28144) multicall /opt/stack/nova/nova/openstack/common/rpc/amqp.py:553
2014-01-17 16:00:21.498 DEBUG nova.openstack.common.rpc.amqp [-] MSG_ID is b52829e908144155970ef23754eb46e3 from (pid=28144) multicall /opt/stack/nova/nova/openstack/common/rpc/amqp.py:556
2014-01-17 16:00:21.498 DEBUG nova.openstack.common.rpc.amqp [-] UNIQUE_ID is 393d7230809b4892a4d7834c5ab56c2f. from (pid=28144) _add_unique_id /opt/stack/nova/nova/openstack/common/rpc/amqp.py:341
2014-01-17 16:00:25.640 WARNING nova.virt.vmwareapi.driver [-] Task [CloneVM_Task] (returnval){
   value = "task-110"
   _type = "Task"
 } status: error Unable to access the virtual machine configuration: Unable to access file [datastore1]

tags: added: vmware
Sabari Murugesan (smurugesan) wrote :

This is because vCenter server is trying to do the following :-

in relocate config spec :-

1. datastore is chosen from the target cluster.
2. host is set to null

The error is because vCenter tries to migrate the VM within the source host, but with a datastore available with the target cluster.

Ideally, the following should have happened in relocate config spec :-

1. datastore is chosen from the target cluster.
2. host is set to null.
3. resource pool is set to the target cluster.

Gary Kotton (garyk) on 2014-01-28
Changed in nova:
status: New → Confirmed
importance: Undecided → Medium
Changed in nova:
assignee: nobody → Sabari Murugesan (smurugesan)
milestone: none → icehouse-3
Changed in nova:
status: Confirmed → In Progress
Thierry Carrez (ttx) on 2014-03-05
Changed in nova:
milestone: icehouse-3 → icehouse-rc1
Tracy Jones (tjones-i) on 2014-03-07
Changed in nova:
milestone: icehouse-rc1 → next
tags: added: icehouse-rc-potential
Sabari Murugesan (smurugesan) wrote :
Tracy Jones (tjones-i) on 2014-03-22
tags: added: icehouse-rc1-potential
removed: icehouse-rc-potential
Tracy Jones (tjones-i) on 2014-03-24
tags: removed: icehouse-rc1-potential
Sean Dague (sdague) on 2014-09-15
Changed in nova:
status: In Progress → Confirmed
assignee: Sabari Murugesan (smurugesan) → nobody

Is this still a problem?

Changed in nova:
status: Confirmed → Incomplete
Tracy Jones (tjones-i) wrote :

rado is working on this

Changed in nova:
status: Incomplete → In Progress
assignee: nobody → Radoslav Gerganov (rgerganov)

@Radoslav:
Could you already make progress here?

@Radoslav: Can you please let us know if you are still working on this bug ?

Radoslav Gerganov (rgerganov) wrote :

Yes, I am working on this. The cold migration shares common code with the live migration which is under review:

https://review.openstack.org/#/c/270116/

I was hoping to get some feedback there before pushing all the patches.

Augustina Ragwitz (auggy) wrote :

If this issue is addressed in that change, could you consider adding a "closes-bug" tag to the commit message?

Looks like this will be driven by the blueprint [1].

[1] https://blueprints.launchpad.net/nova/+spec/vmware-live-migration

Changed in nova:
status: In Progress → Opinion
importance: Medium → Wishlist
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers