VMwareAPI-Team

vmwareapi issues when VC has multiple datacenters defined

Bug #1208906 reported by dan wendlandt
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
High
Shawn Hartsock
OpenStack Compute (nova) havana-3 "h3"
VMwareAPI-Team
New
Undecided
Unassigned

Bug Description

When using a vCenter with multiple data centers, one can observe failures like the following:

2013-07-18 10:59:12.788 DEBUG nova.virt.vmwareapi.vmware_images [req-e8306ffe-c6c7-4d0f-a466-fb532375cbd3 7799f10ca7da47f3b2660feb363b370b 0e1771f8db984a3599596fae62609d9a] [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] Got image size of 687865856 for the image cde14862-60b8-4360-a145-06585b06577c get_vmdk_size_and_properties /usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmware_images.py:156
2013-07-18 10:59:12.963 WARNING nova.virt.vmwareapi.network_util [req-e8306ffe-c6c7-4d0f-a466-fb532375cbd3 7799f10ca7da47f3b2660feb363b370b 0e1771f8db984a3599596fae62609d9a] [(ManagedObjectReference){
   value = "network-1501"
   _type = "Network"
 }, (ManagedObjectReference){
   value = "network-1458"
   _type = "Network"
 }, (ManagedObjectReference){
   value = "network-2085"
   _type = "Network"
 }, (ManagedObjectReference){
   value = "network-1143"
   _type = "Network"
 }]
2013-07-18 10:59:13.326 DEBUG nova.virt.vmwareapi.vmops [req-e8306ffe-c6c7-4d0f-a466-fb532375cbd3 7799f10ca7da47f3b2660feb363b370b 0e1771f8db984a3599596fae62609d9a] [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] Creating VM on the ESX host _execute_create_vm /usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py:207
2013-07-18 10:59:14.258 3145 DEBUG nova.openstack.common.rpc.amqp [-] Making synchronous call on conductor ... multicall /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:583
2013-07-18 10:59:14.259 3145 DEBUG nova.openstack.common.rpc.amqp [-] MSG_ID is 8ef36d061a9341a09d3a5451df798673 multicall /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:586
2013-07-18 10:59:14.259 3145 DEBUG nova.openstack.common.rpc.amqp [-] UNIQUE_ID is 680b790574c64a9783fd2138c43f5f6d. _add_unique_id /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:337
2013-07-18 10:59:18.757 3145 WARNING nova.virt.vmwareapi.driver [-] Task [CreateVM_Task] (returnval){
   value = "task-33558"
   _type = "Task"
 } status: error The input arguments had entities that did not belong to the same datacenter.

2013-07-18 10:59:18.758 ERROR nova.compute.manager [req-e8306ffe-c6c7-4d0f-a466-fb532375cbd3 7799f10ca7da47f3b2660feb363b370b 0e1771f8db984a3599596fae62609d9a] [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] Instance failed to spawn
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] Traceback (most recent call last):
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1103, in _spawn
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] block_device_info)
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 177, in spawn
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] block_device_info)
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py", line 217, in spawn
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] _execute_create_vm()
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py", line 213, in _execute_create_vm
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] self._session._wait_for_task(instance['uuid'], vm_create_task)
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 554, in _wait_for_task
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] ret_val = done.wait()
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] return hubs.get_hub().switch()
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] return self.greenlet.switch()
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] NovaException: The input arguments had entities that did not belong to the same datacenter.
2013-07-18 10:59:18.758 3145 TRACE nova.compute.manager [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539]

2013-07-18 10:59:20.029 ERROR nova.compute.manager [req-e8306ffe-c6c7-4d0f-a466-fb532375cbd3 7799f10ca7da47f3b2660feb363b370b 0e1771f8db984a3599596fae62609d9a] [instance: 5b3961b6-38d9-409c-881e-fe50f67b1539] Error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 848, in _run_instance\n set_access_ip=set_access_ip)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1107, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1103, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 177, in spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py", line 217, in spawn\n _execute_create_vm()\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py", line 213, in _execute_create_vm\n self._session._wait_for_task(instance[\'uuid\'], vm_create_task)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 554, in _wait_for_task\n ret_val = done.wait()\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait\n return hubs.get_hub().switch()\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch\n return self.greenlet.switch()\n', 'NovaException: The input arguments had entities that did not belong to the same datacenter.\n']

2013-07-18 10:59:23.831 3145 WARNING nova.virt.vmwareapi.driver [-] Task [CreateVM_Task] (returnval){
   value = "task-33558"
   _type = "Task"
 } status: error The input arguments had entities that did not belong to the same datacenter.
2013-07-18 10:59:23.832 3145 WARNING nova.virt.vmwareapi.driver [-] In vmwareapi:_poll_task, Got this error Trying to re-send() an already-triggered event.
2013-07-18 10:59:23.833 3145 ERROR nova.utils [-] in fixed duration looping call
2013-07-18 10:59:23.833 3145 TRACE nova.utils Traceback (most recent call last):
2013-07-18 10:59:23.833 3145 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 594, in _inner
2013-07-18 10:59:23.833 3145 TRACE nova.utils self.f(*self.args, **self.kw)
2013-07-18 10:59:23.833 3145 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 580, in _poll_task
2013-07-18 10:59:23.833 3145 TRACE nova.utils done.send_exception(excep)
2013-07-18 10:59:23.833 3145 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 208, in send_exception
2013-07-18 10:59:23.833 3145 TRACE nova.utils return self.send(None, args)
2013-07-18 10:59:23.833 3145 TRACE nova.utils File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 150, in send
2013-07-18 10:59:23.833 3145 TRACE nova.utils assert self._result is NOT_USED, 'Trying to re-send() an already-triggered event.'
2013-07-18 10:59:23.833 3145 TRACE nova.utils AssertionError: Trying to re-send() an already-triggered event.
2013-07-18 10:59:23.833 3145 TRACE nova.utils

Revision history for this message
dan wendlandt (danwent) wrote :

Note: I have heard from Shawn that some of the changes in the "multiple clusters in a single nova-compute" patch may fix this, but if so, we should likely break that out into a separate patch that can be backported to grizzly, as that whole change-set will not be backported.

Revision history for this message
Shawn Hartsock (hartsock) wrote :

This is the same issue as: https://bugs.launchpad.net/nova/+bug/1180044

But, you've done a better job of describing the fault generally than the other bug did.

Changed in nova:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Shawn Hartsock (hartsock)
Shawn Hartsock (hartsock)
Changed in nova:
status: Triaged → In Progress
milestone: none → havana-3
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.