OpenStack Compute (nova)

VMware: occasional failure under load - network not ready

Bug #1637970 reported by Gary Kotton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Triaged
Undecided
Unassigned

Bug Description

Executed the below testcase with 10 iterations and 7 concurrency

1. create 1 security_group
2. create 4 rules
3. create network
4. create subnet
5. create router (Distributed)
6. add router interface
6. set external gateway
5. nova boot servers
6. nova associate fip
7. verify ping from rally server
    to openstack instance

For one of the iteration, nova instance failed to spawn showing the below error.

2016-06-27 14:47:46.432 8124 ERROR nova.compute.manager [req-92a4fb34-f14f-42d0-9677-4e533d7c4c36 6bf77a5f2ea34d27948169813731737a dab0bef5c9fa40dbae556d3e6aaa13f1 - - -] [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Instance failed to spawn
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Traceback (most recent call last):
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2494, in _build_resources
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] yield resources
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2366, in _build_and_run_instance
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] block_device_info=block_device_info)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 497, in spawn
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] admin_password, network_info, block_device_info)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py", line 784, in spawn
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] metadata)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py", line 310, in build_virtual_machine
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] network_info)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vif.py", line 191, in get_vif_info
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] is_neutron, vif))
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vif.py", line 177, in get_vif_dict
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] ref = get_network_ref(session, cluster, vif, is_neutron)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vif.py", line 166, in get_network_ref
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] network_ref = get_neutron_network(session, network_name, cluster, vif)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vif.py", line 150, in get_neutron_network
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] session, network_name, cluster, full_match=False)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/network_util.py", line 103, in get_network_with_the_name
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] network_name, full_match)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/network_util.py", line 64, in _get_network_obj
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] "DistributedVirtualPortgroup", "config")
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 746, in _call_method
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] return self.invoke_api(module, method, self.vim, *args, **kwargs)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 343, in invoke_api
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] return _invoke_api(module, method, *args, **kwargs)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 122, in func
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] return evt.wait()
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] return hubs.get_hub().switch()
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] return self.greenlet.switch()
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 123, in _inner
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] idle = self.f(*self.args, **self.kw)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 95, in _func
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] result = f(*args, **kwargs)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 326, in _invoke_api
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] details=excep.details)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] ManagedObjectNotFoundException: The object has already been deleted or has not been completely created
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Cause: Server raised fault: 'The object has already been deleted or has not been completely created'
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Faults: [ManagedObjectNotFound]
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Details: {'obj': 'dvportgroup-4635'}
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4]
2016-06-27 14:47:46.437 8124 INFO nova.compute.manager [req-92a4fb34-f14f-42d0-9677-4e533d7c4c36 6bf77a5f2ea34d27948169813731737a dab0bef5c9fa40dbae556d3e6aaa13f1 - - -] [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Terminating instance

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/391778

Changed in nova:
assignee: nobody → Gary Kotton (garyk)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Balazs Gibizer (<email address hidden>) on branch: master
Review: https://review.opendev.org/391778
Reason: This is a pretty old patch in merge conflict. Feel free to restore it (or ask gibi on irc to restore it) if you still working on it.

Balazs Gibizer (balazs-gibizer)
Changed in nova:
status: In Progress → Triaged
assignee: Gary Kotton (garyk) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.