I guess network_allocate_retries won't help since that's not in the driver spawn code path that's failing:
2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [req-2b591b2c-36f0-494b-a5b2-06dc2a783c49 tempest-MultipleCreateTestJSON-64962745 tempest-MultipleCreateTestJSON-1348265215] [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] Instance failed to spawn 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] Traceback (most recent call last): 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] File "/opt/stack/new/nova/nova/compute/manager.py", line 2193, in _build_resources 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] yield resources 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] File "/opt/stack/new/nova/nova/compute/manager.py", line 2039, in _build_and_run_instance 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] block_device_info=block_device_info) 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2767, in spawn 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] block_device_info=block_device_info) 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4906, in _create_domain_and_network 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] xml, pause=pause, power_on=power_on) 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4837, in _create_domain 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] guest.launch(pause=pause) 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 142, in launch 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] self._encoded_xml, errors='ignore') 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] self.force_reraise() 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] six.reraise(self.type_, self.value, self.tb) 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 137, in launch 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] return self._domain.createWithFlags(flags) 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] result = proxy_call(self._autowrap, f, *args, **kwargs) 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] rv = execute(f, *args, **kwargs) 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] six.reraise(c, e, tb) 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] rv = meth(*args, **kwargs) 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1059, in createWithFlags 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d] libvirtError: internal error: libxenlight failed to create new domain 'instance-0000003c' 2016-03-04 07:17:13.370 8173 ERROR nova.compute.manager [instance: eb3ec27f-53c0-4ce7-a6ad-58f9bb37b25d]
I guess network_ allocate_ retries won't help since that's not in the driver spawn code path that's failing:
2016-03-04 07:17:13.370 8173 ERROR nova.compute. manager [req-2b591b2c- 36f0-494b- a5b2-06dc2a783c 49 tempest- MultipleCreateT estJSON- 64962745 tempest- MultipleCreateT estJSON- 1348265215] [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] Instance failed to spawn manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] Traceback (most recent call last): manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 2193, in _build_resources manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] yield resources manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] File "/opt/stack/ new/nova/ nova/compute/ manager. py", line 2039, in _build_ and_run_ instance manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] block_device_ info=block_ device_ info) manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] File "/opt/stack/ new/nova/ nova/virt/ libvirt/ driver. py", line 2767, in spawn manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] block_device_ info=block_ device_ info) manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] File "/opt/stack/ new/nova/ nova/virt/ libvirt/ driver. py", line 4906, in _create_ domain_ and_network manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] xml, pause=pause, power_on=power_on) manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] File "/opt/stack/ new/nova/ nova/virt/ libvirt/ driver. py", line 4837, in _create_domain manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] guest.launch( pause=pause) manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] File "/opt/stack/ new/nova/ nova/virt/ libvirt/ guest.py" , line 142, in launch manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] self._encoded_xml, errors='ignore') manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] File "/usr/local/ lib/python2. 7/dist- packages/ oslo_utils/ excutils. py", line 220, in __exit__ manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] self.force_ reraise( ) manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] File "/usr/local/ lib/python2. 7/dist- packages/ oslo_utils/ excutils. py", line 196, in force_reraise manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] six.reraise( self.type_ , self.value, self.tb) manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] File "/opt/stack/ new/nova/ nova/virt/ libvirt/ guest.py" , line 137, in launch manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] return self._domain. createWithFlags (flags) manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 186, in doit manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] result = proxy_call( self._autowrap, f, *args, **kwargs) manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 144, in proxy_call manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] rv = execute(f, *args, **kwargs) manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 125, in execute manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] six.reraise(c, e, tb) manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] File "/usr/local/ lib/python2. 7/dist- packages/ eventlet/ tpool.py" , line 83, in tworker manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] rv = meth(*args, **kwargs) manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] File "/usr/local/ lib/python2. 7/dist- packages/ libvirt. py", line 1059, in createWithFlags manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] if ret == -1: raise libvirtError ('virDomainCrea teWithFlags( ) failed', dom=self) manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d] libvirtError: internal error: libxenlight failed to create new domain 'instance-0000003c' manager [instance: eb3ec27f- 53c0-4ce7- a6ad-58f9bb37b2 5d]
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.
2016-03-04 07:17:13.370 8173 ERROR nova.compute.