train: Instance creation failed: Virtual Interface creation failed

[Jason Hobbs]

When deploying k8s with juju on top of an fcb train, one of the instances failed to start, with this error in the nova-compute logs for it:

2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [req-029bff2d-743e-4a63-aa67-a56ca53992de 21ac5f0d12914cef90758c1246d87fe2 55ac0fc08bc34c179c165b1d88acd415 - 4b5f03b7cdf74b238a7d418687471c1e 4b5f03b7cdf74b238a7d418687471c1e] [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] Failed to allocate network(s): nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] Traceback (most recent call last):
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6223, in _create_domain_and_network
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] network_info)
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] next(self.gen)
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 478, in wait_for_instance_event
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] actual_event = event.wait()
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] File "/usr/lib/python3/dist-packages/eventlet/event.py", line 125, in wait
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] result = hub.switch()
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] File "/usr/lib/python3/dist-packages/eventlet/hubs/hub.py", line 298, in switch
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] return self.greenlet.switch()
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] eventlet.timeout.Timeout: 300 seconds
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e]
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] During handling of the above exception, another exception occurred:
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e]
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] Traceback (most recent call last):
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2322, in _build_and_run_instance
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] block_device_info=block_device_info)
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 3474, in spawn
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] power_on=power_on)
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 6244, in _create_domain_and_network
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] raise exception.VirtualInterfaceCreateException()
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e] nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed
2019-10-24 11:16:45.269 1738196 ERROR nova.compute.manager [instance: 54250ed5-ff27-4cf1-819a-11988194d79e]

[Jason Hobbs (jason-hobbs)]

Artifacts for the run are available here:

https://oil-jenkins.canonical.com/artifacts/0f2eaa09-20dc-49e6-8a4d-aea9701724c7/index.html

[Sahid Orentino (sahid-ferdjaoui)]

Such error is most of the time related to an issue with Neutron, perhaps you should look in the logs of the agents and server.

[Tolga Kaprol (tolgakaprol)]

I experienced a similar issue on the ussuri release. In my case, restarting neutron API services solved my problem.