VirtualInterfaceCreateException due to "Timeout waiting for [('network-vif-plugged'..."

[melanie witt]

Seeing this occasionally in the gate recently, instance spawn fails with VirtualInterfaceCreateException after timing out 300 seconds waiting for the 'network-vif-plugged' event from neutron. The event never arrives (before or after we begin waiting).

Here's log excerpts showing how it looks [1]:

Sep 23 06:10:44.144477 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: DEBUG nova.compute.manager [None req-6d0b7dcf-7f96-47d2-8673-626abdd275b7 tempest-TestSnapshotPattern-20482801 tempest-TestSnapshotPattern-20482801-project] [instance: 487a3a37-fe41-447d-bc77-9ab3fce47473] Preparing to wait for external event network-vif-plugged-9129f5b9-2940-44c2-8d44-04c46df5fe49 {{(pid=110797) prepare_for_instance_event /opt/stack/nova/nova/compute/manager.py:280}}

[...]

Sep 23 06:15:44.961239 ubuntu-focal-rax-iad-0026631025 nova-compute[110797]: WARNING nova.virt.libvirt.driver [None req-6d0b7dcf-7f96-47d2-8673-626abdd275b7 tempest-TestSnapshotPattern-20482801 tempest-TestSnapshotPattern-20482801-project] [instance: 487a3a37-fe41-447d-bc77-9ab3fce47473] Timeout waiting for [('network-vif-plugged', '9129f5b9-2940-44c2-8d44-04c46df5fe49')] for instance with vm_state building and task_state spawning: eventlet.timeout.Timeout: 300 seconds

Logstash query:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Instance%20failed%20to%20spawn%3A%20nova.exception.VirtualInterfaceCreateException%3A%20Virtual%20Interface%20creation%20failed%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d

[1] https://paste.opendev.org/show/809546

[melanie witt (melwitt)]

Another hit on stable/ussuri in the nova-live-migration job:

https://zuul.opendev.org/t/openstack/build/a4bb5c12c35c4a4dad123913666ab231/log/controller/logs/screen-n-cpu.txt#10806

[Sylvain Bauza (sylvain-bauza)]

Looks to me legit.

[Michal Arbet (michalarbet)]

Hi, I've already filled LP bug here -> https://bugs.launchpad.net/nova/+bug/1951720 , it looks like it's same issue, isn't it ?

Have someone look deeper into this issue ?

Thanks,
Michal

[Michal Arbet (michalarbet)]

+ in https://bugs.launchpad.net/nova/+bug/1951623 vif-plugged-in also mentioned.

[Michal Arbet (michalarbet)]

+ Adding logs

[Bogdan Dobrelya (bogdando)]

FTR https://bugs.launchpad.net/nova/+bug/1951720/comments/7

[Michal Arbet (michalarbet)]

from Balazs Gibizer <email address hidden>

gibizer@riverbed:~/tmp/LOGS [000] [11:12:49]
$ egrep -e "instance: 09eff2ce-356f-430f-ab30-5de58f58d698.* Timeout waiting for|network-vif-plugged-0761ff2f-ff6a-40a5-85b7-c9b40386cbe0|network-vif-plugged-e64ddd6e-3128-44ee-afd6-07d13643a4be|network-vif-plugged-bf0f9dd2-4fd3-4028-a3dc-e2581ff081d0|network-vif-plugged-8a472e87-4a7a-4ad4-9fbb-fd9785136611|instance: 09eff2ce-356f-430f-ab30-5de58f58d698.* nova.exception.VirtualInterfaceCreateException" -R
compute0/logs/nova/nova-compute.log:2021-11-25 08:50:24.237 8 DEBUG nova.compute.manager [req-6eab0f2a-d7f4-4251-96f4-177fc0c62212 647a328aef9b4a8b890f102d8f018576 49231caa42f34da8a9d10229b5c7a5d8 - default default] [instance: 09eff2ce-356f-430f-ab30-5de58f58d698] Preparing to wait for external event network-vif-plugged-0761ff2f-ff6a-40a5-85b7-c9b40386cbe0 prepare_for_instance_event /var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py:280
compute0/logs/nova/nova-compute.log:2021-11-25 08:50:24.238 8 DEBUG nova.compute.manager [req-6eab0f2a-d7f4-4251-96f4-177fc0c62212 647a328aef9b4a8b890f102d8f018576 49231caa42f34da8a9d10229b5c7a5d8 - default default] [instance: 09eff2ce-356f-430f-ab30-5de58f58d698] Preparing to wait for external event network-vif-plugged-e64ddd6e-3128-44ee-afd6-07d13643a4be prepare_for_instance_event /var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py:280
compute0/logs/nova/nova-compute.log:2021-11-25 08:50:24.240 8 DEBUG nova.compute.manager [req-6eab0f2a-d7f4-4251-96f4-177fc0c62212 647a328aef9b4a8b890f102d8f018576 49231caa42f34da8a9d10229b5c7a5d8 - default default] [instance: 09eff2ce-356f-430f-ab30-5de58f58d698] Preparing to wait for external event network-vif-plugged-bf0f9dd2-4fd3-4028-a3dc-e2581ff081d0 prepare_for_instance_event /var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py:280
compute0/logs/nova/nova-compute.log:2021-11-25 08:50:24.241 8 DEBUG nova.compute.manager [req-6eab0f2a-d7f4-4251-96f4-177fc0c62212 647a328aef9b4a8b890f102d8f018576 49231caa42f34da8a9d10229b5c7a5d8 - default default] [instance: 09eff2ce-356f-430f-ab30-5de58f58d698] Preparing to wait for external event network-vif-plugged-8a472e87-4a7a-4ad4-9fbb-fd9785136611 prepare_for_instance_event /var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py:280
compute0/logs/nova/nova-compute.log:2021-11-25 08:51:00.817 8 DEBUG nova.compute.manager [req-310bc640-20d2-4ff8-8ab0-7a8ed6a49b32 01a75e3a9a9148218916d3beafae2120 7d16babef0524b7dade9a59b0a3569e1 - default default] [instance: 09eff2ce-356f-430f-ab30-5de58f58d698] Received event network-vif-plugged-0761ff2f-ff6a-40a5-85b7-c9b40386cbe0 external_instance_event /var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py:10439
compute0/logs/nova/nova-compute.log:2021-11-25 08:51:00.819 8 DEBUG nova.compute.manager [req-310bc640-20d2-4ff8-8ab0-7a8ed6a49b32 01a75e3a9a9148218916d3beafae2120 7d16babef0524b7dade9a59b0a3569e1 - default default] [instance: 09eff2ce-356f-430f-ab30-5de58f58d698...

[Felipe Alencastro (falencastro)]

Just had this same issue on charmed Xena, and managed to workaround it by following Tolga Kaprol's suggestion at https://discourse.charmhub.io/t/community-opinion-on-reliability-and-success-rate-of-ussuri-deployments/3510/3

[Alok Prudhvi (alokprudhvi)]

Hello, I'm seeing the same issue where nova error message tells - nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed and the instance creation keeps failing i was able to workaround this by Tolga Kaprol's comments but can i know's causing the exact issue, below are the versions i'm using.

OpenStack Release - Yoga
Nova Version - 25.0.0
Neutron Version - 20.1.0
Libvirt Version - libvirtd (libvirt) 8.0.0

Can someone pls look into and see what's the exact issue.

[Pedro Victor Lourenço Fragola (pedrovlf)]

I had the same issue using Yoga and workaround[0] solved the issue.

[0] https://discourse.charmhub.io/t/community-opinion-on-reliability-and-success-rate-of-ussuri-deployments/3510/3

[yehaifeng (yehaifeng)]

I have the same problem.

Any method to solve?

[Alok Prudhvi (alokprudhvi)]

See Tolga Kaprol's suggestion at https://discourse.charmhub.io/t/community-opinion-on-reliability-and-success-rate-of-ussuri-deployments/3510/3