Instance failed to spawn: VirtualInterfaceCreateException: Virtual Interface creation failed

[Dilip Renkila]

Hi all,

I have a nova or neutron problem.

Sometimes instance creation fails due to follwing error. But sometime it works without a flaw.

2019-01-08 17:50:26.431 16532 ERROR nova.compute.manager [req-4efe4bf0-0d09-4c73-9c0a-e8e57b685575 1f7840b022f84efeaffbd9df5361923e a1dd449c9ce64345af2a7fb05c4aa21f - 66fcc66bd8f74a8898546ea82d3c6c3b 66fcc66bd8f74a8898546ea82d3c6c3b] [instance: 85b0522a-c95a-4b9d-bc53-382ca87288e3] Failed to allocate network(s): VirtualInterfaceCreateException: Virtual Interface creation failed
2019-01-08 17:50:26.431 16532 ERROR nova.compute.manager [instance: 85b0522a-c95a-4b9d-bc53-382ca87288e3] Traceback (most recent call last):
2019-01-08 17:50:26.431 16532 ERROR nova.compute.manager [instance: 85b0522a-c95a-4b9d-bc53-382ca87288e3] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2111, in _build_and_run_instance
2019-01-08 17:50:26.431 16532 ERROR nova.compute.manager [instance: 85b0522a-c95a-4b9d-bc53-382ca87288e3] block_device_info=block_device_info)
2019-01-08 17:50:26.431 16532 ERROR nova.compute.manager [instance: 85b0522a-c95a-4b9d-bc53-382ca87288e3] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3075, in spawn
2019-01-08 17:50:26.431 16532 ERROR nova.compute.manager [instance: 85b0522a-c95a-4b9d-bc53-382ca87288e3] destroy_disks_on_failure=True)
2019-01-08 17:50:26.431 16532 ERROR nova.compute.manager [instance: 85b0522a-c95a-4b9d-bc53-382ca87288e3] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5634, in _create_domain_and_network
2019-01-08 17:50:26.431 16532 ERROR nova.compute.manager [instance: 85b0522a-c95a-4b9d-bc53-382ca87288e3] raise exception.VirtualInterfaceCreateException()
2019-01-08 17:50:26.431 16532 ERROR nova.compute.manager [instance: 85b0522a-c95a-4b9d-bc53-382ca87288e3] VirtualInterfaceCreateException: Virtual Interface creation failed
2019-01-08 17:50:26.431 16532 ERROR nova.compute.manager [instance: 85b0522a-c95a-4b9d-bc53-382ca87288e3]
2019-01-08 17:50:26.444 16532 ERROR nova.compute.manager [req-4efe4bf0-0d09-4c73-9c0a-e8e57b685575 1f7840b022f84efeaffbd9df5361923e a1dd449c9ce64345af2a7fb05c4aa21f - 66fcc66bd8f74a8898546ea82d3c6c3b 66fcc66bd8f74a8898546ea82d3c6c3b] [instance: 85b0522a-c95a-4b9d-bc53-382ca87288e3] Build of instance 85b0522a-c95a-4b9d-bc53-382ca87288e3 aborted: Failed to allocate the network(s), not rescheduling.: BuildAbortException: Build of instance 85b0522a-c95a-4b9d-bc53-382ca87288e3 aborted: Failed to allocate the network(s), not rescheduling.

I am running on openstack rocky. I am seeing these type of errors after the addition of new public ip subnet into the provider network. is this error related to neutron or nova or both or rabbitmq ?

I have been following the official openstack doc. My conf file is

[DEFAULT]
allow_resize_to_same_host=True
log_dir = /var/log/nova
lock_path = /var/lock/nova
state_path = /var/lib/nova
transport_url =
my_ip = 10.20.1.5
use_neutron = true
firewall_driver = nova.virt.firewall.NoopFirewallDriver

compute_monitors = cpu.virt_driver,numa_mem_bw.virt_driver
notify_on_state_change= vm_and_task_state

[api]
auth_strategy = keystone
[api_database]
[barbican]
[cache]
[cells]
enable = False
[cinder]
[compute]
[conductor]
[console]
[consoleauth]
[cors]
[database]
#connection = sqlite:////var/lib/nova/nova.sqlite
[devices]
[ephemeral_storage_encryption]
[filter_scheduler]
[glance]
api_servers = http://10.20.0.199:9292
[guestfs]
[healthcheck]
[hyperv]
[ironic]
[key_manager]
[keystone]
[keystone_authtoken]
auth_url = http://10.20.0.199:5000/v3
memcached_servers = 10.20.1.1:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = nova
password =
[libvirt]
snapshot_image_format = qcow2
virt_type = qemu
inject_password = false
inject_partition = -2
live_migration_uri = qemu+tcp://%s/system
live_migration_flag="VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST,VIR_MIGRATE_TUNNELLED"
hw_disk_discard = unmap
images_type = rbd
images_rbd_pool = ephemeral-volumes
images_rbd_ceph_conf = /etc/ceph/ceph.conf
disk_cachemodes="network=writeback"
rbd_user = cinder
rbd_secret_uuid = ec2bfdd8-d67b-416d-aece-bafd73bf45d7
[matchmaker_redis]
[metrics]
[mks]
[neutron]
service_metadata_proxy = True
metadata_proxy_shared_secret = res2002
url = http://10.20.0.199:9696
auth_url = http://10.20.0.199:5000
auth_type = password
project_domain_name = default
user_domain_name = default
region_name = se-sto1
project_name = service
username = neutron
password = Laew8aeG
service_metadata_proxy = true
metadata_proxy_shared_secret = res2002
[notifications]
[osapi_v21]
[oslo_concurrency]
lock_path = /var/lib/nova/tmp
[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]

topics=notifications
driver=messagingv2
transport_url =

[oslo_messaging_rabbit]
[oslo_messaging_zmq]
[oslo_middleware]
[oslo_policy]
[pci]
[placement]
os_region_name = se-sto1
project_domain_name = Default
project_name = service
auth_type = password
user_domain_name = Default
auth_url =
username =
password =
[powervm]
[quota]
[rdp]
[remote_debug]
[scheduler]
[serial_console]
[service_user]
[spice]
[upgrade_levels]
[vault]
[vendordata_dynamic_auth]
[vmware]
[vnc]
enabled = True
server_listen = 0.0.0.0
server_proxyclient_address =
novncproxy_base_url =
[workarounds]
[wsgi]
[xenserver]
[xvp]

[Matt Riedemann (mriedem)]

It's not clear what's failing. It might be useful to log the exception stacktrace here:

https://github.com/openstack/nova/blob/stable/rocky/nova/virt/libvirt/driver.py#L5628

Or do you have more info in the logs?

[Dilip Renkila (dilip-renkila278)]

It's related to this bug https://bugs.launchpad.net/neutron/+bug/1760047

Ports don't become active because of some errors in DHCP provision ing