The tempest-integrated-compute-centos-9-stream job is broken since 2022-06-16 02:26:37 [1]. Multiple interface attach tempest test fails with:
libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': File descriptor named '(null)' has not been found
Full exception stack trace:
[None req-6f95599e-022a-42ab-a4de-07c7b8f73daf tempest-AttachInterfacesTestJSON-2035965943 tempest-AttachInterfacesTestJSON-2035965943-project] [instance: b41513e6-cb4e-4441-af14-c272393cdafb] attaching network adapter failed.: libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': File descriptor named '(null)' has not been found
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] Traceback (most recent call last):
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2850, in attach_interface
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] guest.attach_device(cfg, persistent=True, live=live)
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 321, in attach_device
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] self._domain.attachDeviceFlags(device_xml, flags=flags)
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 193, in doit
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] result = proxy_call(self._autowrap, f, *args, **kwargs)
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 151, in proxy_call
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] rv = execute(f, *args, **kwargs)
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] six.reraise(c, e, tb)
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/local/lib/python3.9/site-packages/six.py", line 719, in reraise
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] raise value
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] rv = meth(*args, **kwargs)
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/lib64/python3.9/site-packages/libvirt.py", line 706, in attachDeviceFlags
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] raise libvirtError('virDomainAttachDeviceFlags() failed')
Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': File descriptor named '(null)' has not been found
Example run: https://zuul.opendev.org/t/openstack/build/129de28a7a374013a03070f0742c46ac/log/controller/logs/screen-n-cpu.txt?severity=4
libvirt version: libvirt x86_64 8.4.0-1.el9
qemu version: qemu-kvm x86_64 17:7.0.0-4.el9
[1]https://zuul.opendev.org/t/openstack/builds?job_name=tempest-integrated-compute-centos-9-stream&skip=0
That issue is holding all our changes => Critical.