Interface attach fails with libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': File descriptor named '(null)' has not been found

[Balazs Gibizer]

The tempest-integrated-compute-centos-9-stream job is broken since 2022-06-16 02:26:37 [1]. Multiple interface attach tempest test fails with:

libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': File descriptor named '(null)' has not been found

Full exception stack trace:

[None req-6f95599e-022a-42ab-a4de-07c7b8f73daf tempest-AttachInterfacesTestJSON-2035965943 tempest-AttachInterfacesTestJSON-2035965943-project] [instance: b41513e6-cb4e-4441-af14-c272393cdafb] attaching network adapter failed.: libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': File descriptor named '(null)' has not been found

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] Traceback (most recent call last):

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2850, in attach_interface

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] guest.attach_device(cfg, persistent=True, live=live)

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 321, in attach_device

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] self._domain.attachDeviceFlags(device_xml, flags=flags)

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 193, in doit

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] result = proxy_call(self._autowrap, f, *args, **kwargs)

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 151, in proxy_call

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] rv = execute(f, *args, **kwargs)

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] six.reraise(c, e, tb)

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/local/lib/python3.9/site-packages/six.py", line 719, in reraise

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] raise value

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/local/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] rv = meth(*args, **kwargs)

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] File "/usr/lib64/python3.9/site-packages/libvirt.py", line 706, in attachDeviceFlags

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] raise libvirtError('virDomainAttachDeviceFlags() failed')

Jun 17 08:25:43.545335 centos-9-stream-ovh-bhs1-0030065283 nova-compute[70195]: ERROR nova.virt.libvirt.driver [instance: b41513e6-cb4e-4441-af14-c272393cdafb] libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': File descriptor named '(null)' has not been found

Example run: https://zuul.opendev.org/t/openstack/build/129de28a7a374013a03070f0742c46ac/log/controller/logs/screen-n-cpu.txt?severity=4

libvirt version: libvirt x86_64 8.4.0-1.el9
qemu version: qemu-kvm x86_64 17:7.0.0-4.el9

[1]https://zuul.opendev.org/t/openstack/builds?job_name=tempest-integrated-compute-centos-9-stream&skip=0

[Sylvain Bauza (sylvain-bauza)]

That issue is holding all our changes => Critical.

[Sylvain Bauza (sylvain-bauza)]

Patch https://review.opendev.org/c/openstack/nova/+/846292 is proposed for putting this job non-voting until we solve the problem.

[Takashi Kajinami (kajinamit)]

I guess the error is caused by the following libvirt bug in CentOS/RHEL 9.

 https://bugzilla.redhat.com/show_bug.cgi?id=2092856

We recently hit the above bug in TripleO and pinned libvirt.
 https://bugs.launchpad.net/tripleo/+bug/1978969

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/846292
Committed: https://opendev.org/openstack/nova/commit/45c0a3884b6b4c0b9a6b1b3cef47651f5504502e
Submitter: "Zuul (22348)"
Branch: master

commit 45c0a3884b6b4c0b9a6b1b3cef47651f5504502e
Author: Sylvain Bauza <email address hidden>
Date: Fri Jun 17 11:59:55 2022 +0200

    zuul: Temporarly put Centos9 Stream job non-voting

    Due to the bug below, all the nova patches are failing. I accordingly
    propose to make the job non-voting as it would allow our gate to ressurect.

    Of course, this is a transient solution and we need a proper fix for the
    job in order to have Centos 9 Stream support back in place ASAP.

    When the job is back OK, please make it voting again by reverting this patch.

    Change-Id: I5c3fff65fd6d9e4f3632d1ec62ae3f1f9cfbe626
    Related-Bug: #1979047

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/846905

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/846905
Committed: https://opendev.org/openstack/neutron/commit/a1384678b88fc7ffb5eeccfbfc0da16a9cb8910a
Submitter: "Zuul (22348)"
Branch: master

commit a1384678b88fc7ffb5eeccfbfc0da16a9cb8910a
Author: yatinkarel <email address hidden>
Date: Tue Jun 21 11:17:03 2022 +0530

    [fips jobs] blacklist AttachInterfacesTestJSON tempest test

    libvirt-8.4.0-1.el9 caused a regression which resulted into
    AttachInterfacesTestJSON and test_hotplug_nic tempest test to
    fail, until fixed libvirt version is available in CentOS 9-Stream
    repos, let's exclude these tests from jobs running
    on CentOS 9-Stream.

    Related-Bug: #1979047
    Related-Bug: rhbz#2092856
    Change-Id: Ia97fc084dcbdfbced0c2cc79f127a97b4d97d1e3

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/848351

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/848351
Committed: https://opendev.org/openstack/neutron/commit/f879d1f9578739bb3c5c362def665d9ca354b3f5
Submitter: "Zuul (22348)"
Branch: master

commit f879d1f9578739bb3c5c362def665d9ca354b3f5
Author: yatin <email address hidden>
Date: Fri Jul 1 05:39:14 2022 +0000

    Revert "[fips jobs] blacklist AttachInterfacesTestJSON tempest test"

    This reverts commit a1384678b88fc7ffb5eeccfbfc0da16a9cb8910a.

    Reason for revert: https://bugzilla.redhat.com/show_bug.cgi?id=2092856 fixed in libvirt-8.4.0-3.el9 and available in CentOS 9-Stream.

    Change-Id: I2f33543fa7ee65ce3d25f70af49a18b69f799aec
    Related-Bug: #1979047
    Related-Bug: rhbz#2092856

[Balazs Gibizer (balazs-gibizer)]

The libvirt fix was released and the job is green in nova. We moved the job to the periodic queue though.