tripleo-ci-centos-9-standalone fails tempest test with error - 'Failed to attach network adapter device'

[Soniya Murlidhar Vyas]

tripleo-ci-centos-9-standalone fails tempest test with error -Failed to attach network adapter device

Following is the test failing on consecutive 3 runs:-
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_hotplug_nic

Below is the traceback observed:-

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tempest/common/utils/__init__.py", line 70, in wrapper
    return f(*func_args, **func_kwargs)
  File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 537, in test_hotplug_nic
    self._hotplug_server()
  File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 255, in _hotplug_server
    interface = self.interface_client.create_interface(
  File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/interfaces_client.py", line 47, in create_interface
    resp, body = self.post('servers/%s/os-interface' % server_id,
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 299, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python3.9/site-packages/tempest/lib/services/compute/base_compute_client.py", line 47, in request
    resp, resp_body = super(BaseComputeClient, self).request(
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 703, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.9/site-packages/tempest/lib/common/rest_client.py", line 879, in _error_checker
    raise exceptions.ServerFault(resp_body, resp=resp,
tempest.lib.exceptions.ServerFault: Got server fault
Details: Failed to attach network adapter device to 17d5f776-b73f-4aa1-a1c9-39f9d60c5364

For more reference please refer below details:-
1. https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d21/820530/8/check/tripleo-ci-centos-9-standalone/d211165/logs/undercloud/var/log/tempest/stestr_results.html
2. https://714d3b1696d424275e85-cf6cecac4105946f700316db1618e133.ssl.cf5.rackcdn.com/846059/2/check/tripleo-ci-centos-9-standalone/64a8df5/logs/undercloud/var/log/tempest/stestr_results.html
3. https://ad5cca834a49a8119416-e1a0fd98d652767a6b80e489a96366df.ssl.cf2.rackcdn.com/845333/2/check/tripleo-ci-centos-9-standalone/2e18eb8/logs/undercloud/var/log/tempest/stestr_results.html

[Ronelle Landy (rlandy)]

Referencing: https://bugzilla.redhat.com/show_bug.cgi?id=2092856

[Ronelle Landy (rlandy)]

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-standalone&skip=0 - failing since 06/16

[Ronelle Landy (rlandy)]

gthiemonge> rlandy|rover: scenario010 is also affected by this issue
<gthiemonge> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d2a/820530/8/check/tripleo-ci-centos-9-scenario010-standalone/d2a1c9c/logs/undercloud/var/log/containers/nova/nova-api.log

[Ronelle Landy (rlandy)]

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-9-push-master/70a4843/logs/container-builds/3d1b9524-1b00-443b-992b-df0536bda1c2/base/os/nova-base/nova-libvirt/nova-libvirt-build.log

still shows:

 libvirt-client x86_64 8.3.0-1.el9 quickstart-centos-appstreams 413 k
 libvirt-daemon-driver-nodedev x86_64 8.3.0-1.el9 quickstart-centos-appstreams 185 k
 libvirt-daemon-driver-nwfilter x86_64 8.3.0-1.el9 quickstart-centos-appstreams 201 k
 libvirt-daemon-driver-qemu x86_64 8.3.0-1.el9 quickstart-centos-appstreams 894 k
 libvirt-daemon-driver-secret x86_64 8.3.0-1.el9 quickstart-centos-appstreams 162 k
 libvirt-daemon-driver-storage-core x86_64 8.3.0-1.el9 quickstart-centos-appstreams 221 k
 libvirt-libs x86_64 8.3.0-1.el9 quickstart-centos-appstreams 4.5 M

[Ronelle Landy (rlandy)]

http://mirror.mtl01.iweb.opendev.org/centos-stream/9-stream/AppStream/x86_64/os/Packages/ now has:

[ ] libvirt-8.4.0-1.el9.x86_64.rpm 2022-06-07 17:47 11K

[Ronelle Landy (rlandy)]

Workaround patch: https://review.opendev.org/c/openstack/tripleo-quickstart/+/846194/ - Temp downgrade of libvirt 8.4.x - failing tempest

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/846194
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/98d0085b5bacc10cebcc670738d8fee6bdb82210
Submitter: "Zuul (22348)"
Branch: master

commit 98d0085b5bacc10cebcc670738d8fee6bdb82210
Author: Ronelle Landy <email address hidden>
Date: Thu Jun 16 11:42:06 2022 -0400

    Temp downgrade of libvirt 8.4.x - failing tempest

    With libvirt-8.4.0-1 and latest related packages from
    centos-stream/9-stream/AppStream, the test_hotplug_nic
    tempest test and others are failing.

    Related BZ: https://bugzilla.redhat.com/show_bug.cgi?id=2092856

    This review temp removes these libvirt related
    packages to unblock testing while the BZ fix propogates.

    Related-Bug: #1978969
    Change-Id: Icfdea8c5d2f3b3abac6e1bc50c634c644531637c

[Alan Pevec (apevec)]

fixed in libvirt-9.0.0-4.el9