periodic-tripleo-ci-centos-8-standalone-network-wallaby failing consistently network tempest tests

[Pooja Jadhav]

periodic-tripleo-ci-centos-8-standalone-network-wallaby failing consistently on network tempest tests with below traceback :

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 136, in _get_ssh_connection
    sock=proxy_chan)
  File "/usr/lib/python3.6/site-packages/paramiko/client.py", line 368, in connect
    raise NoValidConnectionsError(errors)
paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 192.168.24.199

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/common/utils/__init__.py", line 70, in wrapper
    return f(*func_args, **func_kwargs)
  File "/usr/lib/python3.6/site-packages/tempest/api/compute/servers/test_attach_interfaces.py", line 368, in test_add_remove_fixed_ip
    server, ifs, fip = self._create_server_get_interfaces()
  File "/usr/lib/python3.6/site-packages/tempest/api/compute/servers/test_attach_interfaces.py", line 88, in _create_server_get_interfaces
    self._wait_for_validation(server, validation_resources)
  File "/usr/lib/python3.6/site-packages/tempest/api/compute/servers/test_attach_interfaces.py", line 73, in _wait_for_validation
    linux_client.validate_authentication()
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 31, in wrapper
    return function(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 117, in validate_authentication
    self.ssh_client.test_connection_auth()
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 240, in test_connection_auth
    connection = self._get_ssh_connection()
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 152, in _get_ssh_connection
    password=self.password)
tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.199 via SSH timed out.
User: cirros, Password: password

Detailed Logs :

https://logserver.rdoproject.org/74/36274/32/check/periodic-tripleo-ci-centos-8-standalone-network-wallaby/d7593dc/logs/undercloud/var/log/tempest/stestr_results.html.gz

https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-network-wallaby/bebc93c/logs/undercloud/var/log/tempest/stestr_results.html.gz

https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-network-wallaby/bebc93c/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz

[John Fulton (jfulton-org)]

Saw a similar symptom [1] in tripleo-ci-centos-9-standalone on this patch.

[1] "paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 192.168.24.193"

[2] https://review.opendev.org/q/I137e335abeedccad801cdc03feee654c3e42a0e2

[yatin (yatinkarel)]

All ovn jobs in network component should be impacted, ovn metadata agent is not starting[1], can be closed as duplicate of [2]. pyroute2 update will clear this issue too.

https://review.opendev.org/q/I137e335abeedccad801cdc03feee654c3e42a0e2 doesn't have same issue so can be tracked separately.

[1] https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-network-wallaby/79f8d53/logs/undercloud/var/log/extra/podman/containers/ovn_metadata_agent/stdout.log.txt.gz
[2] https://bugs.launchpad.net/tripleo/+bug/1979646

[Ronelle Landy (rlandy)]

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-network-wallaby&skip=0 is passing now - closing this out