Comment 0 for bug 1968732

Revision history for this message
chandan kumar (chkumar246) wrote :

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-containers-multinode-wallaby&skip=0
and similar version of RHEL job running on RHOS-17 RHEL-9 integration pipeline is failing with the same error at tempest
[1] and [2]
```
0} tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state [355.391581s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 131, in _get_ssh_connection
        ssh.connect(self.host, port=self.port, username=self.username,
      File "/usr/lib/python3.9/site-packages/paramiko/client.py", line 368, in connect
        raise NoValidConnectionsError(errors)
    paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 22 on 192.168.24.165

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/lib/python3.9/site-packages/tempest/common/utils/__init__.py", line 70, in wrapper
        return f(*func_args, **func_kwargs)
      File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 557, in test_update_router_admin_state
        self._check_public_network_connectivity(
      File "/usr/lib/python3.9/site-packages/tempest/scenario/test_network_basic_ops.py", line 212, in _check_public_network_connectivity
        self.check_vm_connectivity(
      File "/usr/lib/python3.9/site-packages/tempest/scenario/manager.py", line 952, in check_vm_connectivity
        self.get_remote_client(ip_address, username, private_key,
      File "/usr/lib/python3.9/site-packages/tempest/scenario/manager.py", line 723, in get_remote_client
        linux_client.validate_authentication()
      File "/usr/lib/python3.9/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 31, in wrapper
        return function(self, *args, **kwargs)
      File "/usr/lib/python3.9/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 117, in validate_authentication
        self.ssh_client.test_connection_auth()
      File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 240, in test_connection_auth
        connection = self._get_ssh_connection()
      File "/usr/lib/python3.9/site-packages/tempest/lib/common/ssh.py", line 150, in _get_ssh_connection
        raise exceptions.SSHTimeout(host=self.host,
    tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.165 via SSH timed out.
    User: cirros, Password: None

```

By comparing with passing job : https://6696badaf160a21814ac-f8803ca7587e43f6b66c9edae98e760b.ssl.cf1.rackcdn.com/837455/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/cd5e873/logs/subnode-1/var/log/extra/errors.txt
and failing job: https://057b70f24ff595a5f3b8-448e63f3241620894d174670888496e6.ssl.cf5.rackcdn.com/837512/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/a2518f7/logs/subnode-1/var/log/extra/errors.txt

```
2022-04-12 12:28:43.660 ERROR /var/log/containers/nova/nova-api.log: 17 ERROR oslo.messaging._drivers.impl_rabbit [-] [f125b2b9-d1f6-4535-b795-a3e41539b09e] AMQP server on centos-9-stream-rax-iad-0029292021.ctlplane.localdomain:5672 is unreachable: <RecoverableConnectionError: unknown error>. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: <RecoverableConnectionError: unknown error>
```

I am not sure what is causing the above error. Moving this test to skiplist till we investigate.

Logs:
[1]. https://057b70f24ff595a5f3b8-448e63f3241620894d174670888496e6.ssl.cf5.rackcdn.com/837512/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/a2518f7/logs/undercloud/var/log/tempest/tempest_run.log

[2]. https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f65/837516/1/check/tripleo-ci-centos-9-containers-multinode-wallaby/f65ef99/logs/undercloud/var/log/tempest/tempest_run.log