tripleo-ci-centos-7-standalone is failing on stable/train

[Michele Baldessari]

Seen this on https://review.opendev.org/#/c/705271/ and https://review.opendev.org/#/c/705606/. We fail the centos-7-standalone job:
https://zuul.opendev.org/t/openstack/build/bb057941873149d89108f0b0cbe33793

Seems tempest fails with a networking timeout. Via https://0a5c623ff913673f34ed-80527d3715cd24d9d5daf12827bd66d5.ssl.cf2.rackcdn.com/705606/1/check/tripleo-ci-centos-7-standalone/bb05794/logs/undercloud/var/log/tempest/tempest_run.log we see:
    2020-02-04 06:53:16,091 101417 WARNING [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.107 (Authentication failed.). Number attempts: 23. Retry after 24 seconds.
    2020-02-04 06:53:40,627 101417 INFO [paramiko.transport] Connected (version 2.0, client dropbear_2015.67)
    2020-02-04 06:53:40,937 101417 INFO [paramiko.transport] Authentication (publickey) failed.
    2020-02-04 06:53:41,065 101417 ERROR [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.107 after 23 attempts
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh Traceback (most recent call last):
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh File "/usr/lib/python2.7/site-packages/tempest/lib/common/ssh.py", line 107, in _get_ssh_connection
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh sock=proxy_chan)
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 437, in connect
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh passphrase,
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 749, in _auth
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh raise saved_exception
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh AuthenticationException: Authentication failed.
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh

[Michele Baldessari (michele)]

Interestingly some ssh attempts fail with authentication failed:
    2020-02-04 06:53:40,627 101417 INFO [paramiko.transport] Connected (version 2.0, client dropbear_2015.67)
    2020-02-04 06:53:40,937 101417 INFO [paramiko.transport] Authentication (publickey) failed.
    2020-02-04 06:53:41,065 101417 ERROR [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@192.168.24.107 after 23 attempts
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh Traceback (most recent call last):
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh File "/usr/lib/python2.7/site-packages/tempest/lib/common/ssh.py", line 107, in _get_ssh_connection
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh sock=proxy_chan)
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 437, in connect
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh passphrase,
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 749, in _auth
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh raise saved_exception
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh AuthenticationException: Authentication failed.
    2020-02-04 06:53:41.065 101417 ERROR tempest.lib.common.ssh

and others with ssh timeout:
    2020-02-04 06:53:42,797 101417 ERROR [tempest.scenario.manager] Public network connectivity check failed. Reason: after re-associate floating ip
    2020-02-04 06:53:42.797 101417 ERROR tempest.scenario.manager Traceback (most recent call last):
    2020-02-04 06:53:42.797 101417 ERROR tempest.scenario.manager File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 729, in check_vm_connectivity
    2020-02-04 06:53:42.797 101417 ERROR tempest.scenario.manager server=server)
    2020-02-04 06:53:42.797 101417 ERROR tempest.scenario.manager File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 501, in get_remote_client
    2020-02-04 06:53:42.797 101417 ERROR tempest.scenario.manager linux_client.validate_authentication()
    2020-02-04 06:53:42.797 101417 ERROR tempest.scenario.manager File "/usr/lib/python2.7/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 60, in wrapper
    2020-02-04 06:53:42.797 101417 ERROR tempest.scenario.manager six.reraise(*original_exception)
    2020-02-04 06:53:42.797 101417 ERROR tempest.scenario.manager File "/usr/lib/python2.7/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 33, in wrapper
    2020-02-04 06:53:42.797 101417 ERROR tempest.scenario.manager return function(self, *args, **kwargs)
    2020-02-04 06:53:42.797 101417 ERROR tempest.scenario.manager File "/usr/lib/python2.7/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 116, in validate_authentication
    2020-02-04 06:53:42.797 101417 ERROR tempest.scenario.manager self.ssh_client.test_connection_auth()
    2020-02-04 06:53:42.797 101417 ERROR tempest.scenario.manager File "/usr/lib/p...

[chandan kumar (chkumar246)]

We have seen similar issue here also https://bugs.launchpad.net/tripleo/+bug/1856016 and the permanent fix is to update tempest to 23.0.0 and we have work in progress patches
https://review.rdoproject.org/r/#/c/24785/ and https://review.rdoproject.org/r/#/c/24786/.

Will let the status know very soon

[wes hayutin (weshayutin)]

going to close this as a dupe of https://bugs.launchpad.net/tripleo/+bug/1856016

Overall the job seems stable enough, let's continue to monitor it after the tempest update.

http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-standalone&branch=stable%2Ftrain