test_traffic_ops octavia tempest tests are failing with Connection to the 192.168.24.149 via SSH timed out.

[chandan kumar]

For testing octavia, we have three jobs to run octavia tempest tests:
* sc10
* sc10-kvm
* sc10-ovn

While running octavia scenario tests, this tests is continuously failing:
{0} setUpClass (octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest) [0.000000s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 112, in _get_ssh_connection
        sock=proxy_chan)
      File "/usr/lib/python3.6/site-packages/paramiko/client.py", line 349, in connect
        retry_on_signal(lambda: sock.connect(addr))
      File "/usr/lib/python3.6/site-packages/paramiko/util.py", line 283, in retry_on_signal
        return function()
      File "/usr/lib/python3.6/site-packages/paramiko/client.py", line 349, in <lambda>
        retry_on_signal(lambda: sock.connect(addr))
    socket.timeout: timed out

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/tempest/test.py", line 188, in setUpClass
        six.reraise(etype, value, trace)
      File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise
        raise value
      File "/usr/lib/python3.6/site-packages/tempest/test.py", line 181, in setUpClass
        cls.resource_setup()
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_traffic_ops.py", line 38, in resource_setup
        super(TrafficOperationsScenarioTest, cls).resource_setup()
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py", line 693, in resource_setup
        cls.webserver1_response)
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py", line 861, in _install_start_webserver
        linux_client.validate_authentication()
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 60, in wrapper
        six.reraise(*original_exception)
      File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise
        raise value
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 33, in wrapper
        return function(self, *args, **kwargs)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/utils/linux/remote_client.py", line 116, in validate_authentication
        self.ssh_client.test_connection_auth()
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 216, in test_connection_auth
        connection = self._get_ssh_connection()
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/ssh.py", line 128, in _get_ssh_connection
        password=self.password)
    tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.149 via SSH timed out.
    User: cirros, Password: None

https://logserver.rdoproject.org/12/27912/5/check/periodic-tripleo-ci-centos-8-scenario010-standalone-master/6029d95/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

It is one of the important test for testing octavia FIP tests. It is failing on train, ussuri and master.
Here is the tempest.conf
load_balancer]
member_role = "Member"
test_with_ipv6 = False
enable_security_groups = True
admin_role = admin
RBAC_test_type = owner_or_admin

In octavia workers logs, we found nothing. There might be some issue on network side.
It is a tracker bug, till we fix it.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/735854

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/735854
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=c5f857606cd0311511838c5b4cb8ffb270ec0aac
Submitter: Zuul
Branch: master

commit c5f857606cd0311511838c5b4cb8ffb270ec0aac
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Tue Jun 16 15:51:17 2020 +0530

    Move TrafficOperationsScenarioTest to skip list

    With octavia tempest tests, TrafficOperationsScenarioTest is failing
    continously with ssh timeout. We are moving it to skip list for
    train, ussuri and master release to enable octavia scenario
    tests.

    Related-Bug: #1883687

    Change-Id: Ibe0aea3d2a45c66f62a32a940c4f52562085ca14
    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>

[Sandeep Yadav (sandeepyadav93)]

periodic-tripleo-ci-centos-7-scenario010-standalone-train is also failing because of this:-

https://logserver.rdoproject.org/openstack-periodic-integration-stable3-centos7/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario010-standalone-train/1c1908f/logs/undercloud/var/log/tempest/stestr_results.html.gz

[Pooja Jadhav (poojajadhav22)]

It is also failing for wallaby too, "periodic-tripleo-ci-centos-8-scenario010-standalone-wallaby" is also consistently failing because of the above tempest test :

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-wallaby/fe93297/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

[Slawek Kaplonski (slaweq)]

Fix https://review.rdoproject.org/r/c/openstack/octavia-tempest-plugin-distgit/+/33352 is now merged and test patch https://review.rdoproject.org/r/c/testproject/+/33735 was running fine so this bug seems to be fixed now.