periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/victoria/ussuri is failing on tempest test - octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest

[Sandeep Yadav]

Description:

periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master is failing on tempest test with Exception: Server 192.168.24.122 on port 60092/8085 did not begin passing traffic within the timeout period.

Logs:-

https://logserver.rdoproject.org/58/28458/29/check/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/f10ba43/logs/undercloud/var/log/tempest/stestr_results.html.gz

octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest

test_source_ip_port_tcp_traffic
~~~
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_traffic_ops.py", line 933, in test_source_ip_port_tcp_traffic
    persistent=False, source_port=60092)
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_traffic_ops.py", line 200, in _test_basic_traffic
    delay=delay)
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/validators.py", line 285, in check_members_balanced
    source_port=source_port)
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/validators.py", line 420, in _wait_for_lb_functional
    raise Exception(message)
Exception: Server 192.168.24.122 on port 60092 did not begin passing traffic within the timeout period. Failing test.
~~~

test_source_ip_port_udp_traffic
~~~
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_traffic_ops.py", line 959, in test_source_ip_port_udp_traffic
    persistent=False, source_port=8085)
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_traffic_ops.py", line 200, in _test_basic_traffic
    delay=delay)
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/validators.py", line 285, in check_members_balanced
    source_port=source_port)
  File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/validators.py", line 420, in _wait_for_lb_functional
    raise Exception(message)
Exception: Server 192.168.24.122 on port 8085 did not begin passing traffic within the timeout period. Failing test.
~~~

[Sandeep Yadav (sandeepyadav93)]

Ussuri also failing on same test:-
~~~
https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-ussuri/5ebc3b3/logs/undercloud/var/log/tempest/stestr_results.html.gz

~~~
octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest
~~~

Victoria is also failing on tempest:-

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-victoria/7478481/job-output.txt

[Sandeep Yadav (sandeepyadav93)]

As victoria job is not capturing logs properly, i loggined into scheduled job periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-victoria:-
~~~
[zuul@standalone tempest]$ sudo podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dc36857a3c94 192.168.24.1:8787/tripleovictoria/openstack-neutron-metadata-agent-ovn:a097d863229f8bf9f00d415feb94a31b-updated-20210210102436 /bin/bash -c HAPR... 10 minutes ago Up 10 minutes ago neutron-haproxy-ovnmeta-7343000f-4638-4fb5-bd4f-f2490e0682b4
~~~

Its failing with same error

[zuul@standalone tempest]$ cat tempest_run.log | grep -i error
    2021-02-10 11:26:46,614 197678 ERROR [octavia_tempest_plugin.tests.validators] Server 192.168.24.138 on port 60092 did not begin passing traffic within the timeout period. Failing test

[Sandeep Yadav (sandeepyadav93)]

Failing tests are in skiplist:-

https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/774419

[Slawek Kaplonski (slaweq)]

Isn't that duplicate of https://bugs.launchpad.net/tripleo/+bug/1914043 ?

[Gregory Thiemonge (gthiemonge)]

It looks like a new error, here the host is unable to connect to a testing server that is used behind the load balancer.

So the error happens even before an Octavia load balancer is created, so it might be an issue with the FIP of the testing server or even an issue while spawning the VM in nova.

[Sandeep Yadav (sandeepyadav93)]

Looking at https://bugs.launchpad.net/tripleo/+bug/1914043 its reported that is failing with ssh errors or timeout

I reported this new bug as failure reason is different:-

~~~
octavia_tempest_plugin.tests.validators] Server 192.168.24.138 on port 60092 did not begin passing traffic within the timeout period. Failing test
~~~

If the above failure ^^ is already taken care in https://bugs.launchpad.net/tripleo/+bug/1914043 , Please suggest us.

[Gregory Thiemonge (gthiemonge)]

Forget my previous comment (#6), the issue happens when we're checking that the LB is functional

[chandan kumar (chkumar246)]

https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-ussuri/d0a3852/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz

```
setUpClass (octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest)
```

We need to add periodic-tripleo-ci-centos-8-scenario010-standalone-ussuri job to skiplist as it missed from https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/774419

[Ronelle Landy (rlandy)]

This entry is still in the skiplist

[Pooja Jadhav (poojajadhav22)]

all the above mentioned jobs are passing now. we can close this out. also posted revert patch for removing skipped test jobs.

https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/814855