Tempest unable to reach cirros instances in Octavia tests.

Bug #1926768 reported by Amol Kahat
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Triaged
Critical
Unassigned

Bug Description

Description
===========

Tempest smoke tests are failing.
- octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest

Actual Error
============
- Log: https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-victoria/f2b2939/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

{1} setUpClass (octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest) [0.000000s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/tempest/test.py", line 182, in setUpClass
        six.reraise(etype, value, trace)
      File "/usr/local/lib/python3.6/site-packages/six.py", line 703, in reraise
        raise value
      File "/usr/lib/python3.6/site-packages/tempest/test.py", line 175, in setUpClass
        cls.resource_setup()
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/scenario/v2/test_traffic_ops.py", line 53, in resource_setup
        super(TrafficOperationsScenarioTest, cls).resource_setup()
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py", line 699, in resource_setup
        cls.webserver1_response)
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/test_base.py", line 925, in _validate_webserver
        cls.validate_URL_response(URL, expected_body=str(start_id))
      File "/usr/lib/python3.6/site-packages/octavia_tempest_plugin/tests/validators.py", line 121, in validate_URL_response
        raise exceptions.TimeoutException()
    tempest.lib.exceptions.TimeoutException: Request timed out
    Details: None

{0} octavia_tempest_plugin.tests.scenario.v2.test_pool.PoolScenarioTest.test_HTTPS_LC_pool_standalone_CRUD [19.011320s] ... ok
{0} octavia_tempest_plugin.tests.scenario.v2.test_pool.PoolScenarioTest.test_HTTPS_LC_pool_with_listener_CRUD [102.464005s] ... ok
{0} octavia_tempest_plugin.tests.scenario.v2.test_pool.PoolScenarioTest.test_HTTPS_RR_pool_standalone_CRUD [59.135684s] ... ok
{0} octavia_tempest_plugin.tests.scenario.v2.test_pool.PoolScenarioTest.test_HTTPS_RR_pool_with_listener_CRUD [73.924172s] ... ok
{0} octavia_tempest_plugin.tests.scenario.v2.test_pool.PoolScenarioTest.test_HTTPS_SIP_pool_standalone_CRUD ... SKIPPED: Provider 'octavia' does not support a requested option: Amphora provider does not support SOURCE_IP_PORT algorithm.
{0} octavia_tempest_plugin.tests.scenario.v2.test_pool.PoolScenarioTest.test_HTTPS_SIP_pool_with_listener_CRUD ... SKIPPED: Provider 'octavia' does not support a requested option: Amphora provider does not support SOURCE_IP_PORT algorithm.

- Log: https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-victoria/f2b2939/logs/undercloud/var/log/extra/errors.txt.txt.gz

2021-04-30 10:44:40.884 ERROR /var/log/containers/octavia/octavia.log: 25 ERROR wsme.api [req-b523232d-3939-42e1-9671-0f7f8233bdbe - c4f39b6cde4846f185c6528c4cbebaee - default default] Server-side error: "Provider 'octavia' does not support a requested option: Amphora provider does not support SOURCE_IP_PORT algorithm.". Detail:
2021-04-30 10:45:02.116 ERROR /var/log/containers/octavia/octavia.log: 24 ERROR wsme.api [req-012a688a-8456-4408-82e2-3926578184c7 - c4f39b6cde4846f185c6528c4cbebaee - default default] Server-side error: "Provider 'octavia' does not support a requested option: Amphora provider does not support SOURCE_IP_PORT algorithm.". Detail:
2021-04-30 10:57:16.126 ERROR /var/log/containers/octavia/octavia.log: 25 ERROR wsme.api [req-36d1e460-3d63-4d75-94ce-a39cf5e0da09 - c4f39b6cde4846f185c6528c4cbebaee - default default] Server-side error: "Provider 'octavia' does not support a requested option: Amphora provider does not support SOURCE_IP_PORT algorithm.". Detail:
2021-04-30 10:57:38.368 ERROR /var/log/containers/octavia/octavia.log: 24 ERROR wsme.api [req-d02ba538-1d44-43b7-a6b8-fefd157f5933 - c4f39b6cde4846f185c6528c4cbebaee - default default] Server-side error: "Provider 'octavia' does not support a requested option: Amphora provider does not support SOURCE_IP_PORT algorithm.". Detail:
2021-04-30 11:05:33.273 ERROR /var/log/containers/octavia/octavia.log: 24 ERROR wsme.api [req-dc9ba8d2-dd89-4f25-a51c-7c86564e9798 - c4f39b6cde4846f185c6528c4cbebaee - default default] Server-side error: "Provider 'octavia' does not support a requested option: Amphora provider does not support SOURCE_IP_PORT algorithm.". Detail:
2021-04-30 11:05:55.286 ERROR /var/log/containers/octavia/octavia.log: 25 ERROR wsme.api [req-461ac6ef-546b-47d6-9815-4597105cf758 - c4f39b6cde4846f185c6528c4cbebaee - default default] Server-side error: "Provider 'octavia' does not support a requested option: Amphora provider does not support SOURCE_IP_PORT algorithm.". Detail:
2021-04-30 11:15:28.779 ERROR /var/log/containers/octavia/octavia.log: 25 ERROR wsme.api [req-4c84bb05-d343-4375-b6ff-d52a197749ab - c4f39b6cde4846f185c6528c4cbebaee - default default] Server-side error: "Provider 'octavia' does not support a requested option: Amphora provider does not support SOURCE_IP_PORT algorithm.". Detail:
2021-04-30 11:15:56.255 ERROR /var/log/containers/octavia/octavia.log: 24 ERROR wsme.api [req-0b1bb0e8-e45a-4473-9217-b69f5d0075f4 - c4f39b6cde4846f185c6528c4cbebaee - default default] Server-side error: "Provider 'octavia' does not support a requested option: Amphora provider does not support SOURCE_IP_PORT algorithm.". Detail:
2021-04-30 11:33:03.179 ERROR /var/log/containers/octavia/octavia.log: 23 ERROR wsme.api [req-0f982b04-d22d-4953-8528-f2af881e99d3 - c4f39b6cde4846f185c6528c4cbebaee - default default] Server-side error: "Provider 'octavia' does not support a requested option: Amphora provider does not support SOURCE_IP_PORT algorithm.". Detail:
2021-04-30 11:34:19.231 ERROR /var/log/containers/octavia/octavia.log: 24 ERROR wsme.api [req-3e3395a9-ed41-4e6b-88a6-60a154c24c25 - c4f39b6cde4846f185c6528c4cbebaee - default default] Server-side error: "Provider 'octavia' does not support a requested option: Amphora provider does not support SOURCE_IP_PORT algorithm.". Detail:
2021-04-30 10:40:18.905 ERROR /var/log/containers/octavia/worker.log: 41 ERROR octavia.amphorae.drivers.haproxy.exceptions [-] Amphora agent returned unexpected result code 409 with response {'message': 'Interface already exists'}
2021-04-30 10:23:36.490 ERROR /var/log/containers/nova/nova-compute.log: 7 ERROR nova.cmd.common [req-3795e332-815d-41d1-88b4-140ecf555967 - - - - -] No db access allowed in nova-compute: File "/usr/bin/nova-compute", line 10, in <module>

Logs:
- https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-victoria/347f143/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

Revision history for this message
Carlos Goncalves (cgoncalves) wrote :

The report title is very misleading, unrelated to the actual failure which is happening on a different test.

The actual failed test reports that tempest timed out connecting to a cirros VM (http server, TCP:80). Since other RDO octavia CI jobs are also seeing these type of issues (tracked under some CIX card), I would say this is likely the same issue under investigation (nested KVM, networking issues).

Revision history for this message
Michael Johnson (johnsom) wrote :

The title to this in incorrect. That is a normal message and those tests are skipped for test runs with the amphora driver (note, SOURCE_IP_PORT is the only algorithm the OVN provider supports and it the only provider that implements it).

I am going to edit the title to make it reflect the actual issue which appears to be a non-Octavia related networking issue.

summary: - Amphora driver does not support SOURCE_IP_PORT algorithm
+ Tempest unable to reach cirros instances in Octavia tests.
Revision history for this message
wes hayutin (weshayutin) wrote :
Revision history for this message
Carlos Goncalves (cgoncalves) wrote :

@Wes, yes, that one.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.