/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master is timing out running tempest - Unable to access /var/lib/neutron/external/pids/XX.pid.haproxy; Error: [Errno 2] No such file or directory: '/var/lib/neutron/external/pids/XX.pid.haproxy

[Ronelle Landy]

Tempest tests are timing out on periodic-tripleo-ci-centos-8-scenario010-standalone-network-master.

https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/e2912a9/job-output.txt:

2020-10-06 12:56:27.620669 | primary | TASK [os_tempest : Execute tempest tests] **************************************
2020-10-06 12:56:27.620700 | primary | Tuesday 06 October 2020 12:56:27 +0000 (0:00:00.047) 0:44:24.486 *******
2020-10-06 15:04:04.838353 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]
2020-10-06 15:04:04.838718 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-periodic-base/post.yaml@master]
2020-10-06 15:04:06.567320

logs/undercloud/var/log/containers/neutron/ovn-metadata-agent.log shows the following error:

2020-10-06 14:39:39.659 166716 DEBUG neutron.agent.linux.utils [-] Unable to access /var/lib/neutron/external/pids/8928af0a-6060-4223-949a-47411cc362c6.pid.haproxy; Error: [Errno 2] No such file or directory: '/var/lib/neutron/external/pids/8928af0a-6060-4223-949a-47411cc362c6.pid.haproxy' get_value_from_file /usr/lib/python3.6/site-packages/neutron/agent/linux/utils.py:264

2020-10-06 13:12:14.522 ERROR /var/log/containers/nova/nova-compute.log: 7 ERROR oslo_messaging.rpc.server neutronclient.common.exceptions.PortNotFoundClient: Port f8f214f9-9793-496e-9cda-3bb0cce990b1 could not be found." in:

2020-10-06 14:39:39.660 166716 DEBUG neutron.agent.ovn.metadata.driver [-] haproxy_cfg = https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/e2912a9/logs/undercloud/var/log/extra/errors.txt.txt.gz

https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/e2912a9/logs/undercloud/home/zuul/tempest/tempest.log.txt.gz shows:
""Provider \'octavia\' does not support a requested option: Amphora provider does not support SOURCE_IP_PORT algorithm.", "debuginfo": null}' _log_request_full /usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py:455"

[Ronelle Landy (rlandy)]

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-network-master

Timeout started on 10/05

[Ronelle Landy (rlandy)]

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-network-master ...

Run time of this test is steadily increasing since 2020-09-18, where is runs a whole hour longer: 2 hrs 44 mins 1 sec with with the sporadic 3hr8min timeout

[Ronelle Landy (rlandy)]

https://github.com/openstack/octavia-tempest-plugin/commit/89bdbcd1252dbce8645c24fa6db3e2d32fa779c3
- a lot of new tests were added. Increasing test timeout

See the diff:
https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/4a06c77/logs/undercloud/var/log/tempest/stestr_results.html.gz

https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/cc5b009/logs/undercloud/var/log/tempest/stestr_results.html.gz

[Ronelle Landy (rlandy)]

https://review.rdoproject.org/r/29951 Increase timeout for scenario010 test