SC10 train periodic job failing on LoadBalancerScenarioTest consistently

[Pooja Jadhav]

periodic-tripleo-ci-centos-7-scenario010-standalone-train is failing with the below error logs :

2020-06-01 05:53:08.194423 | primary | TASK [os_tempest : Ensure private network exists] ******************************
2020-06-01 05:53:08.194501 | primary | Monday 01 June 2020 05:53:08 +0000 (0:00:00.100) 0:39:46.330 ***********
2020-06-01 05:53:11.920072 | primary | FAILED - RETRYING: Ensure private network exists (5 retries left).
2020-06-01 05:53:24.624402 | primary | FAILED - RETRYING: Ensure private network exists (4 retries left).
2020-06-01 05:53:37.223961 | primary | FAILED - RETRYING: Ensure private network exists (3 retries left).
2020-06-01 05:53:49.816758 | primary | FAILED - RETRYING: Ensure private network exists (2 retries left).
2020-06-01 05:54:02.321538 | primary | FAILED - RETRYING: Ensure private network exists (1 retries left).
2020-06-01 05:54:14.724326 | primary | fatal: [undercloud -> 127.0.0.2]: FAILED! => {
2020-06-01 05:54:14.724473 | primary | "attempts": 5,
2020-06-01 05:54:14.724539 | primary | "changed": false
2020-06-01 05:54:14.724566 | primary | }
2020-06-01 05:54:14.724608 | primary |
2020-06-01 05:54:14.724635 | primary | MSG:
2020-06-01 05:54:14.724711 | primary |
2020-06-01 05:54:14.724871 | primary | ConflictException: 409: Client Error for url: http://192.168.24.1:9696/v2.0/networks.json, Unable to create the network. The tunnel ID 1 is in use.

Reference job links :

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-scenario010-standalone-train

https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario010-standalone-train/7b60b68/job-output.txt

https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario010-standalone-train/9a97601/job-output.txt

[Arx Cruz (arxcruz)]

This failure is not exactly related to Tempest. The os_tempest is trying to create a network, however, it's failing because the tunnel ID 1 is in use. This seems to be a network problem, the tempest tests don't even start. In fact, tempest isn't even running.
What I wonder is if is this happening only on scenario010 or in other scenarios as well

[Rafael Folco (rafaelfolco)]

I haven't seen any recent changes in this code:

https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/type_tunnel.py#L295-L311

https://opendev.org/openstack/openstack-ansible-os_tempest/src/branch/master/tasks/tempest_resources.yml#L146-L161

https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config//featureset062.yml#L37

scen10 was using fs052, and we switched to fs062 https://review.rdoproject.org/r/#/c/27901

and testing if this helps here https://review.rdoproject.org/r/#/c/27902/

[Rafael Folco (rafaelfolco)]

changing description as the original issue reported was due to wrong featureset (052). After switching to fs062, getting consistent failures on tempest test LoadBalancerScenarioTest...

https://logserver.rdoproject.org/02/27902/2/check/periodic-tripleo-ci-centos-7-scenario010-standalone-train/39f1f84/logs/undercloud/var/log/tempest/stestr_results.html.gz

octavia_tempest_plugin.tests.scenario.v2.test_load_balancer.LoadBalancerScenarioTest

and there is a proposal to add more tests for scen10 here: https://review.opendev.org/#/c/733038/

[Rafael Folco (rafaelfolco)]

tempest folks testing octavia plugin here:
https://review.rdoproject.org/r/#/c/27912/

[Pooja Jadhav (poojajadhav22)]

Hi all,

Fix is : https://review.rdoproject.org/r/#/c/28069/

[Emilien Macchi (emilienm)]

Confirmed with Chandan that is works:
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-scenario010-standalone-train

[Marios Andreou (marios-b)]

make them voting again with that https://review.opendev.org/#/c/735891/2