neutron_tempest_plugin.scenario.test_floatingip.FloatingIpMultipleRoutersTest.test_reuse_ip_address_with_other_fip_on_other_route failing with 'IP address already allocated'

[Douglas Viroel]

Test 'test_reuse_ip_address_with_other_fip_on_other_route' failed in the last runs for periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-ussuri job:
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-ussuri
stestr errors:
https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-ussuri/8fb38e9/logs/undercloud/var/log/tempest/stestr_results.html.gz
https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-ussuri/75667d0/logs/undercloud/var/log/tempest/stestr_results.html.gz
https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-ussuri/02871d5/logs/undercloud/var/log/tempest/stestr_results.html.gz

This issue is not happening in the standalone deployment:
https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-ussuri/09cc1c2/logs/undercloud/var/log/tempest/stestr_results.html.gz

Error traceback:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_floatingip.py", line 563, in test_reuse_ip_address_with_other_fip_on_other_router
    servers_num=1, fip_addresses=[ip_address])
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_floatingip.py", line 477, in _create_network_and_servers
    network=network, fip_address=fip))
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_floatingip.py", line 498, in _create_server_and_fip
    port=port)
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/api/base.py", line 634, in create_floatingip
    **kwargs)['floatingip']
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/services/network/json/network_client.py", line 972, in create_floatingip
    resp, body = self.post(uri, body)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 300, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 704, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 825, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'IpAddressAlreadyAllocated', 'message': 'IP address 10.0.0.178 already allocated in subnet d1aab874-5fbb-44b3-bfd3-68e762e89606', 'detail': ''}

[Douglas Viroel (dviroel)]

Also happening in periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-wallaby:
https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-wallaby/96402c4/logs/undercloud/var/log/tempest/stestr_results.html.gz

[Slawek Kaplonski (slaweq)]

I checked logs from one of the failed runs (https://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-ussuri/8fb38e9/logs/undercloud/var/log/tempest/stestr_results.html.gz) and I found that the issue is caused by race condition between tests.

Failed test is first creating FIP and attaching it to the server:

2021-08-19 16:17:24.642 26 INFO neutron.wsgi [req-e52c4b65-bb2d-4fdf-b3f9-0ab2aa57809c ace797f733be4e489a8a5fb3fb250d91 54d0a6efccf44556884cda17d59c79f2 - default default] 172.17.0.173 "POST /v2.0/floatingips HTTP/1.1" status: 201 len: 1029 time: 3.5029085

later the FIP is removed (but IP address is saved in the test to be used for new FIP):

2021-08-19 16:18:20.650 26 INFO neutron.wsgi [req-77b3851c-5661-4bc4-8859-c87f006f72f9 ace797f733be4e489a8a5fb3fb250d91 54d0a6efccf44556884cda17d59c79f2 - default default] 172.17.0.173 "DELETE /v2.0/floatingips/19ea7114-e444-4996-a22c-4f85cca6bd20 HTTP/1.1" status: 204 len: 168 time: 1.5269442

In the meantime, some other test creates router with external gateway, and the same IP is allocated as gateway IP for that other router (as this IP is now):

2021-08-19 16:18:28.621 26 DEBUG neutron.db.db_base_plugin_common [req-f6216bfc-8f01-45d9-bafd-4ecbf813534f ace797f733be4e489a8a5fb3fb250d91 54d0a6efccf44556884cda17d59c79f2 - default default] Allocated IP 10.0.0.151 (864fb817-03d6-42bd-983d-a9e20ae46c5d/920a15e8-9bf5-404d-b518-e0b6c6f12c7e/89a748e0-2306-4297-8955-b800922c0b1f) _store_ip_allocation /usr/lib/python3.6/site-packages/neutron/db/db_base_plugin_common.py:137
2021-08-19 16:18:31.194 26 INFO neutron.wsgi [req-f6216bfc-8f01-45d9-bafd-4ecbf813534f ace797f733be4e489a8a5fb3fb250d91 54d0a6efccf44556884cda17d59c79f2 - default default] 172.17.0.173 "POST /v2.0/routers HTTP/1.1" status: 201 len: 846 time: 3.8374729

After that, our broken test is trying to create new FIP with the IP which was used earlier and it fails:

2021-08-19 16:19:04.139 26 INFO neutron.api.v2.resource [req-08e0893b-745e-4df2-b33a-cd9d858eff9b 80362193b55e42c5976ffb94c64d90bc b6eabc92004a485091b791ee933c30a9 - default default] create failed (client error): There was a conflict when trying to complete your request.

[Slawek Kaplonski (slaweq)]

It seems to be the same bug as https://bugs.launchpad.net/neutron/+bug/1880976 which was already fixed by https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/731267 and that fix is in the neutron-tempest-plugin 1.2.0 and newer.
In the failed jobs version 1.1.0 was used.

Regarding failure in wallaby job (https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-wallaby/96402c4/logs/undercloud/var/log/tempest/stestr_results.html.gz) the issue was different as it couldn't SSH to the server.

[Slawek Kaplonski (slaweq)]

I proposed https://review.rdoproject.org/r/c/openstack/neutron-tempest-plugin-distgit/+/35080 to bump neutron-tempest-plugin used in RDO ussuri. In Victoria we already have version with fix mentioned in the previous patch so only ussuri is affected by that issue.