fullstack test neutron.tests.fullstack.test_l3_agent.TestHAL3Agent.test_gateway_ip_changed failed intermittently

[Hongbin Lu]

Neutron fullstack test 'neutron.tests.fullstack.test_l3_agent.TestHAL3Agent.test_gateway_ip_changed' intermittently failed with the following error

Traceback (most recent call last):
  File "/opt/stack/new/neutron/neutron/tests/base.py", line 151, in func
    return f(self, *args, **kwargs)
  File "/opt/stack/new/neutron/neutron/tests/fullstack/test_l3_agent.py", line 428, in test_gateway_ip_changed
    self._test_gateway_ip_changed()
  File "/opt/stack/new/neutron/neutron/tests/fullstack/test_l3_agent.py", line 116, in _test_gateway_ip_changed
    {'ip_address': ip_2}])
  File "/opt/stack/new/neutron/neutron/tests/fullstack/resources/client.py", line 141, in update_port
    return self._update_resource('port', port_id, kwargs)
  File "/opt/stack/new/neutron/neutron/tests/fullstack/resources/client.py", line 57, in _update_resource
    resp = update(id, body=body)
  File "/opt/stack/new/neutron/.tox/dsvm-fullstack/lib/python3.5/site-packages/neutronclient/v2_0/client.py", line 799, in update_port
    revision_number=revision_number)
  File "/opt/stack/new/neutron/.tox/dsvm-fullstack/lib/python3.5/site-packages/neutronclient/v2_0/client.py", line 2375, in _update_resource
    return self.put(path, **kwargs)
  File "/opt/stack/new/neutron/.tox/dsvm-fullstack/lib/python3.5/site-packages/neutronclient/v2_0/client.py", line 363, in put
    headers=headers, params=params)
  File "/opt/stack/new/neutron/.tox/dsvm-fullstack/lib/python3.5/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
    headers=headers, params=params)
  File "/opt/stack/new/neutron/.tox/dsvm-fullstack/lib/python3.5/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
    self._handle_fault_response(status_code, replybody, resp)
  File "/opt/stack/new/neutron/.tox/dsvm-fullstack/lib/python3.5/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
    exception_handler_v20(status_code, error_body)
  File "/opt/stack/new/neutron/.tox/dsvm-fullstack/lib/python3.5/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
    request_ids=request_ids)
neutronclient.common.exceptions.IpAddressAlreadyAllocatedClient: IP address 240.27.66.102 already allocated in subnet 814bfcc0-0d3d-4b45-8436-c76e7a000150
Neutron server returns request_ids: ['req-bf114fe1-66a9-4c86-8bc7-d759e10bf660']

Failure examples:

http://logs.openstack.org/36/617736/10/check/neutron-fullstack/61fa2eb/logs/testr_results.html.gz
http://logs.openstack.org/68/424468/34/check/neutron-fullstack/4422e70/logs/testr_results.html.gz
http://logs.openstack.org/08/620708/2/check/neutron-fullstack/51099db/logs/testr_results.html.gz
http://logs.openstack.org/09/608909/24/check/neutron-fullstack/2891c09/logs/testr_results.html.gz
http://logs.openstack.org/82/610182/13/check/neutron-fullstack/de0ad4f/logs/testr_results.html.gz

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/625359

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/625359
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ae4b331725012c7694132d17e28312159aedb1f3
Submitter: Zuul
Branch: master

commit ae4b331725012c7694132d17e28312159aedb1f3
Author: Hongbin Lu <email address hidden>
Date: Fri Dec 14 22:04:48 2018 +0000

    fullstack: retry on updating port's IP address

    If the update_port call failed with error IpAddressAlreadyAllocatedClient,
    retry a few more times in order to find IP addresses that are available.

    Change-Id: I7c5d51b01fa56083b1a689fa629a9a34c8b77012
    Closes-Bug: #1808595

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.0.0b2 development milestone.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

I found a problem related to this method [1] in [2]. Could be useful to, instead of retrieving the first IP available in the network, retrieve a random one.

[1] https://github.com/openstack/neutron/blob/a63f34474874ff1966cc42d3f437888d4c5813f8/neutron/tests/fullstack/base.py#L128
[2] https://5447eebf83bcd6fc5a63-dc6e15462e9da95e02ee0e042cb47d52.ssl.cf2.rackcdn.com/670177/10/check/neutron-fullstack/83775b4/testr_results.html.gz

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/684279

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/684279
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=94f69ab407d081ce49fa5a93946c4cd41bf6ee89
Submitter: Zuul
Branch: master

commit 94f69ab407d081ce49fa5a93946c4cd41bf6ee89
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Sep 24 11:00:03 2019 +0000

    Randomize BaseFullStackTestCase._find_available_ips

    Instead of retrieving an IP address of the CIDR network in
    ascending order, this patch randomizes the IP address selection.

    Change-Id: I5971d078bd68c3c0b104eea0d443511e741540c4
    Related-Bug: #1808595

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

The same problem is still affecting the related test cases. E.g.:
https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_942/660722/22/check/neutron-fullstack/9422481/testr_results.html.gz

Between the method to retrieve available IPs and the port update, sometimes those IP addresses are used.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

A bug was introduced in [1], as seen in [2]. Error log:

neutronclient.common.exceptions.InvalidIpForNetworkClient: IP address 240.10.143.255 is not a valid IP for any of the subnets on the specified network.

Broadcast address can't be used. The initial position defined in [3] should be reduced by one, to avoid getting the broadcast address in the previously created IPSet.

[1] https://review.opendev.org/#/c/684279/
[2] https://8e5339063fd8d98f3aba-875157a44fab70cabf10b03c4d83bfb0.ssl.cf2.rackcdn.com/660722/22/check/neutron-fullstack/6ea7ccc/testr_results.html.gz
[3] https://github.com/openstack/neutron/blob/0fa7e74ebb386b178d36ae684ff04f03bdd6cb0d/neutron/tests/fullstack/base.py#L140

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/693041

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rodolfo Alonso Hernandez (<email address hidden>) on branch: master
Review: https://review.opendev.org/693041
Reason: In favor of https://review.opendev.org/#/c/692135/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/705210

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/705211

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/705210
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9e8bdef93c90fa2b2fbd34fc5aebadf94df703d0
Submitter: Zuul
Branch: stable/train

commit 9e8bdef93c90fa2b2fbd34fc5aebadf94df703d0
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Sep 24 11:00:03 2019 +0000

    Randomize BaseFullStackTestCase._find_available_ips

    Instead of retrieving an IP address of the CIDR network in
    ascending order, this patch randomizes the IP address selection.

    Change-Id: I5971d078bd68c3c0b104eea0d443511e741540c4
    Related-Bug: #1808595
    (cherry picked from commit 94f69ab407d081ce49fa5a93946c4cd41bf6ee89)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/705211
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e1bcc6022866ec340f58a04d4dbb21595f18b9e6
Submitter: Zuul
Branch: stable/stein

commit e1bcc6022866ec340f58a04d4dbb21595f18b9e6
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Sep 24 11:00:03 2019 +0000

    Randomize BaseFullStackTestCase._find_available_ips

    Instead of retrieving an IP address of the CIDR network in
    ascending order, this patch randomizes the IP address selection.

    Change-Id: I5971d078bd68c3c0b104eea0d443511e741540c4
    Related-Bug: #1808595
    (cherry picked from commit 94f69ab407d081ce49fa5a93946c4cd41bf6ee89)