No more IP addresses available on network, neutron_tempest_plugin.scenario.test_security_groups.NetworkSecGroupTest

[wes hayutin]

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/neutron_tempest_plugin/scenario/test_security_groups.py", line 172, in test_two_sec_groups
    num_servers=1, security_groups=security_groups_list)
  File "/usr/lib/python2.7/site-packages/neutron_tempest_plugin/scenario/test_security_groups.py", line 71, in create_vm_testing_sec_grp
    fips.append(self.create_floatingip(port=port))
  File "/usr/lib/python2.7/site-packages/neutron_tempest_plugin/api/base.py", line 639, in create_floatingip
    **kwargs)['floatingip']
  File "/usr/lib/python2.7/site-packages/neutron_tempest_plugin/services/network/json/network_client.py", line 952, in create_floatingip
    resp, body = self.post(uri, body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 283, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 687, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 808, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {u'message': u'No more IP addresses available on network 4a3c35f1-d6af-457d-9bf6-9d3388aff734.', u'type': u'IpAddressGenerationFailure', u'detail': u''}

http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-full-tempest-train/0e489e7/logs/stestr_results.html

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694595
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=616b0f975bc01dd525bbe6f81fe37521e45fd3ed
Submitter: Zuul
Branch: master

commit 616b0f975bc01dd525bbe6f81fe37521e45fd3ed
Author: Wes Hayutin <email address hidden>
Date: Fri Nov 15 09:43:48 2019 -0700

    add test_security_groups.NetworkSecGroupTest to train skip

    This test is not consistently passing both a full
    deployment and standalone deployment. Let's
    make the test more reliable if possible.

    Related-Bug: #1852770
    Change-Id: I8acbfddaf75e2b66de34d83aa9904f03e7d45397

[Ronelle Landy (rlandy)]

This test is passing again - closing the related CIX card:

 https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-train is passing again

[Ronelle Landy (rlandy)]

Ignore comment above - added to wrong bug

[Sagi (Sergey) Shnaidman (sshnaidm)]

last time it passed in 012 job, can be reverted:

http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train/fcf4022/logs/stestr_results.html

[wes hayutin (weshayutin)]

back in ussuri
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_security_groups.py", line 216, in test_two_sec_groups
    num_servers=1, security_groups=security_groups_list)
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/test_security_groups.py", line 112, in create_vm_testing_sec_grp
    fips.append(self.create_floatingip(port=port))
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/api/base.py", line 634, in create_floatingip
    **kwargs)['floatingip']
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/services/network/json/network_client.py", line 961, in create_floatingip
    resp, body = self.post(uri, body)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 283, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 687, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 808, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'type': 'IpAddressGenerationFailure', 'message': 'No more IP addresses available on network acf5c774-9078-4143-a951-f7f3c88362a2.', 'detail': ''}

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/0ac6061/logs/undercloud/var/log/tempest/stestr_results.html.gz

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/8423f4c/logs/undercloud/var/log/tempest/stestr_results.html.gz

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/720050

[wes hayutin (weshayutin)]

https://review.opendev.org/720050

[Slawek Kaplonski (slaweq)]

I was checking this error from log https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master/0ac6061/logs/undercloud/var/log/tempest/stestr_results.html.gz

Network id was "acf5c774-9078-4143-a951-f7f3c88362a2". It seems it is external network, which is used to create floating IPs. So it's shared between various tests. And it has subnet with only 20 IP addresses:

2020-04-14 02:10:22.705 25 DEBUG neutron.api.v2.base [req-2386087d-45d1-4b27-8cbe-e1e6a957d7af 7e2cab0217074e689712fa223699ccfc 1989bc2138fd47338fbcb3676426d852 - default default] Request body: {'subnet': {'network_id': 'acf5c774-9078-4143-a951-f7f3c88362a2', 'ip_version': 4, 'enable_dhcp': True, 'cidr': '192.168.24.0/24', 'name': 'public-subnet', 'allocation_pools': [{'start': '192.168.24.100', 'end': '192.168.24.120'}], 'gateway_ip': '192.168.24.1'}} prepare_request_body /usr/lib/python3.6/site-packages/neutron/api/v2/base.py:719

So my guess is that it should just have bigger allocation pool set and this issue should be gone.

[Amol Kahat (amolkahat)]

I'm able to see this issue on centos8 jobs.

tempest.scenario.test_network_v6.TestGettingAddress.test_slaac_from_os[compute,id-2c92df61-29f0-4eaa-bee3-7c65bef62a43,network,slow]
------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
        return f(*func_args, **func_kwargs)
      File "/usr/lib/python3.6/site-packages/tempest/scenario/test_network_v6.py", line 244, in test_slaac_from_os
        self._prepare_and_test(address6_mode='slaac')
      File "/usr/lib/python3.6/site-packages/tempest/scenario/test_network_v6.py", line 193, in _prepare_and_test
        sshv4_1, ips_from_api_1, srv1 = self.prepare_server(networks=net_list)
      File "/usr/lib/python3.6/site-packages/tempest/scenario/test_network_v6.py", line 133, in prepare_server
        fip = self.create_floating_ip(thing=srv)
      File "/usr/lib/python3.6/site-packages/tempest/scenario/manager.py", line 1023, in create_floating_ip
        result = client.create_floatingip(**kwargs)
      File "/usr/lib/python3.6/site-packages/tempest/lib/services/network/floating_ips_client.py", line 30, in create_floatingip
        return self.create_resource(uri, post_data)
      File "/usr/lib/python3.6/site-packages/tempest/lib/services/network/base.py", line 61, in create_resource
        resp, body = self.post(req_uri, req_post_data)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 283, in post
        return self.request('POST', url, extra_headers, headers, body, chunked)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 687, in request
        self._error_checker(resp, resp_body)
      File "/usr/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 808, in _error_checker
        raise exceptions.Conflict(resp_body, resp=resp)
    tempest.lib.exceptions.Conflict: Conflict with state of target resource
    Details: {'type': 'IpAddressGenerationFailure', 'message': 'No more IP addresses available on network 21d38e25-5f7d-464e-8c2c-dac3d20b8230.', 'detail': ''}

Job link: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/722662

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/722662
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=d31e0f05d832fb2bd26dbce31eb435e8be965fda
Submitter: Zuul
Branch: master

commit d31e0f05d832fb2bd26dbce31eb435e8be965fda
Author: Arx Cruz <email address hidden>
Date: Fri Apr 24 12:51:30 2020 +0200

    Increase subnet allocation pool

    This might avoid errors with some tests complaing no more ip allocation
    available.

    Change-Id: Idb758f1eb2e74a1f6bbbb1d4243423a3395e3530
    Related-Bug: #1852770

[OpenStack Infra (hudson-openstack)]

Change abandoned by wes hayutin (<email address hidden>) on branch: master
Review: https://review.opendev.org/720050