System tests failed with error: StopIteration

[Timur Nurlygayanov]

Looks like the have some issues with system tests or environments:
http://jenkins-product.srt.mirantis.net:8080/view/6.1_swarm/job/6.1.system_test.centos.services_ha/53/console

+ python fuelweb_test/run_tests.py -q --nologcapture --with-xunit --group=services_ha
Create environment and set up master node ... ERROR
Prepare master node ... SKIP: Failure in <function setup_master at 0x7fb402540758>
Bootstrap 5 slave nodes ... SKIP: Failure in <function setup_master at 0x7fb402540758>
Deploy cluster in HA mode with Sahara and Neutron GRE ... SKIP: Failure in <function setup_master at 0x7fb402540758>
Deploy cluster in ha mode with Murano and Neutron GRE ... SKIP: Failure in <function setup_master at 0x7fb402540758>
Deploy cluster in ha multirole mode with Ceilometer ... SKIP: Failure in <function setup_master at 0x7fb402540758>
Deploy cluster in ha mode with Ceilometer ... SKIP: Failure in <function setup_master at 0x7fb402540758>
Deploy Heat cluster in HA mode ... SKIP: Failure in <function setup_master at 0x7fb402540758>

======================================================================
ERROR: Create environment and set up master node
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 333, in get_state
    self.instance = self.entry.home()
  File "/home/jenkins/workspace/6.1.system_test.centos.services_ha/fuelweb_test/tests/base_test_case.py", line 31, in __init__
    self.env = EnvironmentModel()
  File "/home/jenkins/workspace/6.1.system_test.centos.services_ha/fuelweb_test/models/environment.py", line 48, in __init__
    self.fuel_web = FuelWebClient(self.get_admin_node_ip(), self)
  File "/home/jenkins/workspace/6.1.system_test.centos.services_ha/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.centos.services_ha/fuelweb_test/models/environment.py", line 100, in get_admin_node_ip
    self.d_env.nodes(
  File "/home/jenkins/workspace/6.1.system_test.centos.services_ha/fuelweb_test/models/environment.py", line 187, in d_env
    self._virtual_environment = Environment.describe_environment()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/environment.py", line 196, in describe_environment
    networks.append(environment.create_networks(name))
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/environment.py", line 226, in create_networks
    has_dhcp_server=settings.DHCP.get(name))
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/network.py", line 168, in network_create
    pool=pool)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/django/db/transaction.py", line 431, in inner
    return func(*args, **kwargs)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/models/network.py", line 130, in _safe_create_network
    ip_network = allocated_pool.next()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/network.py", line 43, in next
    return self._generator.next()
StopIteration

----------------------------------------------------------------------
Ran 8 tests in 0.458s

FAILED (SKIP=7, errors=1)
+ ec=1

[Nastya Urlapova (aurlapova)]

@Timur, this is problem with server, cause that network's pool is ended.

[Nastya Urlapova (aurlapova)]

Related to https://bugs.launchpad.net/fuel/+bug/1415024

[Artem Panchenko (apanchenko-8)]

@Timur,

system tests failed because there was a lot of old environments (fuel-devops) on Jenkins node, which use all available IP networks (256 by default). So fuel-devops was unable to allocate IP network for new environment. Here is related issue bug #1415024

[Timur Nurlygayanov (tnurlygayanov)]

Nastya, Artem, thank you for update! Good to know that we are working on fix.

[Timur Nurlygayanov (tnurlygayanov)]

This bug was fixed and now we have other bug https://bugs.launchpad.net/mos/+bug/1442206