[System-tests]Replace 1 controller and re-deploy on ha env with neutron gre failed with "Timeout of deployment is exceeded"

[Artem Hrechanychenko]

Reported by CI https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.controller_replacement/60/console

raceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.controller_replacement/fuelweb_test/helpers/decorators.py", line 80, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.controller_replacement/fuelweb_test/tests/test_environment_action.py", line 341, in deploy_ha_neutron_tun_ctrl_replacement
    self.fuel_web.deploy_cluster_wait(cluster_id, check_services=False)
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.controller_replacement/fuelweb_test/helpers/decorators.py", line 459, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.controller_replacement/fuelweb_test/helpers/decorators.py", line 444, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.controller_replacement/fuelweb_test/helpers/decorators.py", line 495, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.controller_replacement/fuelweb_test/helpers/decorators.py", line 502, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.controller_replacement/fuelweb_test/helpers/decorators.py", line 386, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.controller_replacement/fuelweb_test/models/fuel_web_client.py", line 696, in deploy_cluster_wait
    self.assert_task_success(task, interval=interval, timeout=timeout)
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.controller_replacement/fuelweb_test/__init__.py", line 57, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.controller_replacement/fuelweb_test/models/fuel_web_client.py", line 312, in assert_task_success
    task = self.task_wait(task, timeout, interval)
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.controller_replacement/fuelweb_test/__init__.py", line 57, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.controller_replacement/fuelweb_test/models/fuel_web_client.py", line 1055, in task_wait
    "was exceeded: ".format(task=task["name"], timeout=timeout))
TimeoutError: Waiting task "deploy" timeout 7800 sec was exceeded:

no error in Astute.log, only [645] Timeout of deployment is exceeded.

So need increase deployment timer out of 7800 seconds

[Artem Hrechanychenko (agrechanichenko)]

ISO #245

[Artem Hrechanychenko (agrechanichenko)]

Referring to job #65 passed https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.controller_replacement/65/
and that no errors are present in astute and puppet-apply logs - seems to be some lab overloading

[Ksenia Svechnikova (kdemina)]

The same behavior was reproduced again on this test: https://product-ci.infra.mirantis.net/view/8.0_swarm/job/8.0.system_test.ubuntu.controller_replacement/125/console

[Alexander Kurenyshev (akurenyshev)]

The latest runs are passed (3) and took far less time than 7800 seconds.
This kind of errors is often related to some problems with product, not with tests.
Before opening bugs like this, please, take a look at a test rail statistic, and find out how much time test required in positive case.
Because if we increase timeouts we can miss real error.