[system-tests] Increase timeout in deploy_stop_reset_on_ha test

[Andrey Sledzinskiy]

http://jenkins-product.srt.mirantis.net:8080/view/6.1_swarm/job/6.1.system_test.centos.cluster_actions/22/testReport/%28root%29/deploy_stop_reset_on_ha/deploy_stop_reset_on_ha/

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/6.1.system_test.centos.cluster_actions/fuelweb_test/helpers/decorators.py", line 64, in wrapper
    result = func(*args, **kwagrs)
  File "/home/jenkins/workspace/6.1.system_test.centos.cluster_actions/fuelweb_test/tests/test_environment_action.py", line 254, in deploy_stop_reset_on_ha
    self.fuel_web.wait_nodes_get_online_state(self.env.nodes().slaves[:3])
  File "/home/jenkins/workspace/6.1.system_test.centos.cluster_actions/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.centos.cluster_actions/fuelweb_test/models/fuel_web_client.py", line 1081, in wait_nodes_get_online_state
    timeout=60 * 4)
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 98, in wait
    raise TimeoutError("Waiting timed out")
TimeoutError: Waiting timed out

After revert nodes get to online state, seems that 4 minutes isn't enough

Logs are attached

[Tatyanka (tatyana-leontovich)]

the same happens on flat_deploy_stop_reset

[Andrey Sledzinskiy (asledzinskiy)]

Seems that it reproduced again

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/171497

[Artem Panchenko (apanchenko-8)]

IMHO, we can try to decrease slaves bootstrap time by modifying network interfaces configuration. Currently network scripts are waiting for 2 minutes while trying to get net config via DHCP on eth1-eth4 interfaces (30 seconds for each IF). If we decrease that value to 10 seconds, then we save 80 seconds on each node bootstrap for our tests.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/171986

[Alexander Kurenyshev (akurenyshev)]

Still appears:
======================================================================
FAIL: Stop reset cluster in ha mode
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/6.1.system_test.centos.cluster_actions/fuelweb_test/helpers/decorators.py", line 69, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.centos.cluster_actions/fuelweb_test/tests/test_environment_action.py", line 262, in deploy_stop_reset_on_ha
    self.env.d_env.nodes().slaves[:3])
  File "/home/jenkins/workspace/6.1.system_test.centos.cluster_actions/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.centos.cluster_actions/fuelweb_test/models/fuel_web_client.py", line 1344, in wait_nodes_get_online_state
    'Node {0} has not become online'.format(node.name))
AssertionError: Node slave-01 has not become online

Fuel used:
api: '1.0'
astute_sha: d96a80b63198a578b2c159edbd76048819039eb0
auth_required: true
build_id: 2015-04-13_22-54-31
build_number: '310'
feature_groups:
- mirantis
fuellib_sha: 8b80657e9ceed8d59c2dff1c11e1481c7e69380e
fuelmain_sha: 335d3ed09ed79bd37e1f7a90442c4831c8845582
nailgun_sha: d22c074dec091e5ddd8ea3003c37665058303cd5
openstack_version: 2014.2-6.1
ostf_sha: c2a76a60ec4ebbd78e508216c2e12787bf25e423
production: docker
python-fuelclient_sha: 9208ff4a08dcb674ce2df132399a5aa3ddfac21c
release: '6.1'
release_versions:
  2014.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: d96a80b63198a578b2c159edbd76048819039eb0
      build_id: 2015-04-13_22-54-31
      build_number: '310'
      feature_groups:
      - mirantis
      fuellib_sha: 8b80657e9ceed8d59c2dff1c11e1481c7e69380e
      fuelmain_sha: 335d3ed09ed79bd37e1f7a90442c4831c8845582
      nailgun_sha: d22c074dec091e5ddd8ea3003c37665058303cd5
      openstack_version: 2014.2-6.1
      ostf_sha: c2a76a60ec4ebbd78e508216c2e12787bf25e423
      production: docker
      python-fuelclient_sha: 9208ff4a08dcb674ce2df132399a5aa3ddfac21c
      release: '6.1'

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/171497
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=8aa30f73c02131c0d0d027d23f3a08e475323c71
Submitter: Jenkins
Branch: master

commit 8aa30f73c02131c0d0d027d23f3a08e475323c71
Author: asledzinskiy <email address hidden>
Date: Tue Apr 7 18:10:59 2015 +0300

    Increase wait in cluster actions tests

    - Increase wait time for nodes to get online state
    to 10 minutes in all tests from cluster actions group

    Change-Id: I917ebebe4b83117398374305810e248583a51a6c
    Closes-Bug: #1417009

[Tatyanka (tatyana-leontovich)]

test is passed on 357 iso

[OpenStack Infra (hudson-openstack)]

Change abandoned by Artem Panchenko (<email address hidden>) on branch: master
Review: https://review.openstack.org/171986