Timeout of master node provisioning is not enough

Bug #1530908 reported by Andrey Sledzinskiy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Fuel QA Team
8.0.x
Fix Released
Critical
Fuel QA Team

Bug Description

We have a lot of tests failed on master node provisioning when 10 minutes timeout wasn't enough

Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.ha_neutron_tun/fuelweb_test/helpers/decorators.py", line 82, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.ha_neutron_tun/fuelweb_test/tests/base_test_case.py", line 141, in setup_master
    self.env.setup_environment()
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.ha_neutron_tun/fuelweb_test/models/environment.py", line 440, in setup_environment
    self.wait_for_provisioning()
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.ha_neutron_tun/fuelweb_test/__init__.py", line 57, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.ha_neutron_tun/fuelweb_test/models/environment.py", line 516, in wait_for_provisioning
    (self.d_env.admin_net), 22), timeout=timeout)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 108, in _wait
    return raising_predicate()
  File "/home/jenkins/workspace/8.0.system_test.ubuntu.ha_neutron_tun/fuelweb_test/models/environment.py", line 516, in <lambda>
    (self.d_env.admin_net), 22), timeout=timeout)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 56, in _tcp_ping
    s.connect((str(host), int(port)))
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused

We may need to increase that value - https://github.com/openstack/fuel-qa/blob/master/fuelweb_test/settings.py#L77

Tags: area-qa
Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :

Raised to critical cause it affects too many tests

Changed in fuel:
importance: High → Critical
Changed in fuel:
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (master)

Fix proposed to branch: master
Review: https://review.openstack.org/263414

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/263416

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (stable/8.0)

Reviewed: https://review.openstack.org/263416
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=4db493e4e64cc048a78933aff01a2a3b16b74d3b
Submitter: Jenkins
Branch: stable/8.0

commit 4db493e4e64cc048a78933aff01a2a3b16b74d3b
Author: Tatyana Leontovich <email address hidden>
Date: Mon Jan 4 22:36:21 2016 +0200

    Increase provisioning timeout to 1200

    Change-Id: I62e01894a745df3f3ac6ae2b87dcb28b3687823a
    Closes-Bug: #1530908

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/263414
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=1ffc63bb09e25e5b13fa6d1bece98506195c3277
Submitter: Jenkins
Branch: master

commit 1ffc63bb09e25e5b13fa6d1bece98506195c3277
Author: Tatyana Leontovich <email address hidden>
Date: Mon Jan 4 22:36:21 2016 +0200

    Increase provisioning timeout to 1200

    We have 600s as default for now, so on slow environments
    we do not fit in it

    Change-Id: I62e01894a745df3f3ac6ae2b87dcb28b3687823a
    Closes-Bug: #1530908

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

verified on the latest runs

Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.