[System Tests] Sahara test fails with timeout on OSTF

Bug #1320795 reported by Andrey Sledzinskiy
This bug report is a duplicate of:  Bug #1321284: Sahara deployment fails. Edit Remove
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Confirmed
High
Fuel QA Team

Bug Description

http://jenkins-product.srt.mirantis.net:8080/view/0_0_swarm/job/master_fuelmain.system_test.centos.services/54/testReport/junit/%28root%29/deploy_savanna_simple/deploy_savanna_simple/

Test failed waiting savanna platform OSTF test finished

ERROR: Deploy cluster in simple mode with Savanna
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services/fuelweb_test/helpers/decorators.py", line 49, in wrapper
    return func(*args, **kwagrs)
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services/fuelweb_test/tests/test_services.py", line 118, in deploy_savanna_simple
    timeout=60 * 100)
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services/fuelweb_test/models/fuel_web_client.py", line 522, in run_single_ostf_test
    self.assert_ostf_run(cluster_id, should_fail=should_fail)
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services/fuelweb_test/models/fuel_web_client.py", line 135, in assert_ostf_run
    set_result_list = self._ostf_test_wait(cluster_id, timeout)
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services/fuelweb_test/models/fuel_web_client.py", line 75, in _ostf_test_wait
    timeout=timeout)
  File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 95, in wait
    raise TimeoutError("Waiting timed out")
TimeoutError: Waiting timed out

Tags: system-tests
Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
Revision history for this message
Dmitry Mescheryakov (dmitrymex) wrote :

Sahara run command
sudo su -c "hadoop dfsadmin -report

on master node of Hadoop cluster and didn't get result in 300 seconds. It does not look like a network issue, since Sahara successfully run many other commands on the node before that.

Is the environment deployed on VirtualBox? If yes, my guess is that the reason of failure is slow QEMU virtualization. This can be checked by re-running the tests. They might fail in another place, or even pass.

Revision history for this message
Mike Scherbakov (mihgen) wrote :

It failed again. Check is part of system tests, follow link in the description.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-main (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/94328

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-main (master)

Reviewed: https://review.openstack.org/94328
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=3b84ce8f14319c53d3042a11ddbe23e58e7b9459
Submitter: Jenkins
Branch: master

commit 3b84ce8f14319c53d3042a11ddbe23e58e7b9459
Author: Tatyana Leontovich <email address hidden>
Date: Tue May 20 12:09:30 2014 +0000

    Increase timeout for single run sahara ostf test

    Increase timeout for single run ostf shara test
    according to 10 minutes is too small value
    for this

    Change-Id: I023245f6985a4507e47584d6599fee2f973cf219
    Closes-Bug: #1321194
    Related-Bug: #1320795

Changed in fuel:
status: New → Fix Committed
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :

Move to confirmed according to patch do not fully close the problem, this tests hagng on waiting cluster status more that 1 hour and than fail by timeout. Cluster does not became active

Changed in fuel:
status: Fix Committed → Confirmed
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.