[10.0-community] Failed in method error_ceph_rados_gw

[Vyacheslav Vakhlyuev]

Detailed bug description:
From sys_test.log, summary:
"2016-07-26 08:46:56,886 - INFO decorators.py:314 -- <<<<<****************************************************************************************************>>>>>
2016-07-26 08:46:56,886 - INFO decorators.py:315 -- Failed in method error_ceph_rados_gw. Make snapshot: error_ceph_rados_gw
2016-07-26 08:46:56,886 - INFO decorators.py:325 -- You could revert and ssh to master node: [source /home/jenkins/venv-nailgun-tests-2.9/bin/activate; dos.py revert-resume 10.0-community.main.ubuntu.bvt_2 error_ceph_rados_gw && ssh root@10.109.31.2]
2016-07-26 08:46:56,886 - INFO decorators.py:327 -- <<<<<****************************************************************************************************>>>>>
2016-07-26 08:46:56,887 - ERROR decorators.py:123 -- Traceback (most recent call last):
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 117, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/tests/test_ceph.py", line 505, in ceph_rados_gw
    self.fuel_web.deploy_cluster_wait(cluster_id)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 459, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 444, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 495, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 502, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 386, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 931, in deploy_cluster_wait
    self.assert_task_success(task, interval=interval, timeout=timeout)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/__init__.py", line 60, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 335, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal
    raise ASSERTION_ERROR(message)
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[primary-rabbitmq/1] Stopping the deployment process!'

2016-07-26 08:46:56,887 - INFO decorators.py:124 -- <<<<<****************************************************************************************************>>>>>"

Also, more info from sys_test.log:
"2016-07-26 08:44:55,211 - DEBUG __init__.py:62 -- Done: get_task with result: {u'status': u'running', u'name': u'deploy', u'cluster': 1, u'parent_id': None, u'result': {}, u'progress': 68, u'message': None, u'id': 5, u'uuid': u'ffb623cd-ea99-468a-9fb2-da77fac2c54a'}
2016-07-26 08:45:25,241 - DEBUG __init__.py:56 -- Calling: get_task with args: ([<class 'fuelweb_test.models.nailgun_client.NailgunClient'>(0x7fc354c1ce10), url:None], 5) {}
2016-07-26 08:45:25,252 - DEBUG __init__.py:62 -- Done: get_task with result: {u'status': u'error', u'name': u'deploy', u'cluster': 1, u'parent_id': None, u'result': {}, u'progress': 100, u'message': u'Deployment has failed. All nodes are finished. Failed tasks: Task[primary-rabbitmq/1] Stopping the deployment process!', u'id': 5, u'uuid': u'ffb623cd-ea99-468a-9fb2-da77fac2c54a'}
2016-07-26 08:45:25,252 - DEBUG __init__.py:56 -- Calling: get_task with args: ([<class 'fuelweb_test.models.nailgun_client.NailgunClient'>(0x7fc354c1ce10), url:None], 5) {}
2016-07-26 08:45:25,263 - DEBUG __init__.py:62 -- Done: get_task with result: {u'status': u'error', u'name': u'deploy', u'cluster': 1, u'parent_id': None, u'result': {}, u'progress': 100, u'message': u'Deployment has failed. All nodes are finished. Failed tasks: Task[primary-rabbitmq/1] Stopping the deployment process!', u'id': 5, u'uuid': u'ffb623cd-ea99-468a-9fb2-da77fac2c54a'}
2016-07-26 08:45:25,263 - INFO fuel_web_client.py:1381 -- Task finished. Took 2042.57623601 seconds.
 status error
 name deploy
 cluster 1
 parent_id None
 result {}
 progress 100
 message Deployment has failed. All nodes are finished. Failed tasks: Task[primary-rabbitmq/1] Stopping the deployment process!
 id 5
 uuid ffb623cd-ea99-468a-9fb2-da77fac2c54a
2016-07-26 08:45:25,263 - DEBUG __init__.py:62 -- Done: task_wait with result: {u'status': u'error', u'name': u'deploy', u'cluster': 1, u'parent_id': None, u'result': {}, u'progress': 100, u'message': u'Deployment has failed. All nodes are finished. Failed tasks: Task[primary-rabbitmq/1] Stopping the deployment process!', u'id': 5, u'uuid': u'ffb623cd-ea99-468a-9fb2-da77fac2c54a'}
2016-07-26 08:45:25,264 - ERROR __init__.py:67 -- assert_task_success raised: AssertionError("Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[primary-rabbitmq/1] Stopping the deployment process!'",)
Traceback: Traceback (most recent call last):
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/__init__.py", line 60, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 335, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal
    raise ASSERTION_ERROR(message)
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[primary-rabbitmq/1] Stopping the deployment process!'

2016-07-26 08:45:25,264 - INFO regenerate_repo.py:304 -- Check puppet logs for packages with unmet dependencies.
2016-07-26 08:45:25,264 - DEBUG ssh_manager.py:138 -- SSH_MANAGER:Return existed connection for 10.109.31.2:22
2016-07-26 08:45:25,264 - DEBUG ssh_manager.py:139 -- SSH_MANAGER: Connections {('10.109.31.2', 22): SSHClient(host=10.109.31.2, port=22, auth=SSHAuth(username=root, password=<*masked*>, key=None, keys=[]))}
2016-07-26 08:45:25,264 - DEBUG ssh_client.py:740 -- Executing command: 'cd ~'
2016-07-26 08:45:25,280 - DEBUG ssh_client.py:719 -- cd ~ execution results: Exit code: 0
2016-07-26 08:45:25,281 - DEBUG ssh_client.py:740 -- Executing command: 'fgrep -h -e " Depends: " -e "The following packages have unmet dependencies:" -e "Unable to correct problems, you have held broken packages." /var/log/remote/node-*/puppet*.log'
2016-07-26 08:45:25,387 - DEBUG ssh_client.py:719 -- fgrep -h -e " Depends: " -e "The following packages have unmet dependencies:" -e "Unable to correct problems, you have held broken packages." /var/log/remote/node-*/puppet*.log execution results: Exit code: 1
2016-07-26 08:45:25,387 - INFO regenerate_repo.py:315 -- Check puppet logs completed.
2016-07-26 08:45:25,387 - DEBUG __init__.py:56 -- Calling: get_test_method_name with args: () {}
2016-07-26 08:45:25,392 - DEBUG __init__.py:62 -- Done: get_test_method_name with result: ceph_rados_gw
2016-07-26 08:45:25,392 - DEBUG __init__.py:56 -- Calling: update_yaml with args: (['ceph_rados_gw', 'deploy_cluster_wait'], '2043.44', False) {}
2016-07-26 08:45:25,395 - DEBUG __init__.py:62 -- Done: update_yaml with result: None"

Reproducibility:
The bug seems to be intermittent.

Jenkins jobs:
https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/408/