Comment 0 for bug 1577839

Master systest run https://packaging-ci.infra.mirantis.net/job/master-pkg-systest-ubuntu/691/console failed with the following traceback:

FAIL: Deploy ceph HA with RadosGW for objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 120, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/tests/test_ceph.py", line 502, in ceph_rados_gw
    self.fuel_web.deploy_cluster_wait(cluster_id)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 455, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 440, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 491, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 498, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 382, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 827, in deploy_cluster_wait
    self.assert_task_success(task, interval=interval, timeout=timeout)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 321, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[primary-keystone/1] Stopping the deployment process!'

From the astute.log:

2016-04-29 02:09:45 DEBUG [30241] Node[1]: Node 1: task primary-keystone, task status running
2016-04-29 02:09:45 WARNING [30241] Puppet agent 1 didn't respond within the allotted time
2016-04-29 02:09:45 DEBUG [30241] Task time summary: primary-keystone with status failed on node 1 took 01:00:00
2016-04-29 02:09:45 DEBUG [30241] Node[1]: Decreasing node concurrency to: 0
2016-04-29 02:09:45 DEBUG [30241] Graph[1]: All tasks are finished
2016-04-29 02:09:45 DEBUG [30241] Cluster[]: Process node: Node[3]
2016-04-29 02:09:45 INFO [30241] Casting message to Nailgun:
{"method"=>"deploy_resp",
 "args"=>
  {"task_uuid"=>"47d2f738-0240-4f9e-8cde-a4c1fa7605cf",
   "nodes"=>
    [{"uid"=>"1",
      "status"=>"error",
      "progress"=>100,
      "deployment_graph_task_name"=>"primary-keystone",
      "task_status"=>"error",
      "custom"=>
       ...
           "failed_resources"=>
           "Haproxy_backend_status[keystone-admin],Haproxy_backend_status[keystone-public]",
          "failed"=>2,
       ...

node-1 haproxy.log:

2016-04-29T00:57:08.281784+00:00 alert: Server keystone-1/node-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.281825+00:00 alert: Server keystone-1/node-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.323573+00:00 alert: Server keystone-1/node-2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.323755+00:00 alert: Server keystone-1/node-2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.367596+00:00 alert: Server keystone-1/node-5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.367596+00:00 emerg: proxy keystone-1 has no server available!
2016-04-29T00:57:08.367909+00:00 alert: Server keystone-1/node-5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.367909+00:00 emerg: proxy keystone-1 has no server available!
2016-04-29T00:57:08.407086+00:00 alert: Server keystone-2/node-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.407354+00:00 alert: Server keystone-2/node-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.451365+00:00 alert: Server keystone-2/node-2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 4ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.451466+00:00 alert: Server keystone-2/node-2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 4ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.492305+00:00 alert: Server keystone-2/node-5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.492305+00:00 emerg: proxy keystone-2 has no server available!
2016-04-29T00:57:08.492429+00:00 alert: Server keystone-2/node-5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.492429+00:00 emerg: proxy keystone-2 has no server available!

node-1 puppet-apply.log:

2016-04-29T01:31:30.352294+00:00 err: Timeout waiting for HAProxy backend: 'keystone-2' status to become: 'up' after 1200 seconds!