FAIL: Deploy ceph HA with RadosGW for objects
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 120, in wrapper
result = func(*args, **kwargs)
File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/tests/test_ceph.py", line 502, in ceph_rados_gw
self.fuel_web.deploy_cluster_wait(cluster_id)
File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 455, in wrapper
result = func(*args, **kwargs)
File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 440, in wrapper
result = func(*args, **kwargs)
File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 491, in wrapper
return func(*args, **kwargs)
File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 498, in wrapper
result = func(*args, **kwargs)
File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 382, in wrapper
return func(*args, **kwargs)
File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 827, in deploy_cluster_wait
self.assert_task_success(task, interval=interval, timeout=timeout)
File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped
result = func(*args, **kwargs)
File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 321, in assert_task_success
task["name"], task['status'], 'ready', _message(task)
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[primary-keystone/1] Stopping the deployment process!'
From the astute.log:
2016-04-29 02:09:45 DEBUG [30241] Node[1]: Node 1: task primary-keystone, task status running
2016-04-29 02:09:45 WARNING [30241] Puppet agent 1 didn't respond within the allotted time
2016-04-29 02:09:45 DEBUG [30241] Task time summary: primary-keystone with status failed on node 1 took 01:00:00
2016-04-29 02:09:45 DEBUG [30241] Node[1]: Decreasing node concurrency to: 0
2016-04-29 02:09:45 DEBUG [30241] Graph[1]: All tasks are finished
2016-04-29 02:09:45 DEBUG [30241] Cluster[]: Process node: Node[3]
2016-04-29 02:09:45 INFO [30241] Casting message to Nailgun:
{"method"=>"deploy_resp",
"args"=>
{"task_uuid"=>"47d2f738-0240-4f9e-8cde-a4c1fa7605cf",
"nodes"=>
[{"uid"=>"1",
"status"=>"error",
"progress"=>100,
"deployment_graph_task_name"=>"primary-keystone",
"task_status"=>"error",
"custom"=>
... "failed_resources"=> "Haproxy_backend_status[keystone-admin],Haproxy_backend_status[keystone-public]", "failed"=>2,
...
node-1 haproxy.log:
2016-04-29T00:57:08.281784+00:00 alert: Server keystone-1/node-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.281825+00:00 alert: Server keystone-1/node-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.323573+00:00 alert: Server keystone-1/node-2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.323755+00:00 alert: Server keystone-1/node-2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.367596+00:00 alert: Server keystone-1/node-5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.367596+00:00 emerg: proxy keystone-1 has no server available!
2016-04-29T00:57:08.367909+00:00 alert: Server keystone-1/node-5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.367909+00:00 emerg: proxy keystone-1 has no server available!
2016-04-29T00:57:08.407086+00:00 alert: Server keystone-2/node-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.407354+00:00 alert: Server keystone-2/node-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.451365+00:00 alert: Server keystone-2/node-2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 4ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.451466+00:00 alert: Server keystone-2/node-2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 4ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.492305+00:00 alert: Server keystone-2/node-5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.492305+00:00 emerg: proxy keystone-2 has no server available!
2016-04-29T00:57:08.492429+00:00 alert: Server keystone-2/node-5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2016-04-29T00:57:08.492429+00:00 emerg: proxy keystone-2 has no server available!
node-1 puppet-apply.log:
2016-04-29T01:31:30.352294+00:00 err: Timeout waiting for HAProxy backend: 'keystone-2' status to become: 'up' after 1200 seconds!
Master systest run https:/ /packaging- ci.infra. mirantis. net/job/ master- pkg-systest- ubuntu/ 691/console failed with the following traceback:
FAIL: Deploy ceph HA with RadosGW for objects ------- ------- ------- ------- ------- ------- ------- ------- ------- jenkins/ venv-nailgun- tests-2. 9/local/ lib/python2. 7/site- packages/ proboscis/ case.py" , line 296, in testng_ method_ mistake_ capture_ func ty.capture_ type_error( s_func) jenkins/ venv-nailgun- tests-2. 9/local/ lib/python2. 7/site- packages/ proboscis/ compatability/ exceptions_ 2_6.py" , line 27, in capture_type_error jenkins/ venv-nailgun- tests-2. 9/local/ lib/python2. 7/site- packages/ proboscis/ case.py" , line 350, in func test_case. state.get_ state() ) jenkins/ workspace/ master- pkg-systest- ubuntu/ fuel-qa/ fuelweb_ test/helpers/ decorators. py", line 120, in wrapper jenkins/ workspace/ master- pkg-systest- ubuntu/ fuel-qa/ fuelweb_ test/tests/ test_ceph. py", line 502, in ceph_rados_gw fuel_web. deploy_ cluster_ wait(cluster_ id) jenkins/ workspace/ master- pkg-systest- ubuntu/ fuel-qa/ fuelweb_ test/helpers/ decorators. py", line 455, in wrapper jenkins/ workspace/ master- pkg-systest- ubuntu/ fuel-qa/ fuelweb_ test/helpers/ decorators. py", line 440, in wrapper jenkins/ workspace/ master- pkg-systest- ubuntu/ fuel-qa/ fuelweb_ test/helpers/ decorators. py", line 491, in wrapper jenkins/ workspace/ master- pkg-systest- ubuntu/ fuel-qa/ fuelweb_ test/helpers/ decorators. py", line 498, in wrapper jenkins/ workspace/ master- pkg-systest- ubuntu/ fuel-qa/ fuelweb_ test/helpers/ decorators. py", line 382, in wrapper jenkins/ workspace/ master- pkg-systest- ubuntu/ fuel-qa/ fuelweb_ test/models/ fuel_web_ client. py", line 827, in deploy_cluster_wait assert_ task_success( task, interval=interval, timeout=timeout) jenkins/ workspace/ master- pkg-systest- ubuntu/ fuel-qa/ fuelweb_ test/__ init__. py", line 59, in wrapped jenkins/ workspace/ master- pkg-systest- ubuntu/ fuel-qa/ fuelweb_ test/models/ fuel_web_ client. py", line 321, in assert_task_success keystone/ 1] Stopping the deployment process!'
-------
Traceback (most recent call last):
File "/home/
compatabili
File "/home/
func()
File "/home/
func(
File "/home/
result = func(*args, **kwargs)
File "/home/
self.
File "/home/
result = func(*args, **kwargs)
File "/home/
result = func(*args, **kwargs)
File "/home/
return func(*args, **kwargs)
File "/home/
result = func(*args, **kwargs)
File "/home/
return func(*args, **kwargs)
File "/home/
self.
File "/home/
result = func(*args, **kwargs)
File "/home/
task["name"], task['status'], 'ready', _message(task)
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[primary-
From the astute.log:
2016-04-29 02:09:45 DEBUG [30241] Node[1]: Node 1: task primary-keystone, task status running =>"deploy_ resp", uuid"=> "47d2f738- 0240-4f9e- 8cde-a4c1fa7605 cf", status" =>"error" , progress" =>100, deployment_ graph_task_ name"=> "primary- keystone" , task_status" =>"error" ,
"failed_ resources" =>
"Haproxy_ backend_ status[ keystone- admin], Haproxy_ backend_ status[ keystone- public] ",
"failed" =>2,
2016-04-29 02:09:45 WARNING [30241] Puppet agent 1 didn't respond within the allotted time
2016-04-29 02:09:45 DEBUG [30241] Task time summary: primary-keystone with status failed on node 1 took 01:00:00
2016-04-29 02:09:45 DEBUG [30241] Node[1]: Decreasing node concurrency to: 0
2016-04-29 02:09:45 DEBUG [30241] Graph[1]: All tasks are finished
2016-04-29 02:09:45 DEBUG [30241] Cluster[]: Process node: Node[3]
2016-04-29 02:09:45 INFO [30241] Casting message to Nailgun:
{"method"
"args"=>
{"task_
"nodes"=>
[{"uid"=>"1",
"
"
"
"
"custom"=>
...
...
node-1 haproxy.log:
2016-04- 29T00:57: 08.281784+ 00:00 alert: Server keystone-1/node-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 29T00:57: 08.281825+ 00:00 alert: Server keystone-1/node-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 29T00:57: 08.323573+ 00:00 alert: Server keystone-1/node-2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 29T00:57: 08.323755+ 00:00 alert: Server keystone-1/node-2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 29T00:57: 08.367596+ 00:00 alert: Server keystone-1/node-5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 29T00:57: 08.367596+ 00:00 emerg: proxy keystone-1 has no server available! 29T00:57: 08.367909+ 00:00 alert: Server keystone-1/node-5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 29T00:57: 08.367909+ 00:00 emerg: proxy keystone-1 has no server available! 29T00:57: 08.407086+ 00:00 alert: Server keystone-2/node-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 29T00:57: 08.407354+ 00:00 alert: Server keystone-2/node-1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 29T00:57: 08.451365+ 00:00 alert: Server keystone-2/node-2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 4ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 29T00:57: 08.451466+ 00:00 alert: Server keystone-2/node-2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 4ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 29T00:57: 08.492305+ 00:00 alert: Server keystone-2/node-5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 29T00:57: 08.492305+ 00:00 emerg: proxy keystone-2 has no server available! 29T00:57: 08.492429+ 00:00 alert: Server keystone-2/node-5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. 29T00:57: 08.492429+ 00:00 emerg: proxy keystone-2 has no server available!
2016-04-
2016-04-
2016-04-
2016-04-
2016-04-
2016-04-
2016-04-
2016-04-
2016-04-
2016-04-
2016-04-
2016-04-
2016-04-
2016-04-
2016-04-
node-1 puppet-apply.log:
2016-04- 29T01:31: 30.352294+ 00:00 err: Timeout waiting for HAProxy backend: 'keystone-2' status to become: 'up' after 1200 seconds!