One of the recent builds of 10.0 community BVT (https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/618/) failed with:
======================================================================
FAIL: Deploy ceph HA with RadosGW for objects
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 120, in wrapper
result = func(*args, **kwargs)
File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/tests/test_ceph.py", line 511, in ceph_rados_gw
self.fuel_web.deploy_cluster_wait(cluster_id)
File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 462, in wrapper
result = func(*args, **kwargs)
File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 447, in wrapper
result = func(*args, **kwargs)
File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 498, in wrapper
return func(*args, **kwargs)
File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 505, in wrapper
result = func(*args, **kwargs)
File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 389, in wrapper
return func(*args, **kwargs)
File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 953, in deploy_cluster_wait
self.check_deploy_state(cluster_id, check_services, check_tasks)
File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 903, in check_deploy_state
self.assert_ha_services_ready(cluster_id)
File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/core/helpers/log_helpers.py", line 32, in wrapped
result = func(*args, **kwargs)
File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 205, in assert_ha_services_ready
interval=20, timeout=timeout)
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 126, in wait_pass
return raising_predicate()
File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 204, in <lambda>
should_fail=should_fail),
File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/core/helpers/log_helpers.py", line 32, in wrapped
result = func(*args, **kwargs)
File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 1351, in run_ostf
failed_test_name=failed_test_name, test_sets=test_sets)
File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/core/helpers/log_helpers.py", line 32, in wrapped
result = func(*args, **kwargs)
File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 305, in assert_ostf_run
indent=1)))
AssertionError: Failed 1 OSTF tests; should fail 0 tests. Names of failed tests:
- Check state of haproxy backends on controllers (failure) Dead backends ['object-storage node-3 Status: DOWN/L7STS Sessions: 0 Rate: 0 ']. Please refer to OpenStack logs for more details.
According to haproxy logs it eventually went up on two of three controllers:
<133>Sep 17 18:32:32 node-5 haproxy[26861]: Proxy object-storage started.
<132>Sep 17 18:32:32 node-5 haproxy[26089]: Stopping proxy object-storage in 0 ms.
<132>Sep 17 18:32:32 node-5 haproxy[26089]: Proxy object-storage stopped (FE: 0 conns, BE: 0 conns).
<129>Sep 17 18:32:33 node-5 haproxy[26865]: Server object-storage/node-5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
<129>Sep 17 18:32:35 node-5 haproxy[26865]: Server object-storage/node-2 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
<129>Sep 17 18:32:35 node-5 haproxy[26865]: Server object-storage/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
<128>Sep 17 18:32:35 node-5 haproxy[26865]: proxy object-storage has no server available!
<133>Sep 17 19:12:37 node-5 haproxy[26865]: Server object-storage/node-5 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
<133>Sep 17 19:15:21 node-5 haproxy[26865]: Server object-storage/node-2 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 8ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
In syslog there is just one entree:
<27>Sep 17 19:11:58 node-3 systemd[1]: Failed to start Ceph rados gateway.
diagnostic snapshot: https://drive.google.com/open?id=0B2db-pBC_yblYVI1MFZhUjlUYkk
Still failing https:/ /ci.fuel- infra.org/ job/10. 0-community. main.ubuntu. bvt_2/630/ console