Mirantis OpenStack

[10.0][BVT] object-storage service is down on one of the nodes

Bug #1625227 reported by Roman Podoliaka on 2016-09-19

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Mirantis OpenStack	Confirmed	High	MOS Ceph	Mirantis OpenStack 10.0

Bug Description

One of the recent builds of 10.0 community BVT (https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/618/) failed with:

======================================================================
FAIL: Deploy ceph HA with RadosGW for objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 120, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/tests/test_ceph.py", line 511, in ceph_rados_gw
    self.fuel_web.deploy_cluster_wait(cluster_id)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 462, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 447, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 498, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 505, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 389, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 953, in deploy_cluster_wait
    self.check_deploy_state(cluster_id, check_services, check_tasks)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 903, in check_deploy_state
    self.assert_ha_services_ready(cluster_id)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/core/helpers/log_helpers.py", line 32, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 205, in assert_ha_services_ready
    interval=20, timeout=timeout)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 126, in wait_pass
    return raising_predicate()
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 204, in <lambda>
    should_fail=should_fail),
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/core/helpers/log_helpers.py", line 32, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 1351, in run_ostf
    failed_test_name=failed_test_name, test_sets=test_sets)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/core/helpers/log_helpers.py", line 32, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/10.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 305, in assert_ostf_run
    indent=1)))
AssertionError: Failed 1 OSTF tests; should fail 0 tests. Names of failed tests:
  - Check state of haproxy backends on controllers (failure) Dead backends ['object-storage node-3 Status: DOWN/L7STS Sessions: 0 Rate: 0 ']. Please refer to OpenStack logs for more details.

According to haproxy logs it eventually went up on two of three controllers:

<133>Sep 17 18:32:32 node-5 haproxy[26861]: Proxy object-storage started.
<132>Sep 17 18:32:32 node-5 haproxy[26089]: Stopping proxy object-storage in 0 ms.
<132>Sep 17 18:32:32 node-5 haproxy[26089]: Proxy object-storage stopped (FE: 0 conns, BE: 0 conns).
<129>Sep 17 18:32:33 node-5 haproxy[26865]: Server object-storage/node-5 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
<129>Sep 17 18:32:35 node-5 haproxy[26865]: Server object-storage/node-2 is DOWN, reason: Layer4 timeout, check duration: 2000ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
<129>Sep 17 18:32:35 node-5 haproxy[26865]: Server object-storage/node-3 is DOWN, reason: Layer4 timeout, check duration: 2001ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
<128>Sep 17 18:32:35 node-5 haproxy[26865]: proxy object-storage has no server available!
<133>Sep 17 19:12:37 node-5 haproxy[26865]: Server object-storage/node-5 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
<133>Sep 17 19:15:21 node-5 haproxy[26865]: Server object-storage/node-2 is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 8ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.

In syslog there is just one entree:

<27>Sep 17 19:11:58 node-3 systemd[1]: Failed to start Ceph rados gateway.

diagnostic snapshot: https://drive.google.com/open?id=0B2db-pBC_yblYVI1MFZhUjlUYkk

Tags: