After a successful deployment radosgw fails to start

[Roman Podoliaka]

BVT build https://ci.fuel-infra.org/job/11.0-community.main.ubuntu.bvt_2/292 failed with:

AssertionError: Failed 1 OSTF tests; should fail 0 tests. Names of failed tests:
  - Check state of haproxy backends on controllers (failure) Dead backends ['object-storage node-2 Status: DOWN/L7STS Sessions: 0 Rate: 0 ']. Please refer to OpenStack logs for more details.

apache fails to connect to radosgw fastcgi server:

[Fri Jan 06 23:23:56.249614 2017] [proxy:error] [pid 15361:tid 140243400759040] (111)Connection refused: AH00957: FCGI: attempt to connect to 1
27.0.0.1:9000 (127.0.0.1) failed
[Fri Jan 06 23:23:56.249666 2017] [proxy:error] [pid 15361:tid 140243400759040] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0
.1) for 60s
[Fri Jan 06 23:23:56.249678 2017] [proxy_fcgi:error] [pid 15361:tid 140243400759040] [client 10.109.3.6:44622] AH01079: failed to make connecti
on to backend: 127.0.0.1
[Fri Jan 06 23:23:56.252822 2017] [proxy:error] [pid 15361:tid 140243256715008] AH00940: FCGI: disabled connection for (127.0.0.1)
[Fri Jan 06 23:23:58.096740 2017] [proxy:error] [pid 15362:tid 140243392366336] (111)Connection refused: AH00957: FCGI: attempt to connect to 1
27.0.0.1:9000 (127.0.0.1) failed
[Fri Jan 06 23:23:58.096793 2017] [proxy:error] [pid 15362:tid 140243392366336] AH00959: ap_proxy_connect_backend disabling worker for (127.0.0
.1) for 60s
[Fri Jan 06 23:23:58.096799 2017] [proxy_fcgi:error] [pid 15362:tid 140243392366336] [client 10.109.3.4:34276] AH01079: failed to make connecti
on to backend: 127.0.0.1

daemon.log contains the following error:

2017-01-06T23:23:17.184952+00:00 node-2 systemd[1]: Created slice system-ceph\x2dradosgw.slice.
2017-01-06T23:23:17.208956+00:00 node-2 systemd[1]: Started Ceph rados gateway.
2017-01-06T23:23:17.366630+00:00 node-2 systemd[1]: Reloading.
2017-01-06T23:23:17.667151+00:00 node-2 systemd[1]: apt-daily.timer: Adding 10h 37min 36.754774s random time.
2017-01-06T23:23:17.696703+00:00 node-2 systemd[1]: Started ACPI event daemon.
2017-01-06T23:23:17.709078+00:00 node-2 radosgw[12268]: 2017-01-06 23:23:17.699924 7fc584123a00 -1 Couldn't init storage provider (RADOS)
2017-01-06T23:23:17.731911+00:00 node-2 systemd[1]: <email address hidden>: Main process exited, code=exited, status=5/NOTINSTALLED
2017-01-06T23:23:17.732375+00:00 node-2 systemd[1]: <email address hidden>: Unit entered failed state.
2017-01-06T23:23:17.732541+00:00 node-2 systemd[1]: <email address hidden>: Failed with result 'exit-code'.
2017-01-06T23:23:17.853779+00:00 node-2 systemd[1]: <email address hidden>: Service hold-off time over, scheduling restart.
2017-01-06T23:23:17.882913+00:00 node-2 systemd[1]: Stopped Ceph rados gateway.

[Vitalii Gridnev (vgridnev)]

Failed on 10.0 community bvt:

In logs of job execution (ceph_rados_gw) OSTF tests are failed because object storage backend failure.

2017-01-24 01:47:32 ERROR (nose_storage_plugin) fuel_health.tests.ha.test_haproxy.HAProxyCheck.test_001_check_state_of_backends
Traceback (most recent call last):
 File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 67, in testPartExecutor
   yield
 File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 601, in run
   testMethod()
 File "/usr/lib/python2.7/site-packages/fuel_health/tests/ha/test_haproxy.py", line 94, in test_001_check_state_of_backends
   len(dead_backends) == 0, error_message)
 File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 161, in verify_response_true
   self.fail(message.format(failed_step_msg, msg))
 File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
   raise self.failureException(msg)
AssertionError: Step 2 failed: Dead backends ['object-storage node-1 Status: DOWN/L7STS Sessions: 0 Rate: 0 ']. Please refer to OpenStack logs for more details.
-------------------- >> begin captured logging << --------------------
fuel_health.config: INFO: INSTANCE {}
fuel_health.config: INFO: INITIALIZING NAILGUN CONFIG

[0] https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/1118/
[1] https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/1118/artifact/logs/fail_error_ceph_rados_gw-fuel-snapshot-2017-01-24_02-08-17.tar

[Vitalii Gridnev (vgridnev)]

Another failure on 11.0 bvt:

======================================================================
FAIL: Deploy ceph HA with RadosGW for objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/11.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 120, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/11.0-community.main.ubuntu.bvt_2/fuelweb_test/tests/test_ceph.py", line 511, in ceph_rados_gw
    self.fuel_web.deploy_cluster_wait(cluster_id)
  File "/home/jenkins/workspace/11.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 462, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/11.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 447, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/11.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 498, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/11.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 505, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/11.0-community.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 389, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/11.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 967, in deploy_cluster_wait
    self.check_deploy_state(cluster_id, check_services, check_tasks)
  File "/home/jenkins/workspace/11.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 917, in check_deploy_state
    self.assert_ha_services_ready(cluster_id)
  File "/home/jenkins/workspace/11.0-community.main.ubuntu.bvt_2/core/helpers/log_helpers.py", line 215, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/11.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 207, in assert_ha_services_ready
    interval=20, timeout=timeout)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 126, in wait_pass
    return raising_predicate()
  File "/home/jenkins/workspace/11.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 206, in <lambda>
    should_fail=should_fail),
  File "/home/jenkins/workspace/11.0-community.main.ubuntu.bvt_2/core/helpers/log_helpers.py", line 215, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/11.0-community.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 1381, in run_...

[Vitalii Gridnev (vgridnev)]

https://ci.fuel-infra.org/job/11.0-community.main.ubuntu.bvt_2/366/console